A method, system and electronic device for identifying regions of a fluorescently stained section

Abstract

A region identification method, system and electronic device of a fluorescently stained section, characterized in that: the fluorescently stained section is scanned to obtain a bright field map and a dark field map, and the dark field map is filled into the bright field map to obtain a dark field macroscopic map; the dark field macroscopic map is input into a trained deep learning detection model to output coordinate information of a region of interest of the fluorescently stained section. The present application realizes the speedup of the scanning of the region of interest on the basis of zero additional cost, and can also improve the accuracy of the target acquisition of the region of interest.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of slice image detection, in particular to a region identification method, system and electronic equipment for fluorescently stained slices. BACKGROUND

[0002] At present, the existing region of interest identification of fluorescently stained slices mostly adopts traditional algorithms, which is easy to miss the region of interest and also easy to have extra regions of interest. Because the traditional detection algorithm utilizes morphological features (such as contour) of an image, optical density features (such as brightness, color and reflectivity under microscopic imaging), hierarchical structure features (such as hierarchical relationship between dye and target), threshold processing and the like, which may be a solution under bright field observation, but under dark field, other interference information will be introduced under fluorescent staining. Figure 11 The slice imaging image in the reference, in which the label, glass edge, marker pen and dye may all be an interference, which will affect the detection structure. SUMMARY

[0003] The main purpose of the present application is to overcome the above-mentioned defects of the detection method in the prior art, and to provide a region identification method, system and electronic equipment for fluorescently stained slices, which can improve the accuracy of obtaining the region of interest and realize the speedup of scanning the region of interest on the basis of zero additional cost.

[0004] The present application adopts the following technical solutions:

[0005] A region identification method for fluorescently stained slices, characterized in that it comprises:

[0006] scanning a fluorescently stained slice to obtain a bright field map and a dark field map, filling the dark field map to the bright field map to obtain a dark field macroscopic map; inputting the dark field macroscopic map into a trained deep learning detection model to output coordinate information of a region of interest of the fluorescently stained slice.

[0007] Preferably, the image of the fluorescently stained slice is pre-evaluated and calculated to determine a gain value, and the fluorescently stained slice is scanned based on the gain value to obtain the dark field map.

[0008] Preferably, the evaluation and calculation to determine the gain value are specifically:

[0009] Based on the current gain value, a plurality of fields of view are selected in the middle region of the image of the fluorescently stained slice;

[0010] The proportion of black pixel points in the plurality of fields of view is calculated;

[0011] The calculated black pixel point proportion is compared with a set threshold value, and when the calculated black pixel point proportion is greater than the set threshold value, the current gain value is adopted, and if not, the current gain value is increased.

[0012] Preferably, the black pixel point proportion calculation is specifically: first, the image is subjected to a weighted average method gray value processing:

[0013] FOB gray =(0.299*FOV R +0.587*FOV G +0.114*FOV B )

[0014] Wherein, FOV R is the R channel information of the FOV image, FOV G is the G channel information of the FOV image, and FOV B is the B channel information of the FOV image.

[0015] Then, black pixel statistics and calculation are performed to obtain a black pixel proportion calculation formula:

[0016]

[0017] Wherein, δ(i n ,j n ) is a function, δ(i n ,j n ) = 0 when the gray value of the position information (i n ,j n ) of the image is equal to 0, and otherwise is 1. Sums all the pixel points of the image, and n represents the nth visual field; w n represents the width information of the nth visual field; h m represents the height information of the nth visual field.

[0018] Preferably, the trained deep learning detection model is obtained by training using a YOLOv5 model framework, which includes a Backbone layer, a Neck layer and a Head layer; the Backbone layer is used for feature extraction of the input dark field macrograph to obtain a feature map; the Neck layer feature map is used for multi-scale feature fusion of the feature map to generate a feature map with multi-scale information; and the Head layer detects and regresses the feature map with multi-scale information to output coordinate information of the region of interest of the fluorescent staining section.

[0019] Preferably, before training the YOLOv5 model, different models of scanners are used to scan the same fluorescently stained section to obtain different dark field macrographs as training set images.

[0020] Preferably, the deep learning detection model uses a Faster R-CNN model or an SSD model or a Nanodet model.

[0021] Preferably, it further comprises model conversion of the trained deep learning detection model to TorchScript based on PyTorch for deployment on an application end, a local end or a cloud end.

[0022] A region identification system for fluorescently stained sections, characterized by comprising:

[0023] A scanning device for scanning a fluorescently stained section to obtain a bright field map and a dark field map, and filling the dark field map to the bright field map to obtain a dark field macrograph;

[0024] A deep learning detection model for inputting the dark field macrograph and outputting coordinate information of a region of interest of the fluorescently stained section.

[0025] An electronic device, comprising:

[0026] One or more processors;

[0027] A storage device for storing one or more programs,

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the region identification method for fluorescently stained sections as described.

[0029] As can be seen from the above description of the present application, compared with the prior art, the present application has the following beneficial effects:

[0030] In the present application, a bright field map and a dark field map are obtained by scanning a fluorescently stained section, the dark field map is filled to the bright field map to obtain a dark field macrograph, the dark field macrograph is inputted into a trained deep learning detection model, and coordinate information of a region of interest of the fluorescently stained section is outputted, thereby realizing the speedup of scanning the region of interest on the basis of zero additional cost and improving the accuracy of target acquisition of the region of interest.

[0031] In the present application, the gain value is determined by pre-evaluating and calculating the image of the fluorescently stained section, and the fluorescently stained section is scanned based on the gain value to obtain a dark field map, so as to meet the contrast requirement of the image and ensure that the image quality meets the requirement.

[0032] In the application, the YOLOv5 model is used for training to obtain a deep learning detection model, which has small dependence on computing power, and can obtain high frame rate operation, and is compatible with different application end deployment, and is suitable for current use scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The method flowchart of the application is shown in the figure;

[0034] Figure 2 The dark field map is filled into the bright field map, and the schematic diagram is shown in the figure;

[0035] Figure 3 The operation interface for obtaining the bright field map is shown in the figure;

[0036] Figure 4 The gain value determination flowchart is shown in the figure;

[0037] Figure 5 The bright field map is exemplified, and the figure is shown in the figure;

[0038] Figure 6 The dark field macrograph is exemplified, and the figure is shown in the figure;

[0039] Figure 7 The framework diagram of the YOLOv5 model is shown in the figure;

[0040] Figure 8 The code of the YOLOv5 model is shown in the figure;

[0041] Figure 9 The output dark field macrograph with coordinate information is shown in the figure;

[0042] Figure 10 The existing slice imaging graph is classified, and the figure is shown in the figure;

[0043] Figure 11 The existing slice imaging graph is exemplified, and the figure is shown in the figure;

[0044] The application will be further described in combination with the drawings and specific embodiments. DETAILED DESCRIPTION

[0045] The application will be further described in combination with the drawings and specific embodiments.

[0046] In the present application, the terms "first", "second", "third" and the like are used only to distinguish similar objects, and do not have to be used to describe a particular order or sequence, nor can they be understood to indicate or imply relative importance. In the description, the orientation or position relationship indicated by "up", "down", "left", "right", "front" and "back" and the like is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application, and does not indicate or imply that the device referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the scope of protection of the present application. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0047] In addition, in the description of the present application, "a plurality of" means two or more, unless otherwise specified. The association relationship between the associated objects described by "and / or" indicates that there can be three relationships, for example, A and / or B can represent the three cases of A existing alone, A and B existing together, and B existing alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0048] Related terms involved in the present application are explained:

[0049] YoLo (You Only Look Once) uses a single CNN model to realize end-to-end target detection.

[0050] CNN (Convolutional Neural Network) is used for recognition.

[0051] Bright field map: macroscopic map obtained by shooting under high magnification objective, see Figure 10 In order to obtain various types of bright field maps, the definition of bright field map types is as follows: A class: very obvious outline; B class: poor outline; C class: need to be observed at a specific angle.

[0052] Dark field image: supplement to the information of bright field map.

[0053] Dark field macroscopic map: image obtained after filling the dark field image to the bright field map.

[0054] See Figure 1 , a region recognition method of a fluorescently stained section, comprising the following steps:

[0055] 1) Scanning the fluorescently stained section to obtain a bright field map and a dark field map, filling the dark field map to the bright field map to obtain a dark field macroscopic map, see Figure 2 .

[0056] In this step, the bright field map can be obtained by loading the fluorescently stained section into the scanner for scanning, seeFigure 3 、 Figure 5 .

[0057] For the dark field map, the dark field map is obtained by switching the scanner to a low magnification objective and then performing image acquisition. Specifically, the gain value is determined by evaluating and calculating the image of the fluorescently stained slice in advance, and the fluorescently stained slice is scanned based on the gain value to obtain the dark field map.

[0058] Filling the dark field map into the bright field map to obtain a dark field macroscopic map means that the dark field image is mapped onto the bright field map and spliced, so that the dark field macroscopic map is obtained, i.e. a complete image is obtained through the mapping relationship.

[0059] Further, referring to Figure 4 , the evaluation and calculation to determine the gain value is specifically as follows:

[0060] 1.1) Based on the current gain value, a plurality of fields of view are selected in the middle region of the image of the fluorescently stained slice;

[0061] 1.2) Black pixel point proportion calculation is performed on the plurality of fields of view;

[0062] 1.3) The calculated black pixel point proportion is compared with the set threshold value, when the calculated black pixel point proportion is greater than the set threshold value, the current gain value is used, if not, the current gain value is increased, which can be increased by a set increase value, i.e. the current gain value is added to the increase value, and returns to step 1.1).

[0063] The number of selected fields of view can be set according to actual conditions, for example, 10 fields of view. The value of the set threshold value can be set in advance, for example, it can be 40%, i.e. 0.4, but is not limited thereto. The black pixel point proportion calculation is specifically as follows:

[0064] 1.2.1) First, the gray value of the image is processed by the weighted average method:

[0065] FOB gray =(0.299*FOV R +0.587*FOV G +0.114*FOV B )

[0066] Wherein, FOV R is the R channel information of the FOV image, FOV G is the G channel information of the FOV image, and FOV B is the B channel information of the FOV image.

[0067] 1.2.2) Then, black pixel statistics and calculation are performed to obtain the black pixel proportion calculation formula:

[0068]

[0069] Where δ(i) n ,j n ) is a function that calculates the position information (i) of the image. n ,j n When the grayscale value is equal to 0, δ(i) n ,j n ) = 0, otherwise 1; This represents the summation of all pixels in the image, where n represents the nth field of view; w n h represents the width information of the nth field of view. m The height information of the nth field of view is multiplied to obtain the field of view size information.

[0070] See Figure 6 This is an example of a dark-field macro-light map obtained by filling a bright-field map with a dark-field map.

[0071] 2) Input the dark field macroscopic image into the trained deep learning detection model and output the coordinate information of the region of interest of the fluorescent stained section.

[0072] In this invention, the trained deep learning detection model is obtained by training using the YOLOv5 model framework, see [link to relevant documentation]. Figure 7 It includes a Backbone layer, a Neck layer, and a Head layer. The Backbone layer extracts features from the input dark-field macroscopic image to obtain a feature map for subsequent object detection tasks. The Neck layer's feature map is used for multi-scale feature fusion to generate a feature map with multi-scale information, which is then passed to the Head layer to improve object detection accuracy. The Head layer performs the final detection, performing detection and regression prediction on the feature map with multi-scale information, outputting the coordinate information of the region of interest for the fluorescently stained section.

[0073] In this embodiment, based on the YOLOv5 model framework described above, the data obtained from data collection in step 1) can be used as a training set for training, ultimately yielding a PyTorch model for the deep learning detection model. The YOLOv5 model network has low computational dependence, thus enabling high frame rates, while also being compatible with deployments on different application platforms, making it suitable for current usage scenarios.

[0074] Furthermore, before training the YOLOv5 model, a certain amount of training data needs to be obtained. Different types of scanners can be used to scan the same fluorescently stained slide to obtain different dark-field macroscopic images as training set images; that is, step 1) is used to obtain dark-field macroscopic images of different samples. Then, for each sample, the following formula can be obtained: [M 荧光扫描仪型号 * 荧光扫描仪数量* 荧光切片数量 The amount of data can be increased by combining these factors, which can enhance the model's generalization ability and address the shortcomings of the difficulty and scarcity in obtaining fluorescently stained samples.

[0075] This invention also includes a model deployment step: converting the trained deep learning detection model to TorchScript using PyTorch, and deploying it on the application, local machine, or cloud according to the usage scenario requirements. Specifically, it uses an open-source model training framework, employing the YOLOv5 training framework for training and parameter tuning, and then converting the model using PyTorch for use in a C++ terminal.

[0076] In this step, the output coordinate information includes the X and Y coordinates of the top-left corner of the target, and the X and Y coordinates of the bottom-right corner of the target. This output coordinate information will be applied to a dark-field macro map to obtain... Figure 9 The rectangular frame shown is used for scanning. Instead of scanning the entire slice, the scanner only needs to scan the area within the rectangular frame, thus reducing scanning time.

[0077] In this invention, the deep learning detection model can also be implemented using other models, including Faster R-CNN, SSD, or Nanodet, which can be selected according to the actual situation.

[0078] This invention also proposes a region identification system for fluorescently stained sections, including...

[0079] The scanning device scans the fluorescently stained slides to obtain bright-field and dark-field maps, and then fills the bright-field map with the dark-field map to obtain a dark-field macroscopic map. This device is used to perform step 1 above.

[0080] A deep learning detection model is used as input to a dark-field macroscopic image and outputs the coordinate information of the region of interest in the fluorescently stained section. This device is used to perform step 2 above.

[0081] An electronic device, comprising:

[0082] One or more processors;

[0083] Storage device for storing one or more programs.

[0084] When one or more programs are executed by one or more processors, the one or more processors implement the above-described method for region identification of fluorescently stained sections.

[0085] The present application is used to find the interested region of the fluorescent staining section before entering the scanning process. Then only the interested region is scanned in the scanning process of the fluorescent staining section, thereby reducing the scanning time. Moreover, only the interested region is scanned in the modeling, thereby reducing the modeling time; the above aspects are optimized, and the purpose of reducing the scanning time is finally achieved.

[0086] The above is only a specific embodiment of the present application, but the design concept of the present application is not limited thereto, and any non-essential modification of the present application using this concept shall be deemed to be an infringement of the protection scope of the present application.

Claims

1. A method for region identification of a fluorescently stained section, characterized in that, The method comprises the following steps: Scanning the fluorescently stained slice to obtain a bright field map and a dark field map, pre-evaluating the image of the fluorescently stained slice to calculate a gain value, and scanning the fluorescently stained slice based on the gain value to obtain the dark field map; Mapping the dark field map onto the bright field map to splice the dark field macroscopic map; inputting the dark field macroscopic map into a trained deep learning detection model to output coordinate information of a region of interest of the fluorescently stained slice, wherein the output coordinate information comprises X and Y coordinate information of the upper left corner of a target and X and Y coordinate information of the lower right corner of the target.

2. The method of claim 1, wherein the step of identifying the region of the fluorescently stained section is performed by a computer program. The evaluation and calculation to determine the gain value specifically comprises: Selecting a plurality of fields of view in the middle region of the image of the fluorescently stained slice based on the current gain value; Calculating the proportion of black pixel points in the plurality of fields of view; Comparing the calculated proportion of black pixel points with a set threshold value, and if the calculated proportion of black pixel points is greater than the set threshold value, using the current gain value, otherwise, increasing the current gain value.

3. The method of claim 2, wherein the step of identifying the region of the fluorescently stained section comprises: The calculation of the proportion of black pixel points specifically comprises: first performing a weighted average method on the image to process the gray value: ； wherein, R is R channel information of the FOV image, G is G channel information of the FOV image, B is B channel information of the FOV image; and black pixel statistics and calculation are performed to obtain a black pixel proportion calculation formula: ； wherein, is a function of the position information of the image is equal to 0, otherwise 1; denotes the summation over all pixels of the image, and n denotes the nth field of view; denotes the width information of the nth field of view; denotes the height information of the nth field of view.

4. The method of claim 1, wherein the step of identifying the region of the fluorescently stained section is performed by a computer program. The trained deep learning detection model is trained by using a YOLOv5 model framework, which comprises a Backbone layer, a Neck layer and a Head layer; the Backbone layer is used to extract features of the input dark field macroscopic map to obtain a feature map; the Neck layer feature map is used to fuse multi-scale features of the feature map to generate a feature map with multi-scale information; and the Head layer is used to detect and regress the feature map with multi-scale information to output the coordinate information of the region of interest of the fluorescently stained slice.

5. The method of claim 1, wherein the step of identifying the region of the fluorescently stained section comprises: determining a region of interest in the fluorescently stained section based on the image of the fluorescently stained section. 5 Before training the YOLOv5 model, different models of scanners are used to scan the same fluorescently stained slice to obtain different dark field macroscopic maps as images of a training set.

6. The method of claim 1, wherein the step of identifying the region of the fluorescently stained section is performed by a computer program. The deep learning detection model uses a Faster R-CNN model or an SSD model or a Nanodet model.

7. The method of claim 1, wherein the step of identifying the region of the fluorescently stained section is performed by a computer program. The trained deep learning detection model is further converted to TorchScript based on PyTorch for deployment on an application end, a local end or a cloud end.

8. A system for region identification of fluorescently stained sections, characterized in that The method comprises the following steps: A scanning device is used to scan a fluorescently stained slice to obtain a bright field map, to evaluate an image of the fluorescently stained slice to calculate a gain value, and to scan the fluorescently stained slice based on the gain value to obtain a dark field map; Mapping the dark field map onto the bright field map to splice the dark field macroscopic map; A deep learning detection model is used to input the dark field macroscopic map, to output coordinate information of a region of interest of the fluorescently stained slice, and to output X and Y coordinate information of the upper left corner of a target and X and Y coordinate information of the lower right corner of the target. 9.An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement a method for identifying a region of a fluorescently stained section as recited in any one of claims 1-7.

Images