Method and system for medical slide positioning, identification, and fractionation

Abstract

The application discloses a medical slide positioning, recognizing and slicing method and system, which comprises the following steps: obtaining a slide digital image; performing text recognition on the obtained slide digital image to obtain text information and text position on the slide image; positioning each slice on the obtained slide digital image to obtain the position coordinates of each slice; recognizing a two-dimensional code of the slide digital image to obtain a digital number corresponding to the two-dimensional code; and storing the recognition result and delivering the slice image. The method meets the industrial demand, is simple to operate and high in precision, effectively improves the manual input speed, greatly improves the automation degree of the slide slice sample in transportation, and plays an important role in the full automation of the pathological diagnosis process.

Description

Technical Field

[0001] This invention relates to the field of pathological image processing technology, and in particular to methods and systems for locating, identifying, and segmenting medical slides. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] In the medical diagnostic process, pathological diagnosis is considered the "gold standard" of clinical diagnosis and is a crucial basis for doctors' pathological judgments. Pathological diagnosis primarily relies on the observation of the pathological morphology of cell sections. The delivery and distribution of slides is an indispensable part of the pathological diagnostic process, and the efficiency of slide delivery and distribution is a significant factor affecting the overall efficiency of the pathological diagnostic process.

[0004] In recent years, with the rapid growth in demand for pathological diagnosis, the efficiency of distributing pathological slides has also urgently needed to be further improved. The general method of manually entering slide information requires a lot of unnecessary human resources. A large amount of repetitive work is time-consuming, laborious, and prone to fatigue, leading to errors in data entry, which can no longer meet the current needs.

[0005] Traditional slide distribution methods require manual input of information from each slide label, which consumes a lot of time and effort for slide distributors, resulting in low efficiency and a high risk of errors. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for locating, identifying, and segmenting medical slides;

[0007] In a first aspect, the present invention provides a method for locating, identifying, and segmenting medical glass slides;

[0008] Methods for locating, identifying, and segmenting medical slides include:

[0009] Acquire digital images of glass slides;

[0010] The obtained digital image of the slide is subjected to text recognition to obtain the text information and text location on the slide image;

[0011] Each slice on the obtained digital image of the glass slide is located to obtain the position coordinates of each slice;

[0012] The QR code on the digital image of the glass slide is identified to obtain the corresponding digital number;

[0013] The recognition results are stored, and the sliced ​​images are distributed.

[0014] Secondly, this invention provides a system for positioning, identifying, and segmenting medical slides;

[0015] A system for positioning, identifying, and segmenting medical slides includes:

[0016] The acquisition module is configured to acquire digital images of glass slides.

[0017] The text recognition module is configured to: perform text recognition on the obtained digital image of the slide to obtain the text information and text position on the slide image;

[0018] The positioning module is configured to: locate each slice on the obtained digital image of the slide and obtain the position coordinates of each slice;

[0019] The QR code recognition module is configured to: recognize the QR code in the digital image of the glass slide and obtain the corresponding digital number of the QR code;

[0020] The distribution module is configured to store the recognition results and distribute the sliced ​​images.

[0021] Thirdly, the present invention also provides an electronic device, comprising:

[0022] Memory, used for non-transitory storage of computer-readable instructions; and

[0023] Processor, for executing the computer-readable instructions,

[0024] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0025] Fourthly, the present invention also provides a storage medium for non-transitory storage of computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions for the method described in the first aspect are executed.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] This invention proposes a system and method that can scan, automatically identify, and distribute medical slides. Compared with the traditional method of manually entering slide information, the text recognition rate can reach over 95%, and the recognition rate can reach over 90% while ensuring a certain resolution for QR codes. This meets industrial needs, is simple to operate, and has high accuracy, effectively improving the speed of manual data entry and greatly enhancing the automation of slide sample transportation. It plays an important role in the full automation of the pathological diagnosis process.

[0028] The advantages of additional aspects of the invention will be set forth in part in the description which follows, or may be learned by practice of the invention. Attached Figure Description

[0029] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0030] Figure 1 This is a system flowchart for the automated segmentation of glass slide samples according to the present invention;

[0031] Figure 2 This is a flowchart of the algorithm for automated recognition of slide images according to the present invention;

[0032] Figure 3 This is a diagram showing the results of the automated recognition of glass slide images according to the present invention. Detailed Implementation

[0033] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0034] Example 1

[0035] This embodiment provides a method for locating, identifying, and segmenting medical glass slides;

[0036] like Figure 1 As shown, the methods for locating, identifying, and segmenting medical slides include:

[0037] S101: Acquire digital image of glass slide;

[0038] S102: Perform text recognition on the obtained digital image of the slide to obtain the text information and text position on the slide image;

[0039] S103: Locate each slice on the obtained digital image of the glass slide to obtain the position coordinates of each slice;

[0040] S104: Recognize the QR code in the digital image of the slide to obtain the corresponding digital number;

[0041] S105: Store the recognition results and distribute the sliced ​​images.

[0042] Further, step S101: acquiring a digital image of the glass slide; specifically includes:

[0043] A camera is used to photograph the glass slide and obtain digital images of the slide.

[0044] For example, first, set up a camera device to capture images of the slide, neatly calibrate the position of each slice sample on the slide, place the slide flat under the light source of the document scanner for shooting, and adjust the position of the slide according to the effect to obtain a slide image with high clarity, aligned edges without missing edges, and good quality.

[0045] Further, step S102: performing text recognition on the obtained digital image of the slide to obtain text information and text positions on the slide image; specifically including:

[0046] The obtained digital image of the slide is subjected to OCR (Optical Character Recognition) text recognition to obtain the text information and text position on the slide image.

[0047] For example, the text information includes: identification information such as the number, sequence number, and QR code number of each slice sample.

[0048] like Figure 2 As shown, after receiving the slide image captured by the high-speed scanner, the PaddleOCR tool is used to recognize characters and obtain the text information (number and sequence number) and text position coordinates of each slice. Based on the horizontal coordinate of each slice position, it is divided into two columns and then sorted from top to bottom according to the vertical coordinate, so as to obtain the slice information in the left and right columns for subsequent use.

[0049] Further, step S103 involves locating each slice on the obtained digital image of the slide to obtain the position coordinates of each slice; specifically including:

[0050] S1031: Based on the trained slice target detection model, each slice on the obtained digital image of the glass slide is located to obtain the position coordinates of each slice and the confidence level of identifying it as a slice;

[0051] S1032: After the slice position detection is completed, the recognition result is corrected and adjusted according to the area size and aspect ratio of the target box in the detection result. Target boxes whose area exceeds the set threshold or whose aspect ratio exceeds the set threshold are readjusted to be close to the real target box size.

[0052] S1033: For the text information on the identified slide image, obtain the number information of each slide according to the length and first letter of each text; wherein, the text information includes all the English letters, numbers and punctuation marks identified in the entire image;

[0053] S1034: For each slice number, based on the relative position coordinates of the number and the sequence number, determine the coordinate distance between the current number and all sequence numbers, search for the sequence number that is closest to the current number and meets the set conditions (the horizontal and vertical coordinates of the number and sequence number do not exceed 100 pixels), and form the identification information of the same slice.

[0054] For example, the slice target detection model uses the YOLOv4-Tiny model.

[0055] For example, the filtering is based on conditions such as 8 < text length < 20, and the first letter is k, 2, or 1, to obtain the number of each slice.

[0056] Furthermore, the training steps for the trained slice target detection model include:

[0057] Construct training and test sets; both training and test sets are digital images of glass slides with known coordinates of each slice position.

[0058] The training set is input into the slice object detection model to train the model;

[0059] Input the test set into the slice object detection model and test the model;

[0060] The trained slice target detection model is obtained.

[0061] It should be understood that the YOLOv4-Tiny model is constructed from a series of convolutional layers, pooling layers, activation functions, and batch normalization (BN) layers. After the slide image is input into the network and undergoes multi-layer feature extraction, it predicts and outputs information such as the class, location, and confidence score of each slice sample. Alternatively, other networks, such as SSD and Faster R-CNN, can be used for detection.

[0062] The data annotation can be done by drawing boxes on the original slide images using LabelImg or Labelme software. After training, the object detection model can be used directly to predict the slide images.

[0063] Further, step S104 involves recognizing the QR code on the digital image of the slide to obtain the corresponding digital number; specifically including:

[0064] S1041: During the QR code positioning process, firstly, based on the length and position coordinates of each slice number detected in S102, a fixed-size area (a rectangular area with a side length 1.5 times the number length) in the lower right corner is selected for QR code recognition.

[0065] If the QR code cannot be recognized, the Intersection over Union (IOU) is compared with the slice position identified by the slice target detection model based on the current slice number's position coordinates. If the IOU percentage is greater than a set threshold, the slice is considered to intersect; if the IOU percentage is less than or equal to the set threshold, the slice is considered to not intersect. If the slice is considered to intersect, the QR code is recognized in the target box region identified by the slice target detection model.

[0066] If the QR code cannot be recognized, the target box is detected again. The method is as follows: the target box area detected by the slice target detection model is grayscaled, then Gaussian filtering is performed to reduce noise and the Canny operator is used to detect the image edge information. After that, erosion and dilation are performed to process the lines. Finally, all target contours are found and the target box with the largest area detected is returned as the QR code positioning area.

[0067] S1042: In the process of QR code recognition, the QR code image detected in S1041 is first detected and recognized using the detector.detectAndDecode() function in the cv2.wechat_qrcode_WeChatQRCode tool;

[0068] If the QR code cannot be detected, continue to use the reader.decode_array() function in the zxing tool for recognition.

[0069] After identifying the number, sequence number, and QR code information of each slice, the sequence number of each slice is further optimized. Specifically, the secondary optimization method is as follows: count the number of identified and unidentified sequence numbers. If the number of unidentified sequence numbers is greater than the number of identified sequence numbers, the current image is defined as a type that does not need to identify sequence numbers, and the identified sequence number information is set to empty and an empty string information is returned. In addition, for the identified sequence number, it is determined whether it is in the misidentification list. If it is, forced error correction is performed directly (e.g., if the sequence number Ri-67 is in the error list, it is directly changed to Ki-67).

[0070] It should be understood that the QR code localization process employs relative position localization, IOU intersection localization, and traditional image processing localization methods. The relative position localization method utilizes the relative coordinates of the slice sample number characters and the QR code to locate a fixed-size region to the lower right of the number. The intersection localization method uses the IOU intersection of the number and the target bounding box output by the YOLOv4-tiny slice target detection model to determine the QR code position. The traditional image processing method performs a series of operations on the slide image, including grayscale conversion, Gaussian filtering, Canny edge detection, erosion, dilation, and contour finding, to output the largest possible QR code region.

[0071] Further, step S105: storing the recognition results and distributing the sliced ​​images; specifically including:

[0072] The results after recognition are as follows Figure 3 As shown, the results obtained from the slide recognition are transmitted into the system and distributed to the desired doctors for pathological diagnosis.

[0073] The automated recognition technology includes the following: OCR text recognition information includes the English characters and confidence scores of all text in the slide image, the category, location coordinates, and confidence scores of the target box detection of the slice sample, and QR code recognition information includes location coordinates and characters.

[0074] Example 2

[0075] This embodiment provides a system for positioning, identifying, and segmenting medical slides;

[0076] A system for positioning, identifying, and segmenting medical slides includes:

[0077] The acquisition module is configured to acquire digital images of glass slides.

[0078] The text recognition module is configured to: perform text recognition on the obtained digital image of the slide to obtain the text information and text position on the slide image;

[0079] The positioning module is configured to: locate each slice on the obtained digital image of the slide and obtain the position coordinates of each slice;

[0080] The QR code recognition module is configured to: recognize the QR code in the digital image of the glass slide and obtain the corresponding digital number of the QR code;

[0081] The distribution module is configured to store the recognition results and distribute the sliced ​​images.

[0082] It should be noted that the acquisition module, text recognition module, positioning module, QR code recognition module, and distribution module mentioned above correspond to steps S101 to S105 in Embodiment 1. The examples and application scenarios implemented by the above modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0083] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0084] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0085] Example 3

[0086] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0087] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0088] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0089] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0090] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0091] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0092] Example 4

[0093] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for locating, identifying, and segmenting medical slides, characterized by: include: Acquire digital images of glass slides; The obtained digital image of the slide is subjected to text recognition to obtain the text information and text location on the slide image; Based on the trained slice target detection model, each slice on the obtained digital image of the glass slide is located to obtain the position coordinates of each slice and the confidence level of identifying it as a slice; After the slice location detection is completed, the recognition results are corrected and adjusted according to the area and aspect ratio of the target box in the detection results. Target boxes with an area exceeding the set threshold or an aspect ratio exceeding the set threshold are readjusted to a size close to the real target box. For the text information on the identified slide images, the number information of each slide is obtained according to the length and first letter of each text. For each slice number, based on the relative position coordinates of the number and the sequence number, determine the coordinate distance between the current number and all sequence numbers, search for the sequence number that is closest to the current number and meets the set conditions, and form the identification information of the same slice; The QR code of the digital image of the slide is identified to obtain the corresponding digital number. Specifically, in the process of QR code positioning, a fixed-size area in the lower right corner is selected for QR code recognition based on the length and position coordinates of each detected slide number. If the QR code cannot be recognized, the current slice number's position coordinates are compared with the slice position recognized by the slice target detection model, and the Intersection over Union (IOU) is compared. If the IOU ratio is greater than a set threshold, it is judged as intersecting; if the IOU ratio is less than or equal to the set threshold, it is judged as not intersecting; if it is judged as intersecting, the QR code is recognized in the target box area identified by the slice target detection model. If the QR code still cannot be recognized, the target box will continue to be detected. The method is as follows: the target box area detected by the slice target detection model is grayscaled, then Gaussian filtering is performed to reduce noise and the Canny operator is used to detect the image edge information. After that, erosion and dilation are performed to process the lines. Finally, all target contours are found and the target box with the largest area detected is returned as the QR code positioning area. After identifying the number, sequence number, and QR code information of each slice, the sequence number of each slice is further optimized. Specifically, the secondary optimization method is as follows: count the number of identified and unidentified sequence numbers. If the number of unidentified sequence numbers is greater than the number of identified sequence numbers, the current image is defined as a type that does not need to identify sequence numbers, and the identified sequence number information is set to empty and an empty string information is returned. In addition, for the sequence numbers that have been identified, it is determined whether they are in the misidentification list. If they are, forced error correction is performed directly. The recognition results are stored, and the sliced ​​images are distributed.

2. The method for locating, identifying, and segmenting medical slides as described in claim 1, characterized in that, Acquiring digital images of a glass slide specifically includes: using a camera device to capture images of the glass slide and acquire digital images of the glass slide.

3. The method for locating, identifying, and segmenting medical slides as described in claim 1, characterized in that, The obtained digital image of the slide is subjected to character recognition to obtain the text information and text position on the slide image. Specifically, this includes: performing optical character recognition on the obtained digital image of the slide to obtain the text information and text position on the slide image.

4. The method for locating, identifying, and segmenting medical slides as described in claim 1, characterized in that, The trained slice target detection model includes the following training steps: constructing a training set and a test set; both the training set and the test set are digital images of glass slides with known coordinates of each slice position. The training set is input into the slice object detection model to train the model; Input the test set into the slice object detection model and test the model; The trained slice target detection model is obtained.

5. A system for positioning, identifying, and segmenting medical slides, characterized in that: include: The acquisition module is configured to acquire digital images of glass slides. The text recognition module is configured to: perform text recognition on the obtained digital image of the slide to obtain the text information and text position on the slide image; The localization module is configured to: locate each slice on the obtained digital image of the slide based on the trained slice target detection model, and obtain the position coordinates of each slice and the confidence level of identifying it as a slice; After the slice location detection is completed, the recognition results are corrected and adjusted according to the area and aspect ratio of the target box in the detection results. Target boxes with an area exceeding the set threshold or an aspect ratio exceeding the set threshold are readjusted to a size close to the real target box. For the text information on the identified slide images, the number information of each slide is obtained according to the length and first letter of each text. For each slice number, based on the relative position coordinates of the number and the sequence number, determine the coordinate distance between the current number and all sequence numbers, search for the sequence number that is closest to the current number and meets the set conditions, and form the identification information of the same slice; The QR code recognition module is configured to: recognize the QR code in the digital image of the slide and obtain the corresponding digital number of the QR code. Specifically, during the QR code positioning process, a fixed-size area in the lower right corner is selected for QR code recognition based on the length and position coordinates of each detected slice number. If the QR code cannot be recognized, the current slice number's position coordinates are compared with the slice position recognized by the slice target detection model, and the Intersection over Union (IOU) is compared. If the IOU ratio is greater than a set threshold, it is judged as intersecting; if the IOU ratio is less than or equal to the set threshold, it is judged as not intersecting; if it is judged as intersecting, the QR code is recognized in the target box area identified by the slice target detection model. If the QR code still cannot be recognized, the target box will continue to be detected. The method is as follows: the target box area detected by the slice target detection model is grayscaled, then Gaussian filtering is performed to reduce noise and the Canny operator is used to detect the image edge information. After that, erosion and dilation are performed to process the lines. Finally, all target contours are found and the target box with the largest area detected is returned as the QR code positioning area. After identifying the number, sequence number, and QR code information of each slice, the sequence number of each slice is further optimized. Specifically, the secondary optimization method is as follows: count the number of identified and unidentified sequence numbers. If the number of unidentified sequence numbers is greater than the number of identified sequence numbers, the current image is defined as a type that does not need to identify sequence numbers, and the identified sequence number information is set to empty and an empty string information is returned. In addition, for the sequence numbers that have been identified, it is determined whether they are in the misidentification list. If they are, forced error correction is performed directly. The distribution module is configured to store the recognition results and distribute the sliced ​​images.

6. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-4.

7. A storage medium, characterized in that, The computer-readable instructions are stored non-transitory, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-4 are executed.

Images