A semi-automatic labeling method for chromosome karyogram image data

Abstract

The present application relates to medical chromosome karyotype image processing technical field, provide a kind of semi-automatic labeling method of chromosome karyotype image data.The method includes: obtaining the key information required for automatic labeling;Based on the key information, construct and generate a labeling file according to the specified data format, and extract the coordinate point information of the effective contour of the chromosome required for automatic labeling;Manual annotation is completed using the labeling tool to the unannotated chromosome.The present application automatically annotates the internal objects of chromosome karyotype image by image processing method, and checks and annotates the annotation results manually, shortens the time cost spent on labeling, greatly improves the labeling efficiency.

Description

Technical Field

[0001] This invention relates to the field of medical chromosome karyotype image processing technology, and in particular to a semi-automatic annotation method for chromosome karyotype image data. Background Technology

[0002] In recent years, deep learning has been widely applied in various fields such as medical imaging and human-computer interaction. However, the research and development of deep learning relies heavily on large datasets. Therefore, the annotation of large datasets has become a crucial research topic in the field of deep learning. In the medical field, the study of chromosome karyotype images is extremely important, serving as a necessary step in human genetics research and a key tool to assist doctors in clinical diagnosis. Therefore, research on using deep learning to accelerate chromosome karyotype image processing is urgently needed. However, using deep learning to process chromosome karyotype images requires a large dataset of labeled chromosome karyotype images. Currently, such datasets are very scarce on the internet. Therefore, to use deep learning to process and study chromosome karyotype images, it is essential to first figure out how to generate a large dataset of labeled chromosome karyotype images. Chromosome karyotype image annotation is pixel-level annotation, while manually annotating the outline of each chromosome in a karyotype image at the pixel level is time-consuming.

[0003] Image annotation can be categorized into image-level annotation and pixel-level annotation based on its intended purpose. Image-level annotation is generally used for classification tasks in deep learning, while pixel-level annotation is typically used for segmentation tasks. Within pixel-level image annotation, methods can be further divided into manual annotation and automated annotation, depending on who is primarily responsible for the annotation. Manual annotation involves human intervention and typically involves inviting experts with domain knowledge to manually label objects within the image using specialized annotation tools or software. This process is time-consuming. For example, manually annotating a chromosome karyotype image takes an average of 55 minutes and is prone to significant errors.

[0004] In addition, the traditional method of using only a single fixed threshold for image binarization can lead to drastically different results for different images due to the significant differences in the background grayscale value range of different chromosome karyotype images. This results in poor contour extraction performance. Summary of the Invention

[0005] In view of this, the present invention provides a semi-automatic annotation method for chromosome karyotype image data to solve the problem that the binarization results of different images with a fixed threshold in the prior art are completely different, resulting in poor contour extraction effect and low annotation efficiency.

[0006] This invention provides a semi-automatic annotation method for chromosome karyotype image data, comprising:

[0007] S1 uses image processing methods to obtain the key information required for automatic annotation;

[0008] S2 constructs and generates a labeling file based on the key information according to the specified data format, and extracts the coordinate point information of the effective outline of the chromosome required for automatic labeling;

[0009] S21 uses a filtering algorithm to reduce noise in the chromosome karyotype image;

[0010] S22 performs binarization processing on the chromosome karyotype image using a corresponding fixed threshold;

[0011] S23 uses a detection algorithm to perform edge detection on the chromosome karyotype image after the binarization algorithm;

[0012] S24 Extract and determine whether each edge contour belongs to the normal chromosome range. If yes, continue to S25. If no, discard the edge contours that do not belong to the normal chromosome range.

[0013] S25 counts the number of valid chromosome contours extracted and determines whether the current fixed threshold is the maximum value of valid contours when the number of valid chromosome contours extracted reaches the set number. If so, it is the target binarization threshold. The coordinate point information of all valid chromosome contours under the target binarization threshold is extracted and stored. If not, the current fixed threshold is adjusted and the process returns to S22.

[0014] S26, Based on the number of effective chromosome contours, construct a set of effective chromosome contours, and convert the data format of the effective contours into the data format of the key information of the coordinate points of the chromosome karyotype image contours;

[0015] S3 uses annotation tools to manually annotate and complete unannotated chromosomes.

[0016] Furthermore, the key information includes the size of the chromosome karyotype image, the absolute path of the chromosome karyotype image, the JSON string data of the chromosome karyotype image, the label name of the chromosome karyotype image, and the coordinate points of the outline of the chromosome karyotype image.

[0017] Further, S1 includes:

[0018] S11 obtains the chromosome karyotype image size and the absolute path of the chromosome karyotype image based on the image reading method;

[0019] S12 obtains the raw bytecode data D of the image based on the binary image reading method. rbThe original bytecode data is encoded into base64 bytecode data D using a data encoding method. b64 The data decoding method described above is used to decode the base64 bytecode data into UTF-8 format string data D. utf8 Obtain the JSON string data of the chromosome karyotype image;

[0020] S13 is based on a deep learning data training scenario, obtains data labels, and uses a custom annotation file to annotate the label names of different chromosomes based on the data labels, thereby obtaining the chromosome karyotype image annotation names and annotation information.

[0021] Furthermore, the acquisition of the specified data format in S2 includes: using a labeling tool to manually label the chromosome karyotype image and generate the labeling file;

[0022] Read the annotation file to obtain the specified data format.

[0023] Furthermore, the coordinates of the effective chromosome contour in S25 are obtained by dimensionality reduction of the three-dimensional data of the effective chromosome contour to generate two-dimensional data.

[0024] Furthermore, the current fixed threshold in S25 is adjusted accordingly based on different chromosome karyotype images.

[0025] Furthermore, the specific content of the manual annotation in S3 includes the annotation name.

[0026] Further, S3 includes:

[0027] Open the annotation file using an annotation tool to check the automatic annotation effect;

[0028] Based on the automatic annotation effect, the unannotated chromosomes are annotated and completed.

[0029] The advantages of this invention compared to the prior art are:

[0030] 1. This invention dynamically determines the target binarization threshold by extracting the number of contours, avoiding the errors caused by the traditional method of using a single fixed threshold;

[0031] 2. By binarizing the chromosome karyotype image, an ideal edge contour extraction effect is obtained;

[0032] 3. By using image processing methods to automatically construct annotation files based on the acquired key information, the annotation efficiency is improved. The average time to annotate a chromosome karyotype image using the method of this invention is only 5 minutes. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of a semi-automatic annotation method for chromosome karyotype image data provided by the present invention;

[0035] Figure 2 This is a flowchart of the process for obtaining key information required for automatic annotation, provided by the present invention;

[0036] Figure 3 This is a flowchart provided by the present invention for obtaining chromosome contour coordinate point information required for automatic annotation;

[0037] Figure 4 This is a flowchart of determining the target binarization threshold provided by the present invention;

[0038] Figure 5 This is a schematic diagram illustrating the principle of determining the target binarization threshold provided by the present invention;

[0039] Figure 6 This is a flowchart of the automatic construction and generation of annotation files provided by the present invention;

[0040] Figure 7 This is a diagram showing the effect of semi-automatic annotation of chromosome images provided by the present invention. Detailed Implementation

[0041] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0042] The flowchart of a semi-automatic annotation method for chromosome karyotype image data provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0043] Figure 1 This is a flowchart of a semi-automatic annotation method for chromosome karyotype image data disclosed in this invention.

[0044] like Figure 1 As shown, this semi-automatic annotation method includes:

[0045] S1, based on image processing methods, obtains the key information required for automatic annotation.

[0046] The key information includes the size of the chromosome karyotype image, the absolute path of the chromosome karyotype image, the JSON string data of the chromosome karyotype image, the label name of the chromosome karyotype image, and the coordinate points of the outline of the chromosome karyotype image.

[0047] Figure 2 This is a flowchart provided by the present invention for obtaining key information required for automatic labeling.

[0048] like Figure 2 As shown,

[0049] S11 obtains the chromosome karyotype image size and absolute path based on the image reading method.

[0050] The key information for obtaining the size and path of a chromosome karyotype image is as follows: The image is read using an image reading method to obtain the height, width, and path of the chromosome karyotype image, denoted as H, W, and Path, respectively. In this embodiment, H is 896, W is 1017, and the path is the local image storage path.

[0051] S12 obtains the raw bytecode data D of the image based on the binary image reading method. rb The raw bytecode data is encoded into base64 bytecode data using a data encoding method. b64 The data decoding method is used to decode base64 bytecode data into UTF-8 format string data D. utf8 Obtain the JSON string data of the chromosome karyotype image;

[0052] S13 is based on a deep learning data training scenario, obtains data labels, and uses a custom annotation file to annotate the label names of different chromosomes based on the data labels, thereby obtaining the chromosome karyotype image annotation names and annotation information.

[0053] The labeled dataset used for training deep learning models requires different annotation information for different chromosomes. In the chromosome karyotype image annotation scenario of this invention, there is only one valid category, chromosome, but there are multiple chromosome instances. Therefore, a custom annotation file F is used. L This includes different label names L i These correspond to the label names of different chromosome instances. In this embodiment, the specific content of the different labels includes:

[00] L1 = chromos1, L2 = chromos2, ... L i =chromosi,...L n =chromosn. Where Li is the label name, i = 1, 2, 3...n.

[0054] S2 generates a labeling file based on key information and in a specified data format, and extracts the coordinate point information of the effective outline of the chromosome required for automatic labeling.

[0055] The acquisition of specified data formats includes:

[0056] Annotation tools were used to manually annotate chromosome karyotype images and generate annotation files;

[0057] Among them, annotation tool F tool It belongs to the labelme annotation tool; the labelme annotation tool can annotate various points, lines and shapes in images, such as polygons, rectangles, lines, points and image-level annotations, and has strong universality.

[0058] Read the annotation file and obtain the specified data format.

[0059] Figure 3 This is a flowchart provided by the present invention for obtaining the coordinate points of chromosome outlines required for automatic annotation.

[0060] like Figure 3 As shown, S2 includes:

[0061] S21 uses a filtering algorithm to reduce noise in chromosome karyotype images.

[0062] (1) Since there are many noise points in the chromosome karyotype image, which affect the effect of automatic annotation, a filtering algorithm is required. Denoising is performed on chromosome karyotype images to reduce the impact of noise, resulting in a denoised image. In this embodiment, the filtering algorithm A Gaussian filtering method is used.

[0063] S22 uses a fixed threshold to binarize the chromosome karyotype image.

[0064] (2) Since the background color of the chromosome karyotype image is relatively simple, with only black, white and gray colors distributed, and the chromosomes are all dark pixel areas, the image after noise reduction in step (1) The edge changes are quite obvious, so a fixed threshold T can be used. i right Perform binarization processing, Medium grayscale values ​​below T i Change the value of the pixel to c0, and Medium grayscale values ​​higher than T i The pixel value is changed to c 255 This will make It becomes only c0 and c 255 Image G with two gray valuesgray In this embodiment, c0 represents a background area with a grayscale value of 0, which is represented as black. 255 This indicates a grayscale value of 255, represented as a white region of suspected chromosomes (a suspected chromosome refers to an object that could be a chromosome instance or an intracellular impurity rather than a chromosome instance), with a fixed threshold T. i This refers to using a fixed value for each binarization, specifically:

[00] T min =100, T max =230, T i =T min +i,i=1,2,3,4,...,T min ≤T i ≤T max .

[0065] S23 uses a detection algorithm to perform edge detection on the chromosome karyotype image after binarization.

[0066] (3) Since data annotation actually involves plotting points on the edges of each chromosome instance's region and connecting them to form a closed contour, the G obtained in step (2) above... gray Only two types of regions exist: black non-chromosome regions and white suspected chromosome regions. The edge information is very clear, so the edge detection algorithm ω can be used on G. gray Edge detection is performed to obtain a set S consisting of multiple suspicious chromosome outlines. In this embodiment, the edge detection algorithm ω adopts the Canny edge detection algorithm.

[0067] S24 extracts and determines whether each edge contour belongs to the normal chromosome range. If yes, continue to S25; otherwise, discard the edge contours that do not belong to the normal chromosome range.

[0068] (4) Since the edge detection algorithm obtains a set S of all suspicious chromosome contours, which includes contours of non-chromosome objects (such as large cell fluid contours and small cell debris contours), it is necessary to use chromosome feature information to filter the contours and extract the set S′ of valid chromosome contours in S. valid (Outlines containing chromosomal characteristic information among suspected chromosomes), and record the total number of valid outlines (cnt). The screening method specifically involves detecting the perimeter of each outline (si) through experimental comparison and empirical summarization. And whether the area of ​​the internal region is within the normal range of chromosomes, if and Both fall within the range of normal chromosomes, then s i To obtain a valid chromosome profile, add set S′ valid Simultaneously update cnt; otherwise s iIf the chromosome outline is invalid, the process returns to adjusting the current fixed threshold. In this embodiment, the perimeter and internal region area ranges of a normal chromosome are as follows:

[0069]

[0070]

[0071]

[0072] S25 counts the number of valid chromosome contours extracted and determines whether the current fixed threshold is the maximum value of valid contours when the number of valid chromosome contours extracted reaches the set number. If so, it is the target binarization threshold. The coordinate point information of all valid chromosome contours under the target binarization threshold is extracted and stored. If not, the current fixed threshold is adjusted and the process returns to S22.

[0073] The coordinates of the effective chromosome contour in S25 are obtained by dimensionality reduction of the three-dimensional data of the effective chromosome contour to generate two-dimensional data.

[0074] The current fixed threshold is adjusted accordingly based on different chromosome karyotype images.

[0075] Figure 4 This is a flowchart for determining the target binarization threshold provided by the present invention.

[0076] Figure 5 This is a schematic diagram illustrating the principle of determining the target binarization threshold provided by the present invention.

[0077] (5) Due to the use of a fixed threshold T i right When performing binarization, T may occur. i and The inability to adapt to different images, which affects edge detection performance, is mainly because the backgrounds of different images vary, and a single fixed threshold cannot adapt to all images. Therefore, dynamic adjustment of T is necessary. i Repeat step (2) for The binarization process is performed, and the edge detection in step (3) and the extraction of the effective chromosome contour set S′ in step (4) are repeated. valid In dynamically adjusting T i Generate S′ valid During the process, the T corresponding to the maximum value of cnt i Let it be T best Finally, use Tbe st right Perform binarization processing and execute steps (3) and (4) to generate S′. valid As the final effective set of chromosome contours S valid In this embodiment, T is dynamically adjusted.i And generate S′ valid Determine T best The principle diagram is as follows Figure 4 As shown, the final T best =137, the specific steps are described below:

[0078] Use T i right Perform binarization processing, execute steps (3) and (4) to generate T i Corresponding S′ valid And record the corresponding cnt, where i starts from 0, T i The calculation formula is:

[0079] T i =T i-1 +1, T0 = T min =100, T 130 =T max =230, i=0,1,2,3,…,130

[0080] Select the maximum value among all cnt values, and its corresponding T. i As T best .

[0081] S26, based on the number of effective chromosome contours, construct a set of effective chromosome contours, and convert the data format of the effective contours into the data format of the coordinate point key information of the chromosome karyotype image contour;

[0082] S3 uses annotation tools to manually annotate and complete unannotated chromosomes.

[0083] Figure 6 This is a flowchart of the automatic construction and generation of annotation files provided by the present invention.

[0084] Figure 7 This is a diagram showing the effect of semi-automatic annotation of chromosome images provided by the present invention.

[0085] Open the annotation file using the annotation tool to check the automatic annotation effect;

[0086] Based on the automatic annotation effect, the unannotated chromosomes are annotated and completed.

[0087] The specific content of the manual annotations includes the annotation name.

[0088] The automatic annotation method for chromosome karyotype images used in this invention cannot guarantee that all chromosome instances are annotated. Therefore, it is necessary to generate the annotation file F. label Then, use F tool Annotation tool open F labelThe files are manually checked for automatic annotation, and unannotated chromosome instances are manually annotated and completed. The average time to annotate a chromosome karyotype image using the method of this invention is only 5 minutes.

[0089] For those skilled in the art, various modifications and improvements can be made to the embodiments of the present invention without departing from the inventive concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A semi-automatic annotation method for chromosome karyotype image data, characterized in that, include: S1 uses image processing methods to obtain the key information required for automatic annotation; S2 constructs and generates a labeling file based on the key information according to the specified data format, and extracts the coordinate point information of the effective outline of the chromosome required for automatic labeling; S21 uses a filtering algorithm to reduce noise in the chromosome karyotype image; S22 performs binarization processing on the chromosome karyotype image using a corresponding fixed threshold; S23 uses a detection algorithm to perform edge detection on the chromosome karyotype image after the binarization algorithm; S24 extracts and determines whether each edge contour belongs to the normal chromosome range. If yes, continue to S25; otherwise, discard the edge contours that do not belong to the normal chromosome range. S25 counts the number of valid chromosome contours extracted and determines whether the current fixed threshold is the maximum value of valid contours when the number of valid chromosome contours extracted reaches the set number. If so, it is the target binarization threshold. The coordinate point information of all valid chromosome contours under the target binarization threshold is extracted and stored. If not, the current fixed threshold is adjusted and the process returns to S22. S26, based on the number of effective chromosome outlines, construct a set of effective chromosome outline numbers, and... The effective contour data format conversion transforms the data format of the key information of the coordinate points of the chromosome karyotype image contour; S3 uses annotation tools to manually annotate and complete unannotated chromosomes; S24 specifically includes: detecting each contour. circumference and internal area area Does it fall within the range of normal chromosomes? and If they all fall within the range of normal chromosomes, then For valid chromosome profiles, add them to the valid chromosome profile set. Simultaneously update the total number of valid outlines. ;otherwise If the chromosome outline is invalid, return to adjusting the current fixed threshold; the perimeter and internal region area ranges of normal chromosomes are as follows: ； in, This represents the set of all suspicious chromosome outlines obtained using an edge detection algorithm; In S25, the current fixed threshold is adjusted. The specific method is as follows: 。 2. The semi-automatic annotation method according to claim 1, characterized in that, The key information includes the size of the chromosome karyotype image, the absolute path of the chromosome karyotype image, the JSON string data of the chromosome karyotype image, the label name of the chromosome karyotype image, and the coordinate points of the outline of the chromosome karyotype image.

3. The semi-automatic annotation method according to claim 1, characterized in that, S1 includes: S11 obtains the chromosome karyotype image size and the absolute path of the chromosome karyotype image based on the image reading method; S12 obtains the raw bytecode data of the image based on the binary image reading method. The original bytecode data is encoded into base64 bytecode data using a data encoding method. The base64 bytecode data is decoded into UTF-8 format string data using a data decoding method. Obtain the JSON string data of the chromosome karyotype image; S13 is based on a deep learning data training scenario, obtains data labels, and uses a custom annotation file to annotate the label names of different chromosomes based on the data labels, thereby obtaining the chromosome karyotype image annotation names and annotation information.

4. The semi-automatic annotation method according to claim 1, characterized in that, The acquisition of the specified data format in S2 includes: The chromosome karyotype images were manually annotated using annotation tools to generate the annotation file; Read the annotation file to obtain the specified data format.

5. The semi-automatic annotation method according to claim 1, characterized in that, The coordinates of the effective chromosome contour in S25 are obtained by dimensionality reduction of the three-dimensional data of the effective chromosome contour to generate two-dimensional data.

6. The semi-automatic annotation method according to claim 1, characterized in that, The current fixed threshold in S25 is adjusted accordingly based on different chromosome karyotype images.

7. The semi-automatic annotation method according to claim 1, characterized in that, The specific content of the manual annotation mentioned in S3 includes the annotation name.

8. The semi-automatic annotation method according to claim 1, characterized in that, The S3 includes: Open the annotation file using an annotation tool to check the automatic annotation effect; Based on the automatic annotation effect, the unannotated chromosomes are annotated and completed.

Images