Chromosome recognition method, device, equipment and storage medium based on deep learning

Abstract

The present application relates to a chromosome recognition method, device, computer equipment and storage medium based on deep learning. The method includes: binarizing the chromosome image to be identified to obtain a binary image of the chromosome; orthogonally projecting the binary image to determine the bending center of the chromosome, and cutting the chromosome in the chromosome image from the bending center; straightening the two parts of the chromosome obtained by cutting and splicing them according to the cut; determining a sample image based on the chromosome image to be identified and the image obtained after splicing; performing deep learning training based on the sample image to obtain a chromosome classification model; identifying the category of the chromosome in the chromosome image to be identified through the chromosome classification model. This solution can save costs.

Description

Technical Field

[0001] The present application relates to the fields of computer technology and image processing technology, and in particular to a chromosome recognition method, device, equipment and storage medium based on deep learning. Background Art

[0002] With the development of science and technology, various advanced analysis technologies emerge in an endless stream. Among them, the analysis of chromosome images plays a very important role in many fields (for example, the medical field or the field of genetics research).

[0003] At present, the analysis of chromosome images is generally done manually, with professionals (such as doctors or professional analysts) observing the chromosome images under a high-power microscope, manually segmenting the chromosomes in the image, and then manually analyzing the segmented chromosomes, such as manually classifying the chromosomes in the image. However, this manual analysis of chromosome images by professionals results in cumbersome operations and requires repeated interaction with the machine, resulting in high time costs and resource interaction costs. Therefore, the traditional method has the problem of high costs. Summary of the invention

[0004] Based on this, it is necessary to provide a cost-saving deep learning-based chromosome identification method, device, computer equipment and storage medium to address the above technical problems.

[0005] A chromosome recognition method based on deep learning, the method comprising:

[0006] Binarization processing is performed on the chromosome image to be identified to obtain a binary image of the chromosome;

[0007] Performing orthogonal projection on the binary image to determine the bending center of the chromosome, and segmenting the chromosome in the chromosome image from the bending center;

[0008] The two parts of the chromosome obtained by cutting are straightened and then spliced ​​according to the cut;

[0009] Determining a sample image according to the chromosome image to be identified and the image obtained after splicing;

[0010] Performing deep learning training according to the sample images to obtain a chromosome classification model;

[0011] The chromosome classification model is used to identify the category of the chromosome in the chromosome image to be identified.

[0012] In one embodiment, the step of performing binarization processing on the chromosome image to be identified to obtain a binary image of the chromosome includes:

[0013] Obtain standard chromosome images;

[0014] According to the size of the standard chromosome image, scaling the chromosome image to be identified;

[0015] The chromosome image to be identified after the scaling process is binarized to obtain a binary image of the chromosome.

[0016] In one embodiment, performing orthogonal projection on the binary image to determine the bending center of the chromosome includes:

[0017] Step by step, the binary image is rotated, and the binary image rotated to each angle is horizontally projected and vertically projected respectively;

[0018] According to the vertical projection vector, the binary image is rotated to a target angle; the target angle is the angle at which the chromosomes in the binary image are rotated to be in an upright state;

[0019] The bending center of the chromosome is determined according to the horizontal projection vector when the binary image is rotated to the target angle.

[0020] In one embodiment, determining the bending center of the chromosome according to the horizontal projection vector when the binary image is rotated to the target angle includes:

[0021] Determining two local maxima from the horizontal projection vector when the binary image is rotated to the target angle;

[0022] The bending center of the chromosome is determined according to the global minimum value in the region between the two local maximum values.

[0023] In one embodiment, determining the sample image according to the chromosome image to be identified and the image obtained after splicing includes:

[0024] The chromosome image to be identified and the image obtained after splicing are used as images to be processed, image preprocessing is performed on the image to be processed, and pixels in the image obtained after preprocessing are normalized to obtain a sample image;

[0025] The image preprocessing includes at least one of the following:

[0026] Filling the image to be processed into a preset shape and size using pixel points;

[0027] Performing image enhancement on the image to be processed by histogram equalization processing;

[0028] Smoothing the image to be processed by mean filtering;

[0029] Perform image amplification processing on the image to be processed.

[0030] In one embodiment, performing deep learning training according to the sample image to obtain a chromosome classification model includes:

[0031] Obtain a classification model to be trained; the classification model includes a shallow convolutional network and a deep convolutional network; the deep convolutional network includes at least two convolutional modules with different structures, each convolutional module replaces a second convolutional kernel by cascading multiple first convolutional kernels, and the first convolutional kernel is smaller than the second convolutional kernel;

[0032] The sample image is input into the classification model to be trained in this round, and is sequentially passed through the shallow convolutional network and the deep convolutional network for feature extraction;

[0033] Predicting the category of the chromosome in the sample image based on the extracted features to obtain a prediction result;

[0034] The loss value of this round is determined according to the prediction result, and the model parameters of the classification model to be trained in this round are adjusted according to the loss value, and the next round is taken as this round, and the step of inputting the sample image into the classification model to be trained in this round is returned to perform iterative training until the iteration stops and the chromosome classification model is obtained.

[0035] In one of the embodiments, the loss value is a focus loss value of difficult-to-classify samples based on cross entropy;

[0036] The method further comprises:

[0037] According to the focus loss value of the difficult-to-classify samples in this round, the weight of the easy-to-classify sample images in the next round of training is reduced, and the weight of the difficult-to-classify sample images is increased;

[0038] The step of taking the next round as the current round and returning to execute the step of inputting the sample image into the classification model to be trained in the current round includes:

[0039] The next round is taken as the current round, and the sample image after weight adjustment is taken as the sample image to be input, and the step of inputting the sample image into the classification model to be trained in the current round and the subsequent training steps are performed.

[0040] A chromosome recognition device based on deep learning, the device comprising:

[0041] A binarization module is used to perform binarization processing on the chromosome image to be identified to obtain a binary image of the chromosome;

[0042] A segmentation module, used for performing orthogonal projection on the binary image, determining the bending center of the chromosome, and segmenting the chromosome in the chromosome image from the bending center;

[0043] A splicing module is used to straighten the two parts of the chromosome obtained by segmentation and then splice them according to the cut;

[0044] A model training module is used to determine a sample image based on the chromosome image to be identified and the image obtained after splicing; perform deep learning training based on the sample image to obtain a chromosome classification model;

[0045] The identification module is used to identify the category of the chromosome in the chromosome image to be identified through the chromosome classification model.

[0046] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the chromosome identification method based on deep learning in each embodiment of the present application when executing the computer program.

[0047] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps of a chromosome identification method based on deep learning in each embodiment of the present application are implemented.

[0048] The above-mentioned chromosome recognition method, device, computer equipment and storage medium based on deep learning perform binarization processing on the chromosome image to be identified, realize automatic segmentation of the binary image of the chromosome, and avoid tedious manual segmentation operations. Orthogonal projection is performed on the binary image to determine the bending center of the chromosome, and the chromosome in the chromosome image is cut from the bending center; the two parts of the chromosome obtained by cutting are straightened and spliced ​​according to the incision, which can automatically and conveniently expand the straightened chromosome. Therefore, according to the chromosome image to be identified and the image obtained after splicing, a sample image is determined to expand the sample. Furthermore, deep learning training is performed based on the expanded sample image, and a chromosome classification model with high classification accuracy can be obtained, so that the chromosome category in the chromosome image to be identified can be automatically and accurately identified through the chromosome classification model. Compared with the tedious manual operation of the traditional method, time cost and resource interaction cost are saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is an application environment diagram of a chromosome recognition method based on deep learning in one embodiment;

[0050] Figure 2 Schematic diagram of a flow chart of a chromosome recognition method based on deep learning in one embodiment;

[0051] Figure 3is a schematic diagram of a chromosome image in a bent state in one embodiment;

[0052] Figure 4 is a schematic diagram of a chromosome image after straightening in one embodiment;

[0053] Figure 5 A schematic diagram of the structure of a chromosome classification model in one embodiment;

[0054] Figure 6 is a structural block diagram of a chromosome recognition device based on deep learning in one embodiment;

[0055] Figure 7 is a structural block diagram of a chromosome recognition device based on deep learning in another embodiment;

[0056] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0057] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The chromosome recognition method based on deep learning provided in this application can be applied to Figure 1 In the application environment shown, the computer device 102 can obtain the chromosome image 104 to be identified, and identify the type of chromosome in the chromosome image 104 by executing the chromosome identification method based on deep learning in each embodiment of the present application. The computer device 102 can be a terminal or a server. Figure 1 The terminal is used as an example for illustration. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0060] In one embodiment, Figure 2 As shown, a chromosome recognition method based on deep learning is provided. This embodiment uses the method to be applied to a computer device as an example, and the computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0061] Step 202: binarize the chromosome image to be identified to obtain a binary image of the chromosome.

[0062] The chromosome image to be identified is an image of the chromosome category to be identified. It can be understood that the chromosome image to be identified includes chromosomes. The binary image of chromosomes is the segmented image of chromosomes. In the binary image of chromosomes, chromosomes are foreground areas, and non-chromosome areas in the image are background areas.

[0063] In one embodiment, the chromosome image to be identified can be the original chromosome image, or it can be the chromosome image after the original chromosome image is subjected to size standardization. The original chromosome image is the unprocessed original image. Size standardization refers to standardizing the size of the original chromosome image so that the size of the processed chromosome image meets the standard requirements. It can be understood that the chromosome image to be identified can also be a chromosome image after other preprocessing (such as image enhancement, denoising, etc.) on the basis of the original chromosome image, and is not limited to size standardization.

[0064] Specifically, the computer device can obtain the chromosome image to be identified and perform binary segmentation processing on the chromosome image to be identified, that is, the computer device can calculate the segmentation threshold, take the pixel value greater than the segmentation threshold as the foreground area, and take the pixel value less than the segmentation threshold as the background area, thereby segmenting the chromosome from the chromosome image (the chromosome is the foreground area) and obtaining a binary image of the chromosome.

[0065] In one embodiment, there may be multiple chromosome images to be identified, and multiple means at least two. The computer device may perform steps 202 to 208 for each chromosome image.

[0066] Step 204, orthogonally project the binary image, determine the bending center of the chromosome, and segment the chromosome in the chromosome image from the bending center.

[0067] The orthogonal projection refers to projecting the binary image in two directions that are perpendicular to each other.

[0068] It can be understood that for curved chromosomes, the computer device can execute steps 204 to 206 to expand the chromosomes after the curved chromosomes are straightened. For chromosomes that are not curved (i.e., the chromosomes themselves are straight in shape), the straightening process of steps 204 to 206 can be omitted, and step 208 can be directly executed based on the chromosome image, that is, the chromosome image to be identified can be directly used as a sample image.

[0069] Specifically, the computer device can rotate the binary image step by step, and during the rotation process, perform orthogonal projection on the binary image. The computer device can determine the bending center of the chromosome based on the projection vector obtained by the orthogonal projection. After determining the bending center of the chromosome, the computer device can cut the chromosome in the chromosome image from the bending center to obtain two parts of the cut chromosome.

[0070] In one embodiment, the computer device can perform horizontal projection and vertical projection on the binary image respectively, and determine the bending center of the chromosome according to the horizontal projection vector and the vertical projection vector. It can be understood that in other embodiments, the computer device can also perform projection on the binary image in other directions, as long as the projection directions are orthogonal (or vertical) to each other.

[0071] Step 206, the two parts of the chromosome obtained by cutting are straightened and then spliced ​​according to the cut.

[0072] The straightening process is used to rotate the two parts of the chromosome to a state where they can be vertically connected to each other. It can be understood that after the two parts of the straightened chromosome are spliced ​​according to the incision, a chromosome with a straightened shape can be obtained.

[0073] Specifically, the computer device can perform a straightening process on the two parts of the chromosome obtained by cutting, and the computer device can splice the two parts of the chromosome after straightening according to the cut to obtain a chromosome image after morphological straightening. That is, the morphology of the chromosome in the spliced ​​chromosome image is the straightened morphology.

[0074] In one embodiment, the computer device can rotate the two parts of the chromosome obtained by segmentation so that the directions or trends of the two parts of the chromosome are consistent. It can be understood that the chromosomes with the same directions or trends are in a state of being able to be vertically connected to each other, that is, the rotation process is achieved.

[0075] For ease of understanding, let's take an example. For example, if both parts of the chromosome are rotated to an angle of 30 degrees with the horizontal direction, then the two parts of the chromosome can be spliced ​​along the cut to obtain a straight chromosome. For another example, if both parts of the chromosome are rotated to an angle of 90 degrees with the horizontal direction (i.e. in an upright state), then the two parts of the chromosome can be spliced ​​along the cut to obtain a straight chromosome. As long as the direction or direction of the two parts of the chromosome are consistent, it is not limited to which direction they are rotated.

[0076] In one embodiment, the computer device can rotate the two chromosome parts obtained by cutting to an upright state (i.e., realize the rotation processing of the two chromosome parts obtained by cutting). It can be understood that it is not limited here that the two chromosome parts obtained by cutting must be rotated to an upright state, and can also be rotated to a horizontal state or other directions at the same time.

[0077] For ease of understanding, now combined Figure 3 and Figure 4 Provide a schematic illustration. Figure 3 FIG. 3 is a chromosome image in a bent state in an embodiment. By cutting it according to the bending center 302, two chromosome parts 304 and 306 can be obtained. After the two chromosome parts are straightened, they are spliced ​​according to the cuts, and the following can be obtained: Figure 4 The straightened chromosome image is shown. It can be understood that Figure 4 Both chromosome parts 304 and 306 are in an upright state, that is, they have been turned straight, and a straight chromosome can be obtained by directly splicing them along the incision.

[0078] Step 208, determining a sample image according to the chromosome image to be identified and the image obtained after splicing.

[0079] Among them, the image obtained after splicing is the spliced ​​chromosome image.

[0080] It can be understood that the two orthogonal projection vectors of the binary image of the chromosome contain all the morphological information of the chromosome, and therefore, can be used to achieve chromosome straightening, thereby obtaining a straightened chromosome image, and then data expansion can be achieved based on the straightened chromosome image (i.e., the image obtained after splicing), that is, adding and expanding the chromosome image. The computer device can determine the sample image based on the chromosome image to be identified and the image obtained after splicing, that is, it realizes the effective expansion of the sample data.

[0081] Specifically, the computer device can determine the sample image based on the chromosome image to be identified and the image obtained after splicing.

[0082] In one embodiment, the computer device may directly use the chromosome image to be identified and the image obtained after splicing as sample images.

[0083] In one embodiment, the computer device may also perform at least one preprocessing on the chromosome image to be identified and the image obtained after splicing, and determine the sample image based on the chromosome image to be identified, the image obtained after splicing and the preprocessed image.

[0084] In one embodiment, the preprocessing may include at least one of image enhancement and image amplification. For example, the computer device may perform image amplification processing on the chromosome image to be identified and the image obtained after splicing, or perform image enhancement on the chromosome image to be identified and the image obtained after splicing, or perform image enhancement on the chromosome image to be identified and the image obtained after splicing, and then perform image amplification processing. There is no limitation on this.

[0085] Step 210: Perform deep learning training based on the sample image to obtain a chromosome classification model.

[0086] Among them, the chromosome classification model is a deep learning model used to identify chromosome categories.

[0087] Specifically, the computer device can input the sample image into the classification model to be trained, and iteratively perform deep learning training to obtain a chromosome classification model.

[0088] In one embodiment, the computer device may first determine the total sample images, and then iteratively perform deep learning training based on the determined sample images. In other embodiments, different sample images may be used for each round of training, that is, the computer device may perform at least one preprocessing on the known sample images in each round of iterative training to generate new sample images, and then perform the current round of iterative training based on the new sample images and the known sample images.

[0089] Step 212: Identify the category of the chromosome in the chromosome image to be identified by using the chromosome classification model.

[0090] Specifically, after the chromosome classification model is trained, the computer device can input the chromosome image to be identified into the chromosome classification model to identify the category of the chromosome in the chromosome image.

[0091] The above-mentioned chromosome recognition method based on deep learning performs binarization processing on the chromosome image to be identified, realizes automatic segmentation of the binary image of the chromosome, and avoids the tedious manual segmentation operation. Orthogonal projection is performed on the binary image to determine the bending center of the chromosome, and the chromosome in the chromosome image is cut from the bending center; the two parts of the chromosome obtained by cutting are straightened and spliced ​​according to the incision, which can automatically and conveniently expand the straightened chromosome. Therefore, according to the chromosome image to be identified and the image obtained after splicing, the sample image is determined to expand the sample. Furthermore, deep learning training is performed based on the expanded sample image, and a chromosome classification model with high classification accuracy can be obtained, so that the chromosome category in the chromosome image to be identified can be automatically and accurately identified through the chromosome classification model. Compared with the tedious manual operation of the traditional method, it improves efficiency and saves time cost and resource interaction cost.

[0092] In addition, compared with traditional manual analysis which is too subjective and easily affected by external factors, resulting in low analysis accuracy, automatically identifying the category of chromosomes in chromosome images through chromosome classification models is more objective and improves recognition accuracy.

[0093] Then, by expanding the samples by expanding the straightened chromosomes, the accuracy of the chromosome classification model can be improved, and the accuracy of chromosome classification and recognition can be further improved.

[0094] In one embodiment, step 202 performs binarization processing on the chromosome image to be identified to obtain a binary image of the chromosome, including: obtaining a standard chromosome image; scaling the chromosome image to be identified according to the size of the standard chromosome image; and binarizing the scaled chromosome image to be identified to obtain a binary image of the chromosome.

[0095] The standard chromosome image is a chromosome image used as a reference standard. It can be understood that the chromosomes in the standard chromosome image have the characteristics of complete division and clarity.

[0096] Specifically, the computer device can obtain a standard chromosome image; and scale the chromosome image to be identified according to the size of the standard chromosome image. That is, the size of the chromosome image to be identified after scaling is consistent with the size of the standard chromosome image. The computer device can perform binarization on the chromosome image to be identified after scaling to obtain a binary image of the chromosome. It can be understood that the size consistency can be the same size or the size meets a close condition (i.e., the size is close).

[0097] In one embodiment, the dimensions of the standard chromosome image may include length and width dimensions.

[0098] In one embodiment, the computer device may use the length and width of a standard chromosome image as a standard, and all chromosome images to be identified are scaled according to the standard.

[0099] In one embodiment, the computer device may determine the zoom factor according to the aspect ratio between the standard chromosome image and each chromosome image to be identified, and then perform zoom processing on the chromosome image to be identified according to the zoom factor.

[0100] In one embodiment, the computer device may determine the zoom factor a according to the following formula:

[0101]

[0102] Among them, H 标 and W 标 are the length and width of the standard chromosome image, H 原and W 原 are the length and width of the chromosome image to be identified, and abs(x) is the rounding function.

[0103] In one embodiment, the standard chromosome image can be a chromosome image of metaphase mitosis that is well developed and has clear pixels. Specifically, the user can select an image with good development and appropriate pixels from all chromosome images of metaphase mitosis based on the computer device as the standard chromosome image. The computer device can then obtain the standard chromosome image selected by the user. In other embodiments, the computer device can also automatically perform chromosome integrity detection and pixel clarity detection on the chromosome image of metaphase mitosis, thereby automatically selecting a standard chromosome image from the chromosome image of metaphase mitosis. This is not limited.

[0104] It can be understood that there may be multiple chromosome images to be identified, and the sizes of the multiple chromosome images are very likely to be inconsistent. Therefore, the chromosome image to be identified can be scaled according to the size of the standard chromosome image, eliminating the difference in chromosome size caused by different image sources, thereby improving the accuracy of subsequent processing and reducing unnecessary waste of system processing resources due to size inconsistency.

[0105] In addition, since the chromosomes in the standard chromosome image are relatively complete and clear, the chromosomes under the standard size are easier to process. Therefore, the chromosome image to be identified is scaled to the size of the standard chromosome image, so that the processing based on the standard size is more accurate.

[0106] In one embodiment, the step of performing orthogonal projection on the binary image in step 204 to determine the bending center of the chromosome includes: gradually rotating the binary image, and performing horizontal projection and vertical projection on the binary image rotated to each angle respectively; rotating the binary image to the target angle according to the vertical projection vector; and determining the bending center of the chromosome according to the horizontal projection vector when the binary image is rotated to the target angle.

[0107] The target angle is the angle of rotation when the chromosome in the binary image is in an upright state. The upright state means that the chromosome is in a vertical state and not tilted.

[0108] Horizontal projection refers to projecting a two-dimensional image in rows toward the y-axis, which is equivalent to projecting the two-dimensional image with horizontal light. Vertical projection refers to projecting a two-dimensional image in columns toward the x-axis, which is equivalent to projecting the two-dimensional image with vertical light.

[0109] Specifically, the computer device can perform a plane rotation of the binary image step by step within the range of 0° to 180° according to a preset angle interval, and perform horizontal projection and vertical projection on the binary image rotated to each angle. It can be understood that by performing horizontal projection and vertical projection on the binary image, a horizontal projection vector and a vertical projection vector can be obtained.

[0110] In order to facilitate understanding of the relationship between step-by-step rotation and projection, an example is given. Assuming that the preset angle interval is 5°, the computer device can rotate the binary image every 5°. For example, the first rotation rotates the binary image by 5°, and the binary image rotated by 5° in total is horizontally projected and vertically projected. If the target angle is not reached after rotating by 5°, the binary image is rotated by another 5°, that is, the total rotation angle is 10°, and then the binary image rotated by 10° in total is horizontally projected and vertically projected. By analogy, the binary image is rotated step by step, and the binary image rotated to each angle is horizontally projected and vertically projected respectively.

[0111] It can be understood that the computer device will determine whether the binary image is rotated to the target angle based on the vertical projection vector of the binary image at each angle, that is, whether it is rotated to the angle that makes the chromosomes in the binary image in an upright state. If it is not rotated to the target angle, the rotation continues at a preset interval until the binary image is rotated to the target angle (that is, the chromosomes in the binary image are rotated to an upright state) according to the target vertical projection vector. The computer device can calculate the horizontal projection vector obtained by horizontally projecting the binary image when it is rotated to the target angle, and determine the bending center of the chromosome based on the horizontal projection vector. That is, based on the horizontal projection vector of the chromosome in an upright state, the narrowest point is determined, and the narrowest point is used as the bending center of the chromosome.

[0112] In the above embodiment, by gradually rotating the binary image, and performing horizontal projection and vertical projection on the binary image rotated to each angle, the bending center of the chromosome can be accurately and conveniently determined according to the vertical projection vector and the horizontal projection vector, so that the chromosome can be straightened quickly and accurately.

[0113] In one embodiment, the bending center of the chromosome is determined based on the horizontal projection vector when the binary image is rotated to the target angle, including: determining two local maxima from the horizontal projection vector when the binary image is rotated to the target angle; and determining the bending center of the chromosome based on the global minimum in the area between the two local maxima.

[0114] It can be understood that for a curved chromosome, the head and tail ends of the chromosome are the thickest, and the narrowest place between the head and tail ends is the bending center of the chromosome. Therefore, the computer device can determine two local maxima from the horizontal projection vector when the binary image is rotated to the target angle. The two local maxima are the thickest ends of the chromosome. The computer device can determine the global minimum from the area between the two local maxima, and the global minimum is the narrowest place with the least pixels, so the global minimum can be determined as the bending center of the chromosome.

[0115] In the above embodiment, the global minimum value in the region between two local maximum values ​​is determined from the horizontal projection vector of the chromosome in the upright state, so that the bending center of the chromosome can be accurately and conveniently determined. Thus, the chromosome can be straightened quickly and accurately, thereby realizing the rapid and effective expansion of the chromosome image.

[0116] In one embodiment, determining a sample image according to the chromosome image to be identified and the image obtained after splicing includes: using the chromosome image to be identified and the image obtained after splicing as images to be processed, and performing image preprocessing on the images to be processed.

[0117] It can be understood that the computer device can perform image preprocessing on the chromosome image to be identified and the image obtained after splicing of the chromosome obtained in step 202.

[0118] In one embodiment, the image preprocessing that the computer device can perform on the image to be processed may include at least one of the following: filling the image to be processed with a preset shape and size using pixel points; performing image enhancement on the image to be processed through histogram equalization; smoothing the image to be processed through mean filtering; and performing image amplification on the image to be processed.

[0119] It can be understood that the preprocessing of smoothing the image to be processed by mean filtering is used to weaken or eliminate the high-frequency components in the image to be processed without affecting the low-frequency components in the image to be processed.

[0120] In one embodiment, the specific step of preprocessing by filling the image to be processed with a preset shape and size using pixel points may include: filling the image to be processed (i.e., the binary image of a single chromosome) with a pixel value of 255 into a preset shape, and adjusting the image size to a preset size. In one embodiment, the preset shape may be a square or a rectangle or other shapes, and the preset size may be 200*200 or other sizes, which are not limited. It can be understood that by filling in pixels, the image specifications can be unified, the convenience and accuracy of subsequent processing can be improved, and the loss of processing resources can be reduced.

[0121] It can be understood that histogram equalization processing, filtering low-frequency components in the image to be processed by sharpening filtering, and smoothing the image to be processed by mean filtering all belong to image enhancement processing.

[0122] In one embodiment, image amplification processing of the image to be processed may include at least one of inversion, translation, scaling, brightness change, cropping, changing illumination, color conversion, blurring, grayscale conversion, generating random size masks with random probability, simulating chromosome folding and occlusion phenomenon, increasing data difficulty, etc.

[0123] In one embodiment, the image preprocessing may further include: filtering low-frequency components in the image to be processed by sharpening filtering, that is, reducing or eliminating low-frequency components in the image to be processed without affecting high-frequency components in the image to be processed.

[0124] It is understood that the computer device can perform any one or more of the above preprocessing on the image to be processed. When the computer device needs to perform the above multiple preprocessing on the image to be processed, the processing result of the previous preprocessing can be used as the input of the next preprocessing, thereby realizing multiple preprocessing of the image to be processed. It should be noted that when multiple preprocessing needs to be performed, the execution order of each preprocessing is not limited.

[0125] In a specific embodiment, the computer device can use pixel points to fill the image to be processed into a preset shape and size, and then enhance the filled image through histogram equalization. Then, the image processed by the sharpening filter is smoothed by mean filtering, thereby achieving image enhancement processing. Next, the computer device can perform image amplification on the image enhanced by the above preprocessing. It can be understood that in other embodiments, the order of execution of the preprocessing in this embodiment can also be changed to form a new preprocessing scheme. In one embodiment, before smoothing the image processed by the sharpening filter by mean filtering, the low-frequency components in the image to be processed can also be filtered by sharpening filtering.

[0126] In the above embodiment, determining the sample image after image preprocessing can improve the validity of the sample image, thereby improving the detection accuracy of the chromosome classification model subsequently trained based on the sample image.

[0127] In one embodiment, the computer device may perform normalization processing on the pixels in the image obtained after preprocessing to obtain a sample image. The image obtained after preprocessing includes the image before preprocessing (i.e., the chromosome image to be identified and the image obtained after splicing) and the newly obtained image after preprocessing (for example, at least one of the image after pixel filling, the image after image enhancement and the image after image amplification, all of which belong to the newly obtained image after preprocessing).

[0128] In one embodiment, for each image obtained after preprocessing, the computer device can subtract the mean of all pixels in the image from each pixel in the image, and then divide it by the standard deviation, so as to normalize and normalize the image. This can ensure that the distributions of all images obtained after preprocessing are similar, so that it is easier to converge during deep learning training, that is, the difficulty of convergence is reduced, and then the system resource loss during deep learning training is reduced. In other embodiments, other normalization methods can also be used to normalize the pixels in the image obtained after preprocessing.

[0129] In one embodiment, deep learning training is performed based on sample images to obtain a chromosome classification model, including: obtaining a classification model to be trained; inputting the sample image into the classification model to be trained in this round, and extracting features through a shallow convolutional network and a deep convolutional network in turn; predicting the category of the chromosome in the sample image based on the extracted features to obtain a prediction result; determining a loss value of this round according to the prediction result, and adjusting the model parameters of the classification model to be trained in this round according to the loss value, and taking the next round as this round, returning to execute the step of inputting the sample image into the classification model to be trained in this round, so as to perform iterative training until the iteration stops and the chromosome classification model is obtained.

[0130] The classification model includes shallow convolutional networks and deep convolutional networks. Shallow convolutional networks are used for shallow preliminary feature extraction. Deep convolutional networks perform deeper feature extraction than shallow convolutional networks.

[0131] The deep convolutional network includes at least two convolutional modules with different structures. That is, different convolutional modules have different structures, so that the depth and width of the model can be expanded, so that deeper features can be extracted and overfitting can be avoided.

[0132] Each convolution module in the deep convolutional network replaces the second convolution kernel by cascading multiple first convolution kernels, and the first convolution kernel is smaller than the second convolution kernel. It can be understood that using smaller convolution kernels to replace large convolution kernels can reduce network parameters, thereby improving model training efficiency and reducing system resource loss during training. For example, replacing a 5x5 convolution kernel with two 3*3 convolution kernels, or replacing a 3x3 convolution kernel with two convolution kernels, 1x3 and 3x1, can use smaller kernels to reduce network parameters.

[0133] Specifically, the computer device can perform iterative deep learning training based on the sample image. In each round of iterative training, the computer device can input the sample image into the classification model to be trained in this round, and extract features through the shallow convolution network and the deep convolution network in turn; predict the category of the chromosome in the sample image based on the extracted features to obtain a prediction result. The computer device can determine the loss value of this round based on the difference between the prediction result and the labeled knot. The computer device can adjust the model parameters of the classification model to be trained in this round based on the loss value of this round, and use the next round as this round, and return to execute the step of inputting the sample image into the classification model to be trained in this round, so as to perform iterative training until the iteration stops and the chromosome classification model is obtained.

[0134] In one embodiment, the deep convolutional network may be an InceptionV3 network (i.e., a deep convolutional network developed by Google). The convolutional modules included in the deep convolutional network may be Inception modules of different structures. That is, the deep convolutional network in the embodiment of the present application is composed of a variety of different Inception modules combined with convolutional pooling.

[0135] In order to facilitate the understanding of the structure of the chromosome classification model, we now combine Figure 5 Refer to Figure 5 The chromosome classification model may include a shallow convolutional network 502 and a deep convolutional network 504, wherein the deep convolutional network 504 includes five different convolutional modules A to E (i.e., Inception modules), and each convolutional module includes multiple smaller convolutional kernels to reduce network parameters. Figure 5 Describe the training steps of each round of the chromosome classification model. First, the sample image is enlarged and then input into the shallow convolution network 502 in the chromosome classification model. After preliminary convolution through each convolution layer in 502, it is sequentially deep convolution through each convolution module in the deep convolution network 504. The extracted features are input into the fully connected layer to predict the chromosome category.

[0136] In the above embodiment, training based on the classification model architecture including shallow convolutional networks and deep convolutional networks can extract deep features, improve the accuracy of model training and the precision of the model. In addition, since the deep convolutional network includes convolutional modules with different structures, the depth and width of the model can be expanded, and deeper features can be extracted to avoid overfitting, further improving the precision of the chromosome classification model.

[0137] In one embodiment, the loss value is a focus loss value of difficult-to-classify samples based on cross entropy. The method also includes: according to the focus loss value of difficult-to-classify samples in this round, reducing the weight of easy-to-classify sample images in the next round of training and increasing the weight of difficult-to-classify sample images. In an embodiment of the present application, the next round is taken as the current round, and the step of inputting the sample image into the classification model to be trained in this round is returned, including: taking the next round as the current round, and taking the sample image after the weight adjustment as the sample image to be input, and executing the steps of inputting the sample image into the classification model to be trained in this round and subsequent training steps.

[0138] Among them, the difficult-to-classify sample focus loss value is used to focus on difficult-to-classify samples during training.

[0139] In one embodiment, the focus loss value of the difficult-to-classify sample based on the cross entropy may be a loss value calculated by a focal loss function (ie, a loss function obtained by modifying a standard cross entropy loss function to reduce the weight of easy-to-classify samples).

[0140] Specifically, in addition to adjusting the model parameters of the classification model to be trained in this round according to the focus loss value of the difficult-to-classify samples in this round, the computer device can also reduce the weight of the easy-to-classify sample images in the next round of training and increase the weight of the difficult-to-classify sample images according to the focus loss value of the difficult-to-classify samples in this round, so that the model training can focus more on the samples that are more difficult to learn without ignoring any samples. The computer device can then use the next round as the current round, and use the sample images after weight adjustment as the sample images to be input, and return to execute the sample images to be input into the classification model to be trained in this round and the subsequent training steps to perform a new round of model iterative training.

[0141] In one embodiment, the computer device may calculate the focus loss value of the difficult-to-classify sample according to the following formula:

[0142] CE(P t )=-log(P t );

[0143] FL(P t )=(1-P t ) α CE(P t ).

[0144] Among them, P t is the classification probability of chromosomes in the sample image; CE(P t ) is the cross entropy; (1-P t ) α is the adjustment factor, α is the adjustable focusing parameter; FL(P t ) is the focus loss value of difficult-to-classify samples.

[0145] In the above embodiment, based on the loss value focused on the difficult-to-classify samples, the model training can be more focused on the samples that are more difficult to learn without ignoring any samples, thereby improving the accuracy of the model training and thus improving the accuracy of the chromosome classification model.

[0146] It should be understood that, although each step in the flow chart of each embodiment of the present application is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps of the flow chart of each embodiment of the present application may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0147] like Figure 6 As shown, in one embodiment, a chromosome recognition device based on deep learning is provided, the device comprising: a binarization module 602, a segmentation module 604, a splicing module 606, a model training module 608 and a recognition module 610; wherein:

[0148] The binarization module 602 is used to perform binarization processing on the chromosome image to be identified to obtain a binary image of the chromosome.

[0149] The segmentation module 604 is used to perform orthogonal projection on the binary image, determine the bending center of the chromosome, and segment the chromosome in the chromosome image from the bending center.

[0150] The splicing module 606 is used to straighten the two parts of the chromosome obtained by segmentation and then splice them according to the incision.

[0151] The model training module 608 is used to determine a sample image based on the chromosome image to be identified and the image obtained after splicing; and perform deep learning training based on the sample image to obtain a chromosome classification model.

[0152] The identification module 610 is used to input the chromosome image to be identified into the chromosome classification model to identify the category of the chromosome in the chromosome image.

[0153] In one embodiment, the binarization module 602 is also used to obtain a standard chromosome image; scale the chromosome image to be identified according to the size of the standard chromosome image; and binarize the scaled chromosome image to be identified to obtain a binary image of the chromosome.

[0154] like Figure 7 As shown, in one embodiment, the segmentation module 604 includes:

[0155] The projection module 604a is used to gradually rotate the binary image and perform horizontal and vertical projections on the binary image rotated to various angles; according to the vertical projection vector, the binary image is rotated to a target angle; the target angle is the angle at which the chromosomes in the binary image are rotated to an upright state.

[0156] The bending center determination module 604b is used to determine the bending center of the chromosome according to the horizontal projection vector when the binary image is rotated to the target angle.

[0157] In one embodiment, the bending center determination module 604b is further used to determine two local maxima from the horizontal projection vector when the binary image is rotated to the target angle; and determine the bending center of the chromosome according to the global minimum in the region between the two local maxima.

[0158] In one embodiment, the model training module 608 is further used to use the chromosome image to be identified and the image obtained after splicing as the image to be processed, perform image preprocessing on the image to be processed, and normalize the pixels in the image obtained after preprocessing to obtain a sample image. The image preprocessing includes at least one of the following: filling the image to be processed with a preset shape and size using pixel points; performing image enhancement on the image to be processed by histogram equalization; performing smoothing on the image to be processed by mean filtering; and performing image amplification on the image to be processed.

[0159] In one embodiment, the model training module 608 is also used to obtain a classification model to be trained; the classification model includes a shallow convolutional network and a deep convolutional network; the deep convolutional network includes at least two convolution modules with different structures, each convolution module replaces the second convolution kernel by cascading multiple first convolution kernels, and the first convolution kernel is smaller than the second convolution kernel; the sample image is input into the classification model to be trained in this round, and feature extraction is performed through the shallow convolutional network and the deep convolutional network in turn; the category of the chromosome in the sample image is predicted based on the extracted features to obtain a prediction result; the loss value of this round is determined according to the prediction result, and the model parameters of the classification model to be trained in this round are adjusted according to the loss value, and the next round is used as this round, and the step of inputting the sample image into the classification model to be trained in this round is returned to perform iterative training until the iteration stops to obtain the chromosome classification model.

[0160] In one embodiment, the loss value is a focus loss value of difficult-to-classify samples based on cross entropy. The model training module 608 is also used to reduce the weight of easy-to-classify sample images in the next round of training and increase the weight of difficult-to-classify sample images according to the focus loss value of the difficult-to-classify samples in this round; take the next round as the current round, and take the sample images after weight adjustment as the sample images to be input, and execute the steps of inputting the sample images into the classification model to be trained in this round and subsequent training.

[0161] For the specific definition of the chromosome identification device based on deep learning, please refer to the definition of the chromosome identification method based on deep learning above, which will not be repeated here. Each module in the above-mentioned chromosome identification device based on deep learning can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a chromosome recognition method based on deep learning is implemented.

[0163] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a chromosome recognition method based on deep learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0164] Those skilled in the art will understand that Figure 8 The structure shown in or 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0167] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0168] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A chromosome recognition method based on deep learning, characterized in that: The method comprises: Obtain standard chromosome images; According to the size of the standard chromosome image, scaling the chromosome image to be identified; Binarization is performed on the chromosome image to be identified after scaling to obtain a binary image of the chromosome; Performing orthogonal projection on the binary image to determine the bending center of the chromosome, and segmenting the chromosome in the chromosome image from the bending center; The two parts of the chromosome obtained by cutting are straightened and then spliced ​​according to the cut; The chromosome image to be identified and the image obtained after splicing are used as images to be processed, image preprocessing is performed on the image to be processed, and pixels in the image obtained after preprocessing are normalized to obtain a sample image; Obtain a classification model to be trained; the classification model includes a shallow convolutional network and a deep convolutional network; the deep convolutional network includes at least two convolutional modules with different structures, each convolutional module replaces a second convolutional kernel by cascading multiple first convolutional kernels, and the first convolutional kernel is smaller than the second convolutional kernel; The sample image is input into the classification model to be trained in this round, and is sequentially passed through the shallow convolutional network and the deep convolutional network for feature extraction; Predicting the category of the chromosome in the sample image based on the extracted features to obtain a prediction result; The loss value of this round is determined according to the prediction result, and the model parameters of the classification model to be trained in this round are adjusted according to the loss value, and the next round is taken as the current round, and the step of inputting the sample image into the classification model to be trained in this round is returned to perform iterative training until the iteration stops to obtain a chromosome classification model; through the chromosome classification model, the category of the chromosome in the chromosome image to be identified is identified.

2. The method according to claim 1, characterized in that: The performing orthogonal projection on the binary image to determine the bending center of the chromosome comprises: Step by step, the binary image is rotated, and the binary image rotated to each angle is horizontally projected and vertically projected respectively; According to the vertical projection vector, the binary image is rotated to a target angle; the target angle is the angle at which the chromosomes in the binary image are rotated to be in an upright state; The bending center of the chromosome is determined according to the horizontal projection vector when the binary image is rotated to the target angle.

3. The method according to claim 2, characterized in that Determining the bending center of the chromosome according to the horizontal projection vector when the binary image is rotated to the target angle includes: Determining two local maxima from the horizontal projection vector when the binary image is rotated to the target angle; The bending center of the chromosome is determined according to the global minimum value in the region between the two local maximum values.

4. The method according to claim 1, characterized in that: The image preprocessing includes at least one of the following: Filling the image to be processed into a preset shape and size using pixel points; Performing image enhancement on the image to be processed by histogram equalization processing; Smoothing the image to be processed by mean filtering; Perform image amplification processing on the image to be processed.

5. The method according to claim 1, characterized in that The method further comprises: For each image obtained after preprocessing, a mean value of all pixels in the preprocessed image is subtracted from each pixel in the preprocessed image, and the result is divided by the standard deviation.

6. The method according to claim 1, characterized in that The step of scaling the chromosome image to be identified according to the size of the standard chromosome image comprises: Determining a zoom factor according to the aspect ratio between the standard chromosome image and each of the chromosome images to be identified; The chromosome image to be identified is scaled according to the scaling factor.

7. The method according to claim 6, characterized in that The loss value is a focus loss value of difficult-to-classify samples based on cross entropy.

8. A chromosome recognition device based on deep learning, characterized in that: The device comprises: A binarization module is used to obtain a standard chromosome image; scale the chromosome image to be identified according to the size of the standard chromosome image; and binarize the scaled chromosome image to be identified to obtain a binary image of the chromosome; A segmentation module, used for performing orthogonal projection on the binary image, determining the bending center of the chromosome, and segmenting the chromosome in the chromosome image from the bending center; A splicing module is used to straighten the two parts of the chromosome obtained by segmentation and then splice them according to the cut; A model training module, used to use the chromosome image to be identified and the image obtained after splicing as the image to be processed, perform image preprocessing on the image to be processed, and normalize the pixels in the image obtained after preprocessing to obtain a sample image; obtain a classification model to be trained; the classification model includes a shallow convolutional network and a deep convolutional network, the deep convolutional network includes at least two convolutional modules with different structures, each convolutional module replaces a second convolutional kernel by cascading multiple first convolutional kernels, and the first convolutional kernel is smaller than the second convolutional kernel; The sample image is input into the classification model to be trained in this round, and feature extraction is performed through the shallow convolution network and the deep convolution network in sequence; the category of the chromosome in the sample image is predicted based on the extracted features to obtain a prediction result; the loss value of this round is determined according to the prediction result, and the model parameters of the classification model to be trained in this round are adjusted according to the loss value, and the next round is used as the current round, and the step of inputting the sample image into the classification model to be trained in this round is returned to perform iterative training until the iteration stops and a chromosome classification model is obtained; the category of the chromosome in the chromosome image to be identified is identified through the chromosome classification model; The identification module is used to identify the category of the chromosome in the chromosome image to be identified through the chromosome classification model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images