Chromosome Overlap Detection Method and Device Based on Cross-Modal Feature Fusion

Abstract

The present application provides a chromosome overlap detection method and device based on cross-modal feature fusion, including the following steps: obtaining a ten-fold microscope image to be measured, inputting the ten-fold microscope image to be measured into a chromosome region detection model to obtain a ten-fold microscope chromosome region, obtaining a corresponding hundred-fold microscope chromosome region based on the coordinates of the ten-fold microscope chromosome region, splicing the feature map of the ten-fold microscope chromosome region and the feature map of the corresponding hundred-fold microscope chromosome region to obtain a chromosome detection map, and inputting the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result. By splicing the feature map of the ten-fold microscope chromosome region and the feature map of the corresponding hundred-fold microscope chromosome region as the input image, the detection accuracy of short chromosomes is optimized in this solution.

Description

Technical Field

[0001] This application relates to the field of image detection, and particularly to a chromosome overlap detection method and device based on cross-modal feature fusion. Background Art

[0002] Chromosome karyotype analysis plays a key role in genetic science research and clinical auxiliary diagnosis. As an important auxiliary link before karyotype analysis, chromosome detection and overlap degree analysis are of great significance for subsequent accurate analysis. At present, chromosome overlap detection is mainly achieved by performing image processing on glass slide samples at the metaphase of chromosome mitosis. First, focus and shoot under a ten-fold lens to identify the presence and position of chromosomes, then take a hundred-fold lens image based on this position, and further enhance the hundred-fold image and classify the overlap degree. However, the existing automated chromosome karyotype analysis systems have the following problems:

[0003] 1. Low detection efficiency due to image quality problems: Image quality highly depends on the cleanliness of the glass slide, the clarity of focusing, and the exposure parameters. Bubbles, scratches, etc. on the glass slide will interfere with the image. Focusing errors are likely to make the image blurred or out of focus, and improper exposure parameters will cause overexposure or low light phenomena. Combining these factors, the obtained high-resolution image quality is poor, increasing the time cost of image detection;

[0004] 2. Limitations of traditional detection methods: During ten-fold lens detection, short chromosomes are difficult to distinguish from impurities due to their small pixel ratio, and traditional cropping methods will cause loss of key information; at the same time, physical operations such as dripping oil and smearing oil will cause the field of view coordinates to shift, further magnifying the error of subsequent magnification and recognition. In the hundred-fold lens screening link, the classification of overlapping chromosomes mainly relies on traditional image segmentation methods, such as thresholding and edge detection. These methods are insufficient in dealing with scenarios with sparse clusters and complex cross shapes, and it is difficult to accurately distinguish the overlap degree of chromosomes; moreover, background impurities (strip-shaped or circular interference) are easily confused with chromosome textures, and false detections are also likely to occur for curved chromosomes.

[0005] The above problems seriously affect the accuracy of chromosome karyotype analysis. Therefore, how to accurately locate the chromosome position and screen overlapping chromosomes under poor image quality is an urgent problem to be solved in the prior art. Summary of the Invention

[0006] The embodiments of this application provide a chromosome position method and device based on cross-modal feature fusion. By splicing the feature map of the chromosome region under the ten-fold lens and the feature map of the corresponding chromosome region under the hundred-fold lens as the input image, the detection accuracy of short chromosomes is optimized.

[0007] In a first aspect, an embodiment of the present application provides a chromosome overlap detection method based on cross-modal feature fusion, the method comprising:

[0008] A plurality of slide sample images are acquired as a first image set through a ten-fold mirror imaging system, and pixel brightness adjustment is performed on each slide sample image in the first image set to obtain a second image set, wherein the pixel brightness adjustment is performed by: denoising overexposed pixel points in the slide sample and reconstructing brightness of underexposed pixel points in the slide sample;

[0009] Automatically annotate the chromosomes of each slide sample image in the second image set to obtain a third image set, then balance the positive and negative samples in the third image set to obtain a fourth image set, and use the fourth image set to train a chromosome region detection model, wherein the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set;

[0010] Obtain a 10x image to be tested, input the 10x image to be tested into a chromosome region detection model to obtain a 10x chromosome region, obtain the corresponding 100x chromosome region based on the coordinates of the 10x chromosome region, splice a feature map of the 10x chromosome region with a feature map of the corresponding 100x chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0011] In a second aspect, an embodiment of the present application provides a chromosome overlap detection device based on cross-modal feature fusion, comprising:

[0012] A brightness adjustment module, used to obtain multiple slide sample images as a first image set through a ten-fold mirror imaging system, and to adjust the pixel brightness of each slide sample image in the first image set to obtain a second image set, wherein the pixel brightness adjustment includes: performing noise reduction on overexposed pixel points in the slide sample and performing brightness reconstruction on underexposed pixel points in the slide sample;

[0013] a labeling module, for automatically labeling chromosomes of each slide sample image in the second image set to obtain a third image set, then balancing the positive and negative samples in the third image set to obtain a fourth image set, and using the fourth image set to train a chromosome region detection model, wherein the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set;

[0014] A detection module is used to obtain a 10x microscope image to be tested, input the 10x microscope image to be tested into a chromosome region detection model to obtain a 10x microscope chromosome region, obtain the corresponding 100x microscope chromosome region based on the coordinates of the 10x microscope chromosome region, splice a feature map of the 10x microscope chromosome region with a feature map of the corresponding 100x microscope chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform a chromosome overlap detection method based on cross-modal feature fusion.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a chromosome overlap detection method based on cross-modal feature fusion.

[0017] The main contributions and innovations of the present invention are as follows:

[0018] The embodiment of the present application assigns a higher weight to short chromosomes in the chromosome region detection model, and identifies short chromosomes after improving the resolution, thereby effectively improving the detection accuracy of short chromosome regions and providing more accurate data for subsequent chromosome analysis; this solution cross-modally integrates the 10x microscope positioning information with the 100x microscope texture features, giving full play to the advantages of two lenses with different magnifications, so that short chromosomes can be more accurately identified and located during the detection process, further improving the overall detection performance; the embodiment of the present application adds overlap optimization loss as an additional loss function during the training process of the chromosome overlap detection model, thereby improving the detection effect of the chromosome overlap detection model on the chromosome overlap region and better detecting the chromosome position.

[0019] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 It is a flowchart of a chromosome overlap detection method based on cross-modal feature fusion according to an embodiment of the present application;

[0022] Figure 2 Schematic diagram of the detection head structure of a chromosome region detection model according to an embodiment of the present application;

[0023] Figure 3 Schematic diagram of separating the foreground region and the background region of a chromosome detection map according to an embodiment of the present application;

[0024] Figure 4 Block diagram of the structure of a chromosome overlap detection device based on cross-modal feature fusion according to an embodiment of the present application;

[0025] Figure 5 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Specific embodiments

[0026] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that: in other embodiments, the steps of the corresponding methods are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0028] Embodiment 1

[0029] The embodiment of the present application provides a chromosome overlap detection method based on cross-modal feature fusion. By splicing the feature map of the ten-fold chromosome region and the feature map of the corresponding hundred-fold chromosome region as the input image, the detection accuracy of short chromosomes is optimized, and by setting an overlap optimization loss as an additional loss function in the chromosome overlap detection model, the detection effect of the chromosome overlap detection model on the chromosome overlap region is improved. Specifically, referring to Figure 1 , the method includes:

[0030] A plurality of slide sample images are acquired as a first image set through a ten-fold mirror imaging system, and pixel brightness adjustment is performed on each slide sample image in the first image set to obtain a second image set, wherein the pixel brightness adjustment is performed by: denoising overexposed pixel points in the slide sample and reconstructing brightness of underexposed pixel points in the slide sample;

[0031] Automatically annotate the chromosomes of each slide sample image in the second image set to obtain a third image set, then balance the positive and negative samples in the third image set to obtain a fourth image set, and use the fourth image set to train a chromosome region detection model, wherein the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set;

[0032] Obtain a 10x image to be tested, input the 10x image to be tested into a chromosome region detection model to obtain a 10x chromosome region, obtain the corresponding 100x chromosome region based on the coordinates of the 10x chromosome region, splice a feature map of the 10x chromosome region with a feature map of the corresponding 100x chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0033] In some embodiments, in the step of adjusting pixel brightness of each glass slide sample image in the first image set, pixels in the glass slide sample image whose brightness value is greater than or equal to the overexposure threshold are regarded as overexposed pixels, and each overexposed pixel is denoised by a gradient domain denoising method.

[0034] Specifically, this solution sets the overexposure threshold to 200, and the gradient domain denoising method operates on the slide sample image in the gradient space, thereby reducing the gradient change of the overexposed pixels therein, thereby avoiding texture loss and halo effect caused by overexposure.

[0035] In some embodiments, in the step of adjusting pixel brightness of each glass slide sample image in the first image set, pixels in the glass slide sample whose brightness value is less than the underexposure threshold are regarded as underexposed pixels, and the brightness of each underexposed pixel is reconstructed using Retinex theory.

[0036] Specifically, the Retinex theory emphasizes separating reflectance and illumination components from an image and preserving the texture details of the image by reconstructing the brightness part. Therefore, this scheme uses the Retinex theory to perform brightness reconstruction, which can effectively enhance the details in low-light areas and avoid the loss of details caused by underexposure.

[0037] Specifically, the underexposure threshold of this solution is set to 15. In the glass slide sample image, when the brightness of a pixel point is less than a certain value or close to 0, it is difficult for the human eye to see clearly. Therefore, this solution uses 15 as the underexposure threshold to perform brightness reconstruction on those unclear pixel points.

[0038] In some specific embodiments, the overexposed pixel points in each glass slide sample are used as the foreground region. If the foreground region in the glass slide sample is greater than or equal to the first limit, the weight for noise reduction of overexposed pixel points is greater than the weight for brightness reconstruction of underexposed pixel points. If the foreground region in the glass slide sample is less than or equal to the first limit, the weight for noise reduction of overexposed pixel points is less than the weight for brightness reconstruction of underexposed pixel points.

[0039] Specifically, the first limit is 10%. That is to say, when the foreground region in the glass slide sample image is greater than or equal to 10%, it is considered that the proportion of the chromosome region in the glass slide sample image is high. Therefore, more attention should be paid to the noise reduction effect of overexposed pixel points to avoid detail loss caused by overexposure. When the foreground region in the glass slide sample image is less than 10%, the useless background region occupies most of the area. At this time, more emphasis should be placed on the brightness reconstruction effect of underexposed pixel points to highlight structural details such as short bands.

[0040] In some specific embodiments, if the proportion of overexposed pixel points in the glass slide sample image obtained by the ten - fold magnification imaging system is greater than the second limit, the gain reduction mechanism is triggered and the exposure duration of the ten - fold magnification imaging system is shortened. If the proportion of underexposed pixel points in the glass slide sample image obtained by the ten - fold magnification imaging system is greater than the second limit, the exposure time is extended.

[0041] Further, the second limit is greater than the first limit. The second limit in this solution is set to 30%.

[0042] In some other embodiments, the number of pixels in the foreground region is divided by the total number of pixels in the corresponding glass slide sample image as the image cleanliness. If the image cleanliness is less than the first cleanliness threshold, the weight for noise reduction of overexposed pixel points is less than the weight for brightness reconstruction of underexposed pixel points. If the image cleanliness is greater than or equal to the second cleanliness threshold, the weight for noise reduction of overexposed pixel points is greater than the weight for brightness reconstruction of underexposed pixel points, where the first cleanliness threshold is less than the second cleanliness threshold.

[0043] Specifically, the first cleanliness threshold in this solution is 0.3, and the second cleanliness threshold is 0.7.

[0044] In some embodiments, a pre-trained pseudo-label generator is used to label each chromosome region in each slide sample image to obtain a first labeling result, and the labeling confidence of each chromosome region is calculated. The regions with low confidence are screened out by a threshold method for the labeling confidence of each chromosome region, the low-confidence regions in the slide sample image are image-enhanced to obtain an image enhancement result, and then the pseudo-label generator is used to re-label the image enhancement result to obtain a second labeling result.

[0045] Specifically, taking the second labeling result as the final labeling result in the slide sample image, this solution performs image enhancement on the low-confidence regions in the slide sample image by the method of local contrast stretching. By performing image enhancement on the low-confidence results in the first labeling result, chromosome regions can be further screened, thereby improving the accuracy of short chromosome recognition and reducing the false detection rate of impurities.

[0046] In some other embodiments, SAM is used to automatically label each slide sample image to obtain a plurality of candidate labeling masks, and then the chromosome morphological prior knowledge is used to screen each candidate labeling mask to filter out the candidate labeling masks in non-chromosome regions to obtain chromosome region labels. Finally, each chromosome region label is corrected manually. By the above method, the labeling time can be effectively reduced.

[0047] In some specific embodiments, based on the labeling information in the third image set, the chromosome regions in each slide sample image are cropped as positive samples, and the regions other than the chromosome regions are used as negative samples. All positive and negative samples are integrated and balanced based on the generator to obtain a fourth image set.

[0048] Furthermore, in the step of balancing positive and negative samples based on the generator, the Retinex decomposition theory is embedded into the GAN generator and the GAN generator is pre-trained, and the pre-trained GAN generator is used to balance positive and negative samples.

[0049] Among them, the structure of the generator is:

[0050] # The generator is divided into two branches: reflectance map (R) and illumination map (I)

[0051] class RetinexGenerator(nn.Module):

[0052] def __init__(self):

[0053] super().__init__()

[0054] self.R_branch = UNet()# Reflectance map branch (chromosome texture)

[0055] self.I_branch = CNN()# Illumination map branch (background brightness)

[0056] def forward(self, x_lowlight):

[0057] R = self.R_branch(x_lowlight)# Output reflectance map

[0058] I = self.I_branch(x_lowlight)# Output illumination map

[0059] x_enhanced = R * I# Reconstruct enhanced image

[0060] return x_enhanced, R, I

[0061] Specifically, the loss function of the GAN generator in this solution includes reflectance consistency loss and illumination smoothness loss, and the formula is as follows:

[0062]

[0063]

[0064] Among them, is the reflectance consistency loss, which is used to measure the closeness between the enhancement result and the real image. is the reflectance map extracted from the original low-light image, is the reflectance map extracted from different illumination images at the same position, is the illumination smoothness loss, represents the squared L2 norm of the gradient of the illumination map (I), that is, the degree of change of the image in space. A large gradient represents a drastic change in brightness, and a small gradient represents smooth brightness.

[0065] Specifically, by embedding the Retinex decomposition theory into the GAN generator, the texture detail retention rate is increased by 35%.

[0066] Specifically, by using the GAN generator to balance positive and negative samples, the chromosome region detection model can correctly identify the chromosome region, avoiding over-learning the positive sample features or negative sample features due to the imbalance between positive and negative samples.

[0067] In some specific embodiments, the detection head structure of the chromosome region detection model is as Figure 2As shown, during the training process of the chromosome region detection model, chromosomes with lengths less than the length threshold are regarded as short chromosomes, and the weight of each short chromosome region is set to twice that of normal chromosomes.

[0068] Specifically, the length threshold in this solution is 50 pixels, that is, chromosomes with lengths less than 50 pixels are regarded as short chromosomes.

[0069] Furthermore, during the training process of the chromosome region detection model, K-means is used to cluster the labeled anchor boxes in each slide sample image, thereby defining the anchor box size range of the chromosome region detection model.

[0070] Specifically, traditional anchor boxes are generally between 1:1 and 1:3. However, due to the special shape of chromosomes, the anchor box sizes of the chromosome region detection model are redefined through clustering to improve the detection accuracy. Among them, the anchor box size range of the chromosome region detection model in this solution is 1:2 to 1:5.

[0071] Furthermore, MobileViT layers are added to the anchor box prediction and confidence prediction channels of the chromosome region detection model, thereby enhancing the ability to distinguish between dense chromosomes and background impurities.

[0072] Furthermore, the chromosome region detection model includes a first confidence prediction channel and a second confidence prediction channel. The first confidence prediction channel is used to predict regular chromosome regions, and the second confidence prediction channel is used to predict short chromosome regions. Among them, in the second confidence prediction channel, the feature map of the short chromosome region is first upsampled once.

[0073] Specifically, the detection effect of short chromosome regions can be further improved through the design of double detection boxes.

[0074] Furthermore, the chromosome region detection model is trained in a self-supervised manner. That is, during the training process, some unlabeled chromosome images are introduced, and 20% - 50% of the regions in the unlabeled chromosome images are randomly covered.

[0075] Furthermore, pruning operations are performed on the trained chromosome region detection model, thereby reducing the volume size of the chromosome region detection model and reducing the detection time.

[0076] Specifically, the backbone network of the chromosome region detection model is quantized using INT8, and the backbone network is pruned according to the quantization results. The detection head of the chromosome region detection model is detected with FP16 precision, and the detection head is pruned according to the precision detection results.

[0077] In some other embodiments, the feature map extracted by the trained chromosome overlap detection model in the feature extraction part is input into the ResNet model for post-processing classification to obtain the ten-fold chromosome region.

[0078] Specifically, by inputting the feature map extracted by the trained chromosome overlap detection model in the feature extraction part into the ResNet model, computational redundancy can be reduced, thereby improving the classification inference speed and reducing the end-to-end latency.

[0079] In some embodiments, MobileNetV2 is used to detect the number of chromosome regions in the ten-fold image to be measured. If the number of chromosome regions is less than the complexity threshold, the ten-fold image to be measured is downsampled and then input into the chromosome region detection model.

[0080] Specifically, if the number of chromosome regions is less than the complexity threshold, it indicates that the complexity of the image is low. Therefore, it is downsampled and input into the chromosome region detection model to improve the detection speed. Similarly, if the number of chromosome regions is greater than or equal to the complexity threshold, it indicates that the complexity of the ten-fold image to be measured is high, and the resolution of the ten-fold image to be measured is not adjusted.

[0081] In some embodiments, the coordinates of the ten-fold chromosome region are mapped to the corresponding hundred-fold slide image through coordinate mapping to obtain the hundred-fold chromosome region. The hundred-fold chromosome region is downsampled, and the feature maps of the downsampled hundred-fold chromosome region and the ten-fold chromosome region are spliced through a Transformer to obtain a chromosome detection map.

[0082] Specifically, the Transformer is good at capturing long-range dependencies. Between different scales of images, there may be relatively distant associations between features. Traditional CNNs learn local features through local convolution operations, but for the relationships between different scales (such as low-magnification and high-magnification lenses), the locality of the convolutional structure may be insufficient to effectively capture these long-range dependencies. Chromosome images may be affected by background noise, object deformation, etc. Especially during the conversion from ten-fold to hundred-fold magnification, the geometric shape and texture information of the image may change significantly. The Transformer, through its powerful transformation ability and global information propagation ability, can effectively extract and fuse information under these challenges, reduce noise interference, and improve detection accuracy.

[0083] In some embodiments, before inputting the chromosome detection map into the pre-trained chromosome overlap detection model, the foreground region and the background region of the chromosome detection map are separated through K-means clustering, thereby reducing the interference of background impurities.

[0084] Specifically, the schematic diagram of separating the foreground region and the background region of the chromosome detection map is as follows Figure 3 shown Figure 3 In (a) of Figure 3 , it represents the schematic diagram where the foreground and background regions are not separated. Figure 3 In (b) of Figure 3 , it represents the schematic diagram where the foreground region and the background region are separated. During the process of separating the foreground region and the background region of the chromosome detection map, the chromosome detection map is evenly divided into 8 intervals in the gradient direction of 0° to 360° to generate an 8-channel directional heat map to obtain gradient amplitude features. Each channel corresponds to the response intensity of a direction interval. Then, the rotation-invariant features of the chromosome detection map are calculated, and the gradient amplitude features and the rotation-invariant features are weighted and fused to obtain a clustering feature map. The formula is expressed as follows:

[0085] Ftexture = 0.7Gfusion + 0.3LBPri

[0086] where Ftexture is the clustering feature map, Gfusion is the gradient amplitude feature, LBPri is the rotation-invariant feature, and 0.7 and 0.3 are the corresponding weights.

[0087] Furthermore, based on the feature space density distribution of the clustering feature map, the initial clustering center is selected, and clustering iteration is performed based on the initial clustering center to separate the foreground region and the background region of the chromosome detection map.

[0088] Specifically, based on the feature space density distribution, the point with the highest density is selected as the initial clustering center. The formula is expressed as follows:

[0089]

[0090] where is the number of neighbors of point within the radius r, is the initial clustering center.

[0091] Subsequently, the probability of selecting the subsequent clustering center is inversely proportional to the distance to the selected center and the local density. The formula is expressed as follows:

[0092]

[0093] where is the distance from point to the selected center point, is the number of neighbors of point within the radius r, is the probability that point is selected as the clustering center.

[0094] Specifically, during the iterative clustering process, the overall silhouette coefficient S(t) after each iteration is calculated. If |S(t) - S(t - 1)| < 0.001 for three consecutive iterations, or when the iteration round is reached, the iteration is terminated.

[0095] In some specific embodiments, an overlap optimization loss is set as the loss function during the training process of the chromosome overlap detection model. The overlap optimization loss includes an angle optimization loss, a shape constraint loss, and an overlap region loss.

[0096] Specifically, the formula for the angle optimization loss is expressed as follows:

[0097]

[0098] where predicts the angle of the overlap detection box, is the angle of the true overlap detection box, is the angle optimization loss.

[0099] Specifically, the formula for the shape constraint loss is expressed as follows:

[0100]

[0101] where is the width of the predicted overlap detection box, is the height of the predicted overlap detection box, is the width of the true overlap detection box, is the height of the true overlap detection box, is the shape constraint loss.

[0102] Specifically, the formula for the overlap region loss is expressed as follows:

[0103]

[0104] where, is the predicted overlap region area, is the true overlap region area, is a measure of the consistency of features within the overlap region, is the overlap region loss, is the weight of.

[0105] In summary, the overlap optimization loss is + + .

[0106] In some specific embodiments, by setting the overlap optimization loss, the chromosome overlap detection model can better detect chromosomes in the overlapping region, thereby improving the classification accuracy of chromosomes in the overlapping region.

[0107] Embodiment 2

[0108] Based on the same idea, refer to Figure 4 , the present application also proposes a chromosome overlap detection device based on cross-modal feature fusion, comprising:

[0109] A brightness adjustment module, used to obtain multiple slide sample images as a first image set through a ten-fold mirror imaging system, and to adjust the pixel brightness of each slide sample image in the first image set to obtain a second image set, wherein the pixel brightness adjustment includes: performing noise reduction on overexposed pixel points in the slide sample and performing brightness reconstruction on underexposed pixel points in the slide sample;

[0110] a labeling module, for automatically labeling chromosomes of each slide sample image in the second image set to obtain a third image set, then balancing the positive and negative samples in the third image set to obtain a fourth image set, and using the fourth image set to train a chromosome region detection model, wherein the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set;

[0111] A detection module is used to obtain a 10x microscope image to be tested, input the 10x microscope image to be tested into a chromosome region detection model to obtain a 10x microscope chromosome region, obtain the corresponding 100x microscope chromosome region based on the coordinates of the 10x microscope chromosome region, splice a feature map of the 10x microscope chromosome region with a feature map of the corresponding 100x microscope chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0112] Embodiment 3

[0113] This embodiment also provides an electronic device, referring to Figure 5 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0114] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0115] Among them, the memory 404 may include a mass storage 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include removable or non-removable (or fixed) media. In appropriate cases, the memory 404 may be internal or external to the data processing device. In a particular embodiment, the memory 404 is non-volatile memory. In a particular embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. In appropriate cases, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended date out dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0116] The memory 404 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402.

[0117] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the chromosome overlap detection methods based on cross-modal feature fusion in the above embodiments.

[0118] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. Among them, the transmission device 406 is connected to the above processor 402, and the input / output device 408 is connected to the above processor 402.

[0119] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above network may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0120] The input / output device 408 is used to input or output information. In this embodiment, the input information can be a slide sample image, etc., and the output information can be a chromosome overlap detection result, etc.

[0121] Optionally, in this embodiment, the above processor 402 can be set to execute the following steps through a computer program:

[0122] Obtain a plurality of slide sample images as a first image set through a ten-fold magnification imaging system, and perform pixel brightness adjustment on each slide sample image in the first image set to obtain a second image set. The pixel brightness adjustment is: perform noise reduction on overexposed pixel points in the slide sample, and perform brightness reconstruction on underexposed pixel points in the slide sample;

[0123] Perform automatic annotation of chromosomes on each slide sample image in the second image set to obtain a third image set, then balance the positive and negative samples in the third image set to obtain a fourth image set, and use the fourth image set to train a chromosome region detection model, where the positive sample is the chromosome region of each slide sample image in the third image set, and the negative sample is the non-chromosome region of each slide sample image in the third image set;

[0124] Obtain a 10x image to be tested, input the 10x image to be tested into a chromosome region detection model to obtain a 10x chromosome region, obtain the corresponding 100x chromosome region based on the coordinates of the 10x chromosome region, splice a feature map of the 10x chromosome region with a feature map of the corresponding 100x chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0125] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0126] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0127] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 5 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0128] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity in description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0129] The above embodiments only express several implementation manners of the present application, and the description thereof is relatively specific and detailed. However, it should not be construed as a limitation to the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A chromosome overlap detection method based on cross-modal feature fusion, characterized in that, It includes the following steps: Obtain multiple slide sample images as a first image set through a ten - fold magnification imaging system, and perform pixel brightness adjustment on each slide sample image in the first image set to obtain a second image set. The pixel brightness adjustment is as follows: perform noise reduction on over - exposed pixel points in the slide sample, and perform brightness reconstruction on under - exposed pixel points in the slide sample; Perform automatic annotation of chromosomes on each slide sample image in the second image set to obtain a third image set, then balance the positive and negative samples in the third image set to obtain a fourth image set, and use the fourth image set to train a chromosome region detection model. Among them, the positive sample is the chromosome region of each slide sample image in the third image set, and the negative sample is the non - chromosome region of each slide sample image in the third image set; Obtain a ten - fold magnification test image, input the ten - fold magnification test image into the chromosome region detection model to obtain a ten - fold magnification chromosome region, obtain the corresponding hundred - fold magnification chromosome region based on the coordinates of the ten - fold magnification chromosome region, splice the feature map of the ten - fold magnification chromosome region and the feature map of the corresponding hundred - fold magnification chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre - trained chromosome overlap detection model to obtain a chromosome overlap detection result.

2. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, wherein In the step of performing pixel brightness adjustment on each slide sample image in the first image set, pixel points with brightness values greater than or equal to the over - exposure threshold in the slide sample image are used as over - exposed pixel points, and each over - exposed pixel point is denoised by the gradient domain denoising method; pixel points with brightness values less than the under - exposure threshold in the slide sample are used as under - exposed pixel points, and each under - exposed pixel point is subjected to brightness reconstruction by the Retinex theory.

3. A method for detecting chromosome overlap based on cross-modal feature fusion according to claim 1, wherein Take the over - exposed pixel points in each slide sample as the foreground region. If the foreground region in the slide sample is greater than or equal to the first limit, the weight for denoising the over - exposed pixel points is greater than the weight for brightness reconstruction of the under - exposed pixel points. If the foreground region in the slide sample is less than or equal to the first limit, the weight for denoising the over - exposed pixel points is less than the weight for brightness reconstruction of the under - exposed pixel points.

4. A method for detecting chromosome overlap based on cross-modal feature fusion according to claim 1, characterized in that, Use a pre - trained pseudo - annotation generator to mark each chromosome region in each slide sample image to obtain a first annotation result, calculate the annotation confidence of each chromosome region, screen the annotation confidence of each chromosome region by the threshold method to obtain low - confidence regions, perform image enhancement on the low - confidence regions in the slide sample image to obtain an image enhancement result, and then re - perform secondary annotation on the image enhancement result based on the pseudo - annotation generator to obtain a second annotation result.

5. A method for detecting chromosome overlap based on cross-modal feature fusion according to claim 1, characterized in that Crop out the chromosome regions in each slide sample image as positive samples based on the annotation information in the third image set, take the regions other than the chromosome regions as negative samples, integrate all positive and negative samples, and balance the positive and negative samples based on the generator to obtain a fourth image set.

6. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, wherein, The coordinates of the chromosome region under the ten - fold microscope are mapped to the corresponding hundred - fold microscope slide image through coordinate mapping to obtain the chromosome region under the hundred - fold microscope. The chromosome region under the hundred - fold microscope is downsampled, and the downsampled chromosome region under the hundred - fold microscope and the chromosome region under the ten - fold microscope are stitched feature maps through a Transformer to obtain a chromosome detection map.

7. A chromosome overlap detection method based on cross-modal feature fusion according to claim 1, characterized in that During the training process of the chromosome overlap detection model, an overlap optimization loss is set as the loss function. The overlap optimization loss includes an angle optimization loss, a shape constraint loss, and an overlap region loss. The formula of the angle optimization loss is expressed as follows: Among them, Predict the angle of the overlapping detection box, is the angle of the ground truth overlapping detection box, is the angle optimization loss; The formula of the shape constraint loss is expressed as follows: Among them, is the width of the predicted overlapping detection box, is the height of the predicted overlapping detection box, is the width of the ground-truth overlapping detection box, is the height of the ground-truth overlapping detection box, is the shape constraint loss; The formula of the overlap region loss is expressed as follows: Among them, is the predicted overlapping area, is the actual overlapping area, is a measure of the consistency of features within the overlapping area, is the overlapping area loss, is the weight of.

8. A chromosome overlap detection device based on cross-modal feature fusion, characterized in that, Including: A brightness adjustment module, which is used to obtain multiple slide sample images as the first image set through the ten - fold microscope imaging system, and perform pixel brightness adjustment on each slide sample image in the first image set to obtain the second image set. The pixel brightness adjustment is: denoising over - exposed pixel points in the slide sample and reconstructing the brightness of under - exposed pixel points in the slide sample. A labeling module, which is used to automatically label the chromosomes of each slide sample image in the second image set to obtain the third image set, then balance the positive and negative samples in the third image set to obtain the fourth image set, and use the fourth image set to train a chromosome region detection model. Among them, the positive sample is the chromosome region of each slide sample image in the third image set, and the negative sample is the non - chromosome region of each slide sample image in the third image set. A detection module, which is used to obtain the ten - fold microscope image to be measured, input the ten - fold microscope image to be measured into the chromosome region detection model to obtain the chromosome region under the ten - fold microscope, obtain the corresponding chromosome region under the hundred - fold microscope based on the coordinates of the chromosome region under the ten - fold microscope, stitch the feature map of the chromosome region under the ten - fold microscope and the feature map of the corresponding chromosome region under the hundred - fold microscope to obtain a chromosome detection map, and input the chromosome detection map into the pre - trained chromosome overlap detection model to obtain the chromosome overlap detection result.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is set to run the computer program to execute a chromosome overlap detection method according to any one of claims 1 - 7, which is based on cross - modal feature fusion.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program. The computer program includes program codes for controlling a process to execute the process. When the program codes are executed by the processor, a chromosome overlap detection method according to any one of claims 1 - 7, which is based on cross - modal feature fusion, is implemented.

Images