Chromosome overlapping detection method and device based on cross-modal feature fusion

Abstract

The invention provides a chromosome overlapping detection method and device based on cross-modal feature fusion, and the method comprises the following steps: obtaining a ten-time lens to-be-detected image, inputting the ten-time lens to-be-detected image into a chromosome region detection model, and obtaining a ten-time lens chromosome region, acquiring a corresponding hundred-fold mirror chromosome region based on the coordinates of the ten-fold mirror chromosome region, and splicing the feature map of the ten-fold mirror chromosome region and the feature map of the corresponding hundred-fold mirror chromosome region to obtain a chromosome detection map, and inputting the chromosome detection graph into a pre-trained chromosome overlapping detection model to obtain a chromosome overlapping detection result. According to the scheme, the feature map of the ten-fold mirror chromosome region and the feature map of the corresponding hundred-fold mirror chromosome region are spliced as the input image, so that the detection precision of the short chromosome is optimized.

Description

Technical Field

[0001] This application relates to the field of image detection, and in particular, to a chromosome overlap detection method and device based on cross-modal feature fusion. Background Art

[0002] Chromosome karyotype analysis plays a crucial role in genetic science research and clinical auxiliary diagnosis. As an important auxiliary link before karyotype analysis, chromosome detection and overlap degree analysis are of great significance for subsequent accurate analysis. Currently, chromosome overlap detection is mainly achieved by performing image processing on glass slide samples in the metaphase of chromosome mitosis. First, focus and shoot under a ten-fold lens to identify the presence and position of chromosomes, then take a hundred-fold lens image based on this position, and further enhance and classify the overlap degree of the hundred-fold image. However, the existing chromosome karyotype automatic analysis systems have the following problems: 1. Low detection efficiency due to image quality problems: Image quality highly depends on the cleanliness of the glass slide, the clarity of focus, and the exposure parameters. Bubbles, scratches, etc. on the glass slide will interfere with the image. Focus errors are likely to make the image blurred or out of focus, and improper exposure parameters will cause overexposure or low light phenomena. These factors combined make the obtained high-resolution image quality poor, increasing the time cost of image detection; 2. Limitations of traditional detection methods: When detecting under a ten-fold lens, short chromosomes are difficult to distinguish from impurities due to their small pixel ratio, and traditional cropping methods will cause loss of key information; at the same time, physical operations such as dripping oil and smearing oil will cause the field of view coordinates to shift, further magnifying the error of subsequent magnification and recognition. In the hundred-fold lens screening link, the classification of overlapping chromosomes mainly relies on traditional image segmentation methods, such as thresholding, edge detection, etc. These methods are insufficient in dealing with scenarios with sparse clusters and complex cross shapes, and it is difficult to accurately distinguish the overlap degree of chromosomes; moreover, background impurities (strip-shaped or circular interference) are easily confused with chromosome textures, and false detections are also likely to occur for curved chromosomes.

[0003] The above problems seriously affect the accuracy of chromosome karyotype analysis. Therefore, how to accurately locate the chromosome position and screen overlapping chromosomes under poor image quality is an urgent problem to be solved in the prior art. Summary of the Invention

[0004] The embodiments of this application provide a chromosome position method and device based on cross-modal feature fusion. By splicing the feature map of the chromosome region under a ten-fold lens and the feature map of the corresponding chromosome region under a hundred-fold lens as the input image, the detection accuracy of short chromosomes is optimized.

[0005] In a first aspect, the embodiments of this application provide a chromosome overlap detection method based on cross-modal feature fusion, and the method includes: Obtain multiple glass slide sample images as the first image set through a ten - fold magnification imaging system, and perform pixel brightness adjustment on each glass slide sample image in the first image set to obtain the second image set. The pixel brightness adjustment is as follows: perform noise reduction on over - exposed pixel points in the glass slide sample, and perform brightness reconstruction on under - exposed pixel points in the glass slide sample; Perform automatic annotation of chromosomes on each glass slide sample image in the second image set to obtain the third image set, then balance the positive and negative samples in the third image set to obtain the fourth image set, and use the fourth image set to train a chromosome region detection model. Among them, the positive sample is the chromosome region of each glass slide sample image in the third image set, and the negative sample is the non - chromosome region of each glass slide sample image in the third image set; Obtain a ten - fold magnification image to be measured, input the ten - fold magnification image to be measured into the chromosome region detection model to obtain the ten - fold magnification chromosome region, obtain the corresponding hundred - fold magnification chromosome region based on the coordinates of the ten - fold magnification chromosome region, splice the feature map of the ten - fold magnification chromosome region and the feature map of the corresponding hundred - fold magnification chromosome region to obtain a chromosome detection map, and input the chromosome detection map into the pre - trained chromosome overlap detection model to obtain the chromosome overlap detection result.

[0006] In a second aspect, an embodiment of the present application provides a chromosome overlap detection device based on cross - modal feature fusion, including: A brightness adjustment module, configured to obtain multiple glass slide sample images as the first image set through a ten - fold magnification imaging system, and perform pixel brightness adjustment on each glass slide sample image in the first image set to obtain the second image set. The pixel brightness adjustment is as follows: perform noise reduction on over - exposed pixel points in the glass slide sample, and perform brightness reconstruction on under - exposed pixel points in the glass slide sample; An annotation module, configured to perform automatic annotation of chromosomes on each glass slide sample image in the second image set to obtain the third image set, then balance the positive and negative samples in the third image set to obtain the fourth image set, and use the fourth image set to train a chromosome region detection model. Among them, the positive sample is the chromosome region of each glass slide sample image in the third image set, and the negative sample is the non - chromosome region of each glass slide sample image in the third image set; A detection module, configured to obtain a ten - fold magnification image to be measured, input the ten - fold magnification image to be measured into the chromosome region detection model to obtain the ten - fold magnification chromosome region, obtain the corresponding hundred - fold magnification chromosome region based on the coordinates of the ten - fold magnification chromosome region, splice the feature map of the ten - fold magnification chromosome region and the feature map of the corresponding hundred - fold magnification chromosome region to obtain a chromosome detection map, and input the chromosome detection map into the pre - trained chromosome overlap detection model to obtain the chromosome overlap detection result.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute a chromosome overlap detection method based on cross-modal feature fusion.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium. A computer program is stored in the readable storage medium. The computer program includes program code for controlling a process to execute the process, and the process includes a chromosome overlap detection method based on cross-modal feature fusion.

[0009] The main contributions and innovations of the present invention are as follows: In the embodiment of the present application, by assigning higher weights to short chromosomes in the chromosome region detection model and performing identification after improving the resolution for short chromosomes, the detection accuracy of short chromosome regions is effectively improved, providing more accurate data for subsequent chromosome analysis. In this solution, the cross-modal fusion of ten-fold microscope positioning information and one-hundred-fold microscope texture features is carried out, giving full play to the advantages of two different magnification lenses, enabling short chromosomes to be more accurately identified and located during the detection process, and further improving the overall detection performance. In the embodiment of the present application, by adding an overlap optimization loss as an additional loss function during the training process of the chromosome overlap detection model, the detection effect of the chromosome overlap detection model on the chromosome overlap region is improved, and the chromosome position is better detected.

[0010] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 is a flowchart of a chromosome overlap detection method based on cross-modal feature fusion according to an embodiment of the present application; Figure 2 is a schematic diagram of the detection head structure of a chromosome region detection model according to an embodiment of the present application; Figure 3 is a schematic diagram of separating the foreground region and the background region of a chromosome detection map according to an embodiment of the present application; Figure 4 is a block diagram of the structure of a chromosome overlap detection device based on cross-modal feature fusion according to an embodiment of the present application; Figure 5 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0012] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0013] It should be noted that: In other embodiments, the steps of the corresponding methods are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0014] Embodiment 1 The embodiment of this application provides a chromosome overlap detection method based on cross-modal feature fusion. By splicing the feature maps of the chromosome regions at the ten-fold magnification with the corresponding feature maps of the chromosome regions at the hundred-fold magnification as the input image, the detection accuracy for short chromosomes is optimized. And by setting an overlap optimization loss as an additional loss function in the chromosome detection model, the detection effect of the chromosome overlap detection model on the chromosome overlap region is improved. Specifically, referring to Figure 1 , the method includes: Obtain a plurality of slide sample images as the first image set through a ten-fold magnification imaging system, and perform pixel brightness adjustment on each slide sample image in the first image set to obtain a second image set. The pixel brightness adjustment is: perform noise reduction on overexposed pixel points in the slide sample, and perform brightness reconstruction on underexposed pixel points in the slide sample; Automatically label the chromosomes for each slide sample image in the second image set to obtain a third image set, then balance the positive and negative samples in the third image set to obtain a fourth image set, and use the fourth image set to train a chromosome region detection model, where the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set; Obtain a 10x image to be tested, input the 10x image to be tested into a chromosome region detection model to obtain a 10x chromosome region, obtain the corresponding 100x chromosome region based on the coordinates of the 10x chromosome region, splice a feature map of the 10x chromosome region with a feature map of the corresponding 100x chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0015] In some embodiments, in the step of adjusting pixel brightness of each glass slide sample image in the first image set, pixels in the glass slide sample image whose brightness value is greater than or equal to the overexposure threshold are regarded as overexposed pixels, and each overexposed pixel is denoised by a gradient domain denoising method.

[0016] Specifically, this solution sets the overexposure threshold to 200, and the gradient domain denoising method operates on the slide sample image in the gradient space, thereby reducing the gradient change of the overexposed pixels therein, thereby avoiding texture loss and halo effect caused by overexposure.

[0017] In some embodiments, in the step of adjusting pixel brightness of each glass slide sample image in the first image set, pixels in the glass slide sample whose brightness value is less than the underexposure threshold are regarded as underexposed pixels, and the brightness of each underexposed pixel is reconstructed using Retinex theory.

[0018] Specifically, the Retinex theory emphasizes separating reflectance and illumination components from an image and preserving the texture details of the image by reconstructing the brightness part. Therefore, this scheme uses the Retinex theory to perform brightness reconstruction, which can effectively enhance the details in low-light areas and avoid the loss of details caused by underexposure.

[0019] Specifically, this scheme sets the underexposure threshold to 15. Since in the slide sample image, when the brightness of a pixel is less than a certain value or close to 0, it is difficult for the human eye to see clearly, so this scheme uses 15 as the underexposure threshold to reconstruct the brightness of those unclear pixels.

[0020] In some specific embodiments, the overexposed pixels in each glass slide sample are taken as the foreground area. If the foreground area in the glass slide sample is greater than or equal to a first limit, the weight for denoising the overexposed pixels is greater than the weight for reconstructing the brightness of the underexposed pixels. If the foreground area in the glass slide sample is less than or equal to the first limit, the weight for denoising the overexposed pixels is less than the weight for reconstructing the brightness of the underexposed pixels.

[0021] Specifically, the first threshold is 10%. That is to say, when the foreground area in the slide sample image is greater than or equal to 10%, it is considered that the proportion of the chromosome area in the slide sample image is high. Therefore, more attention should be paid to the noise reduction effect of overexposed pixel points to avoid detail loss caused by overexposure. When the foreground area in the slide sample image is less than 10%, the useless background area occupies most of the area. At this time, more emphasis should be placed on the brightness reconstruction effect of underexposed pixel points to highlight structural details such as short bands.

[0022] In some specific embodiments, if the proportion of overexposed pixel points in the slide sample image obtained by the ten-fold magnification imaging system is greater than the second threshold, the gain reduction mechanism is triggered and the exposure duration of the ten-fold magnification imaging system is shortened. If the proportion of underexposed pixel points in the slide sample image obtained by the ten-fold magnification imaging system is greater than the second threshold, the exposure time is extended.

[0023] Further, the second threshold is greater than the first threshold, and the second threshold in this solution is set to 30%.

[0024] In some other embodiments, the number of pixels in the foreground area is divided by the total number of pixels in the corresponding slide sample image as the image cleanliness. If the image cleanliness is less than the first cleanliness threshold, the weight for noise reduction of overexposed pixel points is less than the weight for brightness reconstruction of underexposed pixel points. If the image cleanliness is greater than or equal to the second cleanliness threshold, the weight for noise reduction of overexposed pixel points is greater than the weight for brightness reconstruction of underexposed pixel points, where the first cleanliness threshold is less than the second cleanliness threshold.

[0025] Specifically, the first cleanliness threshold in this solution is 0.3, and the second cleanliness threshold is 0.7.

[0026] In some embodiments, a pre-trained pseudo-label generator is used to mark each chromosome area in each slide sample image to obtain a first annotation result, and the annotation confidence of each chromosome area is calculated. The annotation confidence of each chromosome area is screened by a threshold method to obtain low-confidence areas, and the low-confidence areas in the slide sample image are image-enhanced to obtain an image enhancement result, and then the image enhancement result is re-annotated based on the pseudo-label generator to obtain a second annotation result.

[0027] Specifically, the second annotation result is used as the final annotation result in the slide sample image. In this solution, the local contrast stretching method is used to image-enhance the low-confidence areas in the slide sample image. By image-enhancing the low-confidence results in the first marking result, chromosome areas can be further screened, thereby improving the accuracy of short chromosome recognition and reducing the impurity misdetection rate.

[0028] In some other embodiments, each glass slide sample image is automatically annotated through SAM to obtain a plurality of candidate annotation masks, and then each candidate annotation mask is filtered through the prior knowledge of chromosome morphology to filter out the candidate annotation masks in non-chromosome regions to obtain chromosome region markers. Finally, each chromosome region marker is corrected manually. By the above method, the annotation time can be effectively reduced.

[0029] In some specific embodiments, the chromosome regions in each glass slide sample image are cropped based on the annotation information in the third image set as positive samples, and the regions other than the chromosome regions are used as negative samples. All positive and negative samples are integrated and balanced based on the generator to obtain the fourth image set.

[0030] Further, in the step of balancing positive and negative samples based on the generator, the Retinex decomposition theory is embedded in the GAN generator and the GAN generator is pre-trained, and the pre-trained GAN generator is used to balance positive and negative samples.

[0031] Among them, the structure of the generator is as follows: # The generator is divided into two branches: reflectance map (R) and illumination map (I) class RetinexGenerator(nn.Module): def __init__(self): super().__init__() self.R_branch = UNet() # Reflectance map branch (chromosome texture) self.I_branch = CNN() # Illumination map branch (background brightness) def forward(self, x_lowlight): R = self.R_branch(x_lowlight) # Output reflectance map I = self.I_branch(x_lowlight) # Output illumination map x_enhanced = R * I # Reconstruct the enhanced image return x_enhanced, R, I Specifically, the loss function of the GAN generator in this solution includes reflectance consistency loss and illumination smoothness loss, and the formula is as follows:

[0032]

[0033] Among them, is the reflectance consistency loss, which is used to measure the closeness between the enhancement result and the real image. is the reflectance map extracted from the original low-light image, is the reflectance map extracted from different illumination images at the same position, is the illumination smoothness loss, represents the L2 norm of the square of the gradient of the illumination map (I), that is, the degree of change of the image in space. A large gradient represents a drastic change in brightness, and a small gradient represents smooth brightness.

[0034] Specifically, by embedding the Retinex decomposition theory into the GAN generator, the texture detail retention rate is increased by 35%.

[0035] Specifically, by using the GAN generator to balance positive and negative samples, the chromosome region detection model can correctly identify chromosome regions, avoiding overlearning positive or negative sample features due to unbalanced positive and negative samples.

[0036] In some specific embodiments, the detection head structure of the chromosome region detection model is as Figure 2 shown. During the training process of the chromosome region detection model, chromosomes with lengths less than the length threshold are regarded as short chromosomes, and the weight of each short chromosome region is set to twice that of normal chromosomes.

[0037] Specifically, the length threshold in this solution is 50 pixels, that is, chromosomes with lengths less than 50 pixels are regarded as short chromosomes.

[0038] Furthermore, during the training process of the chromosome region detection model, K-means is used to cluster the labeled anchor boxes in each slide sample image, thereby defining the anchor box size interval of the chromosome region detection model.

[0039] Specifically, traditional anchor boxes are generally between 1:1 and 1:3. However, due to the special shape of chromosomes, the anchor box sizes of the chromosome region detection model are redefined by the clustering method to improve the detection accuracy. Among them, the anchor box size interval of the chromosome region detection model in this solution is 1:2 to 1:5.

[0040] Furthermore, MobileViT layers are added to the anchor box prediction and confidence prediction channels of the chromosome region detection model, thereby enhancing the ability to distinguish between dense chromosomes and background impurities.

[0041] Furthermore, the chromosome region detection model includes a first confidence prediction channel and a second confidence prediction channel, wherein the first confidence prediction channel is used to predict regular chromosome regions, and the second confidence prediction channel is used to predict short chromosome regions, wherein the feature map of the short chromosome region is first upsampled in the second confidence prediction channel.

[0042] Specifically, the detection effect on short chromosome regions can be further improved through the design of double detection frames.

[0043] Furthermore, the chromosome region detection model is trained in a self-supervised manner, that is, some unlabeled chromosome images are introduced during the training process, and 20% to 50% of the regions in the unlabeled chromosome images are randomly covered.

[0044] Furthermore, a pruning operation is performed on the trained chromosome region detection model, thereby reducing the size of the chromosome region detection model and reducing the detection time.

[0045] Specifically, the backbone network of the chromosome region detection model is quantized using INT8, and the backbone network is pruned according to the quantization results. The detection head of the chromosome region detection model is detected using FP16 precision, and the detection head is pruned according to the precision detection results.

[0046] In some other embodiments, the feature map extracted by the trained chromosome detection model in the feature extraction part is input into the resnet model for post-processing and classification to obtain the ten-fold chromosome region.

[0047] Specifically, by using the feature map extracted by the trained chromosome detection model in the feature extraction part and inputting it into the resnet model, computational redundancy can be reduced, thereby improving the classification reasoning speed and reducing end-to-end latency.

[0048] In some embodiments, MobileNetV2 is used to detect the number of chromosome regions in the ten-fold magnification image to be tested. If the number of chromosome regions is less than the complexity threshold, the ten-fold magnification image to be tested is downsampled and then input into the chromosome region detection model.

[0049] Specifically, if the number of chromosome regions is less than the complexity threshold, it means that the complexity of the image is low, so it is downsampled and input into the chromosome region detection model to increase the detection speed. Similarly, if the number of chromosome regions is greater than or equal to the complexity threshold, it means that the complexity of the image to be tested by the ten-fold microscope is high, and the resolution of the image to be tested by the ten-fold microscope is not adjusted.

[0050] In some embodiments, the coordinates of the ten - fold magnified chromosomal region are mapped to the corresponding hundred - fold magnified slide image through coordinate mapping to obtain the hundred - fold magnified chromosomal region. The hundred - fold magnified chromosomal region is downsampled, and the downsampled hundred - fold magnified chromosomal region and the ten - fold magnified chromosomal region are stitched together in terms of feature maps through a Transformer to obtain a chromosome detection map.

[0051] Specifically, the Transformer is good at capturing long - range dependencies. Among different scales of images, there may be relatively distant associations between features. Traditional CNNs learn local features through local convolution operations, but for the relationships between different scales (such as low - magnification and high - magnification lenses), the locality of the convolution structure may be insufficient to effectively capture these long - range dependencies. Chromosome images may be affected by background noise, object deformation, etc. Especially during the conversion from ten - fold magnification to hundred - fold magnification, the geometric shape and texture information of the image may change significantly. The Transformer, through its powerful transformation ability and global information propagation ability, can effectively extract and fuse information under these challenges, reduce noise interference, and improve detection accuracy.

[0052] In some embodiments, before inputting the chromosome detection map into a pre - trained chromosome overlap detection model, the foreground region and the background region of the chromosome detection map are separated through K - means clustering, so as to reduce the interference of background impurities.

[0053] Specifically, the schematic diagram of separating the foreground region and the background region of the chromosome detection map is as Figure 3 shown, Figure 3 in which (a) represents the schematic diagram before the foreground and background regions are separated, Figure 3 in which (b) represents the schematic diagram after the foreground region and the background region are separated. During the process of separating the foreground region and the background region of the chromosome detection map, the chromosome detection map is evenly divided into 8 intervals in the 0° - 360° gradient direction to generate an 8 - channel direction heat map to obtain gradient amplitude features. Each channel corresponds to the response intensity of a direction interval. Then, the rotation - invariant features of the chromosome detection map are calculated, and the gradient amplitude features and the rotation - invariant features are weighted and fused to obtain a clustering feature map. The formula is as follows: Ftexture = 0.7Gfusion+0.3LBPri where Ftexture is the clustering feature map, Gfusion is the gradient amplitude feature, LBPri is the rotation - invariant feature, and 0.7 and 0.3 are the corresponding weights.

[0054] Furthermore, the initial clustering centers are selected based on the feature space density distribution of the clustering feature map, and clustering iterations are performed based on the initial clustering centers to separate the foreground region and the background region of the chromosome detection map.

[0055] Specifically, the point with the highest density is selected as the initial clustering center based on the density distribution in the feature space, and the formula is as follows:

[0056] Among them, is the number of neighbors of point within the radius r, is the initial clustering center.

[0057] The probability of subsequent clustering center selection is inversely proportional to the distance to the selected center and the local density, and the formula is as follows:

[0058] Among them, is the distance from point to the selected center point, is the number of neighbors of point within the radius r, is the probability that point is selected as the clustering center.

[0059] Specifically, during the iterative clustering process, the overall silhouette coefficient S(t) after each iteration is calculated. If |S(t) - S(t - 1)| < 0.001 in three consecutive iterations, or when the iteration round is reached, the iteration is terminated.

[0060] In some specific embodiments, an overlap optimization loss is set as the loss function during the training process of the chromosome overlap detection model, and the overlap optimization loss includes an angle optimization loss, a shape constraint loss, and an overlap region loss.

[0061] Specifically, the formula of the angle optimization loss is as follows:

[0062] Among them, predicts the angle of the overlap detection box, is the angle of the true overlap detection box, is the angle optimization loss.

[0063] Specifically, the formula of the shape constraint loss is as follows:

[0064] Among them, is the width of the predicted overlap detection box, is the height of the predicted overlap detection box, is the width of the true overlap detection box, is the height of the true overlap detection box, is the shape constraint loss.

[0065] Specifically, the formula for the overlap area loss is as follows:

[0066] in, is the predicted overlapping area, is the actual overlapping area, is a measure of the consistency of features in the overlapping region, is the overlap area loss, for The weight of .

[0067] In summary, the overlap optimization loss is + + .

[0068] In some specific embodiments, by setting the overlap optimization loss, the chromosome overlap detection model can better detect chromosomes in the overlapping region, thereby improving the classification accuracy of chromosomes in the overlapping region.

[0069] Embodiment 2 Based on the same idea, refer to Figure 4 , the present application also proposes a chromosome overlap detection device based on cross-modal feature fusion, comprising: A brightness adjustment module, used to obtain multiple slide sample images as a first image set through a ten-fold mirror imaging system, and to adjust the pixel brightness of each slide sample image in the first image set to obtain a second image set, wherein the pixel brightness adjustment includes: performing noise reduction on overexposed pixel points in the slide sample and performing brightness reconstruction on underexposed pixel points in the slide sample; a labeling module, for automatically labeling chromosomes of each slide sample image in the second image set to obtain a third image set, then balancing the positive and negative samples in the third image set to obtain a fourth image set, and using the fourth image set to train a chromosome region detection model, wherein the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set; A detection module is used to obtain a 10x microscope image to be tested, input the 10x microscope image to be tested into a chromosome region detection model to obtain a 10x microscope chromosome region, obtain the corresponding 100x microscope chromosome region based on the coordinates of the 10x microscope chromosome region, splice a feature map of the 10x microscope chromosome region with a feature map of the corresponding 100x microscope chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0070] Embodiment 3 This embodiment also provides an electronic device. Referring to Figure 5 , it includes a memory 404 and a processor 402. A computer program is stored in the memory 404, and the processor 402 is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0071] Specifically, the above-mentioned processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0072] Among them, the memory 404 may include a mass memory 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include removable or non-removable (or fixed) media. In appropriate cases, the memory 404 may be internal or external to the data processing device. In a particular embodiment, the memory 404 is non-volatile memory. In a particular embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. In appropriate cases, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0073] The memory 404 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402.

[0074] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the cross-modal feature fusion-based chromosome overlap detection methods in the above embodiments.

[0075] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. Among them, the transmission device 406 is connected to the above processor 402, and the input / output device 408 is connected to the above processor 402.

[0076] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above network may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0077] The input / output device 408 is used to input or output information. In this embodiment, the input information can be a slide sample image, etc., and the output information can be a chromosome overlap detection result, etc.

[0078] Optionally, in this embodiment, the above processor 402 can be set to execute the following steps through a computer program: Obtain a plurality of slide sample images as a first image set through a ten-fold microscope imaging system, and perform pixel brightness adjustment on each slide sample image in the first image set to obtain a second image set. The pixel brightness adjustment is: denoise overexposed pixel points in the slide sample and perform brightness reconstruction on underexposed pixel points in the slide sample; Automatically annotate the chromosomes of each slide sample image in the second image set to obtain a third image set, then balance the positive and negative samples in the third image set to obtain a fourth image set, and use the fourth image set to train a chromosome region detection model, where the positive sample is the chromosome region of each slide sample image in the third image set, and the negative sample is the non-chromosome region of each slide sample image in the third image set; Obtain a 10x image to be tested, input the 10x image to be tested into a chromosome region detection model to obtain a 10x chromosome region, obtain the corresponding 100x chromosome region based on the coordinates of the 10x chromosome region, splice a feature map of the 10x chromosome region with a feature map of the corresponding 100x chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

[0079] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0080] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0081] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 5 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0082] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered that the scope described in this specification is covered.

[0083] The above embodiments only express several implementation manners of the present application. The description is relatively specific and detailed, but it should not be understood as a limitation to the scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A chromosome overlap detection method based on cross-modal feature fusion, characterized in that: The following steps are involved: A plurality of slide sample images are acquired as a first image set through a ten-fold mirror imaging system, and pixel brightness adjustment is performed on each slide sample image in the first image set to obtain a second image set, wherein the pixel brightness adjustment is performed by: denoising overexposed pixel points in the slide sample and reconstructing brightness of underexposed pixel points in the slide sample; Automatically annotate the chromosomes of each slide sample image in the second image set to obtain a third image set, then balance the positive and negative samples in the third image set to obtain a fourth image set, and use the fourth image set to train a chromosome region detection model, wherein the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set; Obtain a 10x image to be tested, input the 10x image to be tested into a chromosome region detection model to obtain a 10x chromosome region, obtain the corresponding 100x chromosome region based on the coordinates of the 10x chromosome region, splice a feature map of the 10x chromosome region with a feature map of the corresponding 100x chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome overlap detection model to obtain a chromosome overlap detection result.

2. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, characterized in that: In the step of adjusting the pixel brightness of each glass slide sample image in the first image set, the pixels in the glass slide sample image whose brightness value is greater than or equal to the overexposure threshold are regarded as overexposed pixels, and the gradient domain denoising method is used to reduce the noise of each overexposed pixel; the pixels in the glass slide sample whose brightness value is less than the underexposure threshold are regarded as underexposed pixels, and the brightness of each underexposed pixel is reconstructed by the Retinex theory.

3. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, characterized in that: The overexposed pixels in each slide sample are taken as the foreground area. If the foreground area in the slide sample is greater than or equal to the first limit, the weight for denoising the overexposed pixels is greater than the weight for reconstructing the brightness of the underexposed pixels. If the foreground area in the slide sample is less than or equal to the first limit, the weight for denoising the overexposed pixels is less than the weight for reconstructing the brightness of the underexposed pixels.

4. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, characterized in that: A pre-trained pseudo-annotation generator is used to mark each chromosome region in each slide sample image to obtain a first annotation result, and the annotation confidence of each chromosome region is calculated. The annotation confidence of each chromosome region is screened by a threshold method to obtain a low-confidence region, and image enhancement is performed on the low-confidence region in the slide sample image to obtain an image enhancement result. The image enhancement result is then re-annotated based on the pseudo-annotation generator to obtain a second annotation result.

5. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, characterized in that: Based on the annotation information in the third image set, the chromosome region in each slide sample image is cropped as a positive sample, and the region other than the chromosome region is used as a negative sample. All positive and negative samples are integrated and balanced based on the generator to obtain the fourth image set.

6. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, characterized in that: The coordinates of the ten-fold microscope chromosome region are mapped to the corresponding one hundred-fold microscope slide image through coordinate mapping to obtain the one hundred-fold microscope chromosome region, the one hundred-fold microscope chromosome region is downsampled, and the downsampled one hundred-fold microscope chromosome region and the ten-fold microscope chromosome region are spliced ​​with feature maps through Transformer to obtain a chromosome detection map.

7. The chromosome overlap detection method based on cross-modal feature fusion according to claim 1, characterized in that: In the training process of the chromosome overlap detection model, an overlap optimization loss is set as a loss function, wherein the overlap optimization loss includes an angle optimization loss, a shape constraint loss, and an overlap area loss. The formula of the angle optimization loss is as follows: in, Predict the angle of the overlapping detection box, is the angle of the real overlapping detection frame, Optimize loss for angle; The formula of the shape constraint loss is expressed as follows: in, To predict the width of the overlapping detection box, To predict the height of the overlapping detection box, is the width of the real overlapping detection box, The height of the real overlapping detection box is is the shape constraint loss; The formula for the overlap area loss is as follows: in, is the predicted overlapping area, is the actual overlapping area, is a measure of the consistency of features in the overlapping region, is the overlap area loss, for The weight of .

8. A chromosome overlap detection device based on cross-modal feature fusion, characterized in that: include: A brightness adjustment module, used to obtain multiple slide sample images as a first image set through a ten-fold mirror imaging system, and to adjust the pixel brightness of each slide sample image in the first image set to obtain a second image set, wherein the pixel brightness adjustment includes: performing noise reduction on overexposed pixel points in the slide sample and performing brightness reconstruction on underexposed pixel points in the slide sample; a labeling module, for automatically labeling chromosomes of each slide sample image in the second image set to obtain a third image set, then balancing the positive and negative samples in the third image set to obtain a fourth image set, and using the fourth image set to train a chromosome region detection model, wherein the positive samples are the chromosome regions of each slide sample image in the third image set, and the negative samples are the non-chromosome regions of each slide sample image in the third image set; A detection module is used to obtain a 10x microscope image to be tested, input the 10x microscope image to be tested into a chromosome region detection model to obtain a 10x microscope chromosome region, obtain the corresponding 100x microscope chromosome region based on the coordinates of the 10x microscope chromosome region, splice a feature map of the 10x microscope chromosome region with a feature map of the corresponding 100x microscope chromosome region to obtain a chromosome detection map, and input the chromosome detection map into a pre-trained chromosome detection model to obtain a chromosome overlap detection result.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute a chromosome overlap detection method based on cross-modal feature fusion as described in any one of claims 1-7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, wherein the process includes a chromosome overlap detection method based on cross-modal feature fusion according to any one of claims 1-7.

Images