Target local enhancement method and device suitable for dyed sample and storage medium

Abstract

The invention provides a target local enhancement method suitable for a dyed sample, and the method comprises the steps: obtaining a to-be-enhanced dyed sample image, carrying out the negative and positive binary classification recognition of the to-be-enhanced dyed sample image through a shared backbone network, and extracting the image features of the to-be-enhanced dyed sample image; if the to-be-enhanced dyed sample image is recognized as positive, fusing the image features based on a DAMO-YOLO GFPN, detecting targets in the to-be-enhanced dyed sample image, and generating an enhancement parameter vector corresponding to each target; and according to each detected target and the corresponding enhancement parameter vector, performing adaptive enhancement on the target area of the to-be-enhanced dyed sample image to obtain a local enhancement image of the dyed sample. The technical scheme provided by the invention can be accurately, efficiently and adaptively suitable for target local enhancement of the dyed sample.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical detection, in particular to a target local enhancement method and device suitable for dyed samples and a storage medium. BACKGROUND

[0002] In gynecological clinical detection, fluorescence staining technology is widely used for microbial observation of reproductive tract samples, especially for identification and analysis of specific flora such as lactobacillus, short bacillus, and coccus. The morphology and distribution of these microorganisms are of great significance for diagnosing vaginal microecological balance and judging infection types. At present, the conventional operation is to collect sample images under a 40-fold fluorescence microscope, and to identify target microorganisms by relying on manual interpretation or computer-aided analysis. However, due to the inherent characteristics of fluorescence signals such as fast decay, low signal-to-noise ratio, and susceptibility to environmental light interference, the boundaries of small-sized targets (such as coccus or short bacillus) are blurred and the contrast is insufficient in the imaging process, and their morphological characteristics are often not obvious in the image, which brings great difficulty to accurate identification and quantitative analysis.

[0003] Existing image enhancement methods mainly include global enhancement and local enhancement. Global enhancement methods (such as histogram equalization and contrast stretching) are simple to calculate, but they can easily cause over-enhancement of the background area and have limited effect on the detail recovery of small targets, and even may introduce global noise. Local enhancement methods (such as Retinex-based algorithms or local filtering techniques) can improve the contrast of local regions to some extent, but they usually rely on manual parameter setting, have poor adaptability, and are prone to produce artifacts or edge distortion in complex backgrounds or low signal-to-noise ratio images. In recent years, deep learning-based image enhancement methods have made significant progress in image denoising and super-resolution reconstruction, but these methods usually require large amounts of high-quality training data, and have large model parameter quantities and high computational resource consumption, which makes it difficult to deploy them in conventional medical equipment or low-configuration computing environments.

[0004] In addition, most existing methods uniformly process the entire image, and lack specific enhancement strategies for small targets. In gynecological sample images, small targets are sparsely distributed and occupy a small proportion of the total image area, so if the entire image is enhanced without distinction, not only the computational efficiency is low, but also the enhancement effect of key regions may be weakened. Although some target detection algorithms can locate target regions, they are not designed for low-quality fluorescence images, and have limited accuracy in low-contrast and small target detection tasks, and the detection and enhancement processes are usually separate, making it difficult to achieve end-to-end optimization.

[0005] Therefore, the inventors find that the prior art has the following main deficiencies: first, the enhancement effect on small target features is limited, and it is difficult to effectively improve the visual separability of the target in low signal-to-noise ratio fluorescent images; second, most enhancement methods have high computational complexity and are not suitable for real-time processing in medical sites or resource-limited environments; third, the model relies on a large amount of labeled data for training, which is costly and difficult to promote; fourth, there is a lack of an integrated solution that combines target detection and local enhancement, making it difficult to achieve precise and adaptive enhancement effect. SUMMARY

[0006] Therefore, the present application provides a target local enhancement method, device and storage medium suitable for stained samples to solve the technical problem that it is difficult to precisely and adaptively enhance the local of a stained sample in the prior art.

[0007] In a first aspect, the present application provides a target local enhancement method suitable for stained samples, comprising:

[0008] obtaining a to-be-enhanced stained sample image, performing positive-negative binary classification identification on the to-be-enhanced stained sample image using a shared backbone network, and extracting image features of the to-be-enhanced stained sample image;

[0009] if the to-be-enhanced stained sample image is identified as positive, fusing the image features based on a DAMO-YOLO-based GFPN, detecting targets in the to-be-enhanced stained sample image, and generating an enhancement parameter vector corresponding to each target;

[0010] According to the detected each target and the corresponding enhancement parameter vector, the target region of the to-be-enhanced stained sample image is adaptively enhanced to obtain a local enhancement image of the stained sample.

[0011] Preferably, the Backbone searched by the DAMO-YOLO-based MAE-NAS is used as the shared backbone network;

[0012] The shared backbone network comprises a lightweight binary classification branch; the lightweight binary classification branch identifies the confidence of no target in the to-be-enhanced stained sample image;

[0013] If the confidence of no target is greater than a preset threshold, the to-be-enhanced stained sample image is identified as negative;

[0014] Otherwise, the to-be-enhanced stained sample image is identified as positive.

[0015] Preferably, the DAMO-YOLO-based GFPN fuses the image features, detects the targets in the to-be-enhanced stained sample image, and generates an enhancement parameter vector corresponding to each target, comprising:

[0016] The DAMO-YOLO-based GFPN fuses the image features and generates a multi-scale feature map;

[0017] The first decoupling branch of the DAMO-YOLO performs regression, classification and confidence detection based on the multi-scale feature map, and outputs the detection result of each target, including a bounding box, a class and a confidence;

[0018] The second decoupling branch of the DAMO-YOLO generates an enhanced parameter vector corresponding to each target based on the image features combined with the multi-scale feature map.

[0019] Preferably, the target region of the staining sample image to be enhanced is adaptively enhanced according to each detected target and the corresponding enhanced parameter vector, to obtain a local enhanced image of the staining sample, including:

[0020] According to the position of each target on the feature map, a corresponding Gaussian weighted mask is generated;

[0021] The enhanced parameter vector corresponding to each target is mapped to a dynamic convolution kernel weight;

[0022] The feature of the region corresponding to each target on the feature map is transformed, and the formula is:

[0023] F mod =F o +∑ i M i ·(K i *F o )

[0024] Wherein, F mod is the transformed feature map, F o is the feature map before transformation, M i is the Gaussian weighted mask of target i, K i is the dynamic convolution kernel weight of target i, and i is the target identifier;

[0025] The feature map after region feature transformation is reconstructed by U-Net decoding to generate a local enhanced image of the staining sample.

[0026] Preferably, the multi-scale feature map includes a first scale feature map, a second scale feature map and a third scale feature map;

[0027] If the area of the target is greater than a preset area threshold A, the second scale feature map is selected as the feature map before transformation;

[0028] Otherwise, the feature transformation is performed on the region corresponding to the target with an area not greater than the preset threshold on the first scale feature map, and the feature transformation is performed on the region corresponding to the target with an area greater than the preset threshold on the second scale feature map, and the feature fusion is performed on the first scale feature map after the feature transformation and the second scale feature map after the feature transformation, and the formula is:

[0029]

[0030] Wherein, i∈m, m is a target set with an area greater than a preset threshold, j∈s, s is a target set with an area not greater than a preset threshold; is the dynamic convolution kernel weight of the first scale feature map F i generated by the enhanced parameter vector P m for the target i. is the dynamic convolution kernel weight of the second scale feature map F j generated by the enhanced parameter vector P s for the target j.

[0031] Preferably, the following formula is used to optimize target detection and target enhancement:

[0032]

[0033] Wherein, is the total loss, is the detection loss, is the enhancement loss, λ damo is the detection loss weight, λ enh is the enhancement loss weight;

[0034] is the quality focus loss, is the distribution focus loss, is the intersection over union loss, λ qfl is the quality focus loss weight, λ dfl is the distribution focus loss weight, λ IoU is the intersection over union loss weight;

[0035] is the weighted pixel-level fidelity loss, is the perception loss, is the semantic contrast constraint loss, λ pixel-weighted is the fidelity loss weight, λ perceptual is the perception loss weight, λ semantic is the semantic contrast constraint loss weight;

[0036]

[0037] Wherein, x is the real label; σ is the class confidence; is the intersection over union quality score; β is a tuning parameter;

[0038]

[0039] where y is the real coordinate; y i , y i+1 are the two integer coordinates closest to the real coordinate; S i , S i+1 is the predicted distribution value, and the probability of the real coordinate falling in the interval [y i , y i+1 ] after Softmax;

[0040]

[0041] where ρ is the Euclidean distance; (b, b gt ) are the center points of the predicted box and the real box, respectively; c is the diagonal length of the minimum closed box covering the two boxes; w, h, w gt , h gt are the width and height of the predicted box and the real box, respectively; v is the consistent aspect ratio item;

[0042]

[0043] where N obj represents the number of targets in the image, |B i | represents the pixels within the target i, A i represents the area of the target i, A min represents a preset area threshold; η represents a preset basic weight, ω i represents the target weight of the target i after correction; I' represents the enhanced image, and I represents the real image;

[0044]

[0045] where, represents a pre-trained feature extraction function;

[0046]

[0047] where, represents the feature vector extracted from the target i region of the enhanced image I', x represents the feature vector extracted from the target i region of the real image I, represents the feature vector extracted from the background region k of the real image I bg , and τ is a temperature coefficient.

[0048] Preferably, a conditional diffusion model is adopted to simulate fluorescence decay, optical blur and Poisson noise to generate synthetic data for model training, wherein the conditional information is a bounding box and a class label of the target.

[0049] In a second aspect, the present application provides a target local enhancement device suitable for staining samples, comprising:

[0050] An image input and feature extraction module is configured to obtain a staining sample image to be enhanced, perform positive-negative binary classification recognition on the staining sample image to be enhanced by using a shared backbone network, and extract image features of the staining sample image to be enhanced.

[0051] A positive image target detection and enhancement parameter generation module is configured to, if the staining sample image to be enhanced is recognized as positive, fuse the image features by using a DAMO-YOLO-based GFPN, detect targets in the staining sample image to be enhanced, and generate an enhancement parameter vector corresponding to each target.

[0052] A target region adaptive enhancement and image output module is configured to perform adaptive enhancement on a target region of the staining sample image to be enhanced according to each target and the corresponding enhancement parameter vector, and obtain a local enhancement image of the staining sample.

[0053] In a third aspect, the present application provides a computer device, comprising a memory and a processor, which are communicatively connected, and the memory stores computer instructions, and the processor implements the target local enhancement method suitable for staining samples by executing the computer instructions.

[0054] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the target local enhancement method suitable for staining samples.

[0055] The target local enhancement method suitable for staining samples, the device and the storage medium provided by the present application have at least the following beneficial effects:

[0056] The technical scheme provided in the application can realize precise target enhancement, significantly improve target visual separability, provide clear image support for accurate identification and quantitative analysis of gynecological reproductive tract microorganisms, and help diagnose vaginal microecological balance and judge infection types. The pain points of high calculation complexity and dependence on a large amount of labeled data are solved, a lightweight network architecture and a synthetic data generation strategy are adopted to reduce the consumption of computing resources and the dependence on real labeled data, the method can be efficiently deployed in a conventional medical device or a resource-limited environment, and the clinical detection practicability is improved. The limitation of separating the target detection and image enhancement processes is broken, an integrated processing scheme is constructed, end-to-end optimization is realized, the precision loss caused by process separation is avoided, and manual parameter setting is not required, the method is self-adaptive to different imaging conditions and target types, and the adaptability and robustness of the method are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical scheme in the specific embodiments or prior art of the present application, the drawings needed to be used in the description of the specific embodiments or prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor. It should be noted that the drawings in the following description are schematic and should not be understood as limiting the present application. In the drawings:

[0058] Figure 1 A schematic diagram of a target local enhancement method suitable for a stained sample in an embodiment of the present application is shown;

[0059] Figure 2 A schematic diagram of a target local enhancement device suitable for a stained sample in an embodiment of the present application is shown;

[0060] Figure 3 A schematic diagram of a computer device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creating any creative labor belong to the scope to be protected by the present application.

[0062] Embodiment 1

[0063] Please refer to Figure 1An embodiment of the present application provides a target local enhancement method suitable for dyeing samples, which can include the following steps.

[0064] A target local enhancement method suitable for dyeing samples, comprising:

[0065] In step S101, a dyeing sample image to be enhanced is acquired, a shared backbone network is used for positive and negative classification identification of the dyeing sample image to be enhanced, and image features of the dyeing sample image to be enhanced are extracted.

[0066] In step S102, if the dyeing sample image to be enhanced is identified as positive, the image features are fused based on a DAMO-YOLO GFPN, targets in the dyeing sample image to be enhanced are detected, and an enhancement parameter vector corresponding to each target is generated.

[0067] In step S103, according to each target detected and the corresponding enhancement parameter vector, a target region of the dyeing sample image to be enhanced is adaptively enhanced, and a local enhancement image of the dyeing sample is obtained.

[0068] Step S101 in the present application is used for image input and feature extraction. An image to be enhanced is obtained, which is usually obtained from fluorescence microscope imaging in gynecological clinical detection, and contains target microorganisms such as lactobacillus, short rod, and coccus, and a complex background. The Backbone searched by MAE-NAS (Masked Autoencoder based Neural Architecture Search) of DAMO-YOLO is used as a shared backbone network, that is, the DAMO-YOLO backbone network searched by MAE-NAS is used as a shared backbone network; this network contains an inverted residual and an attention mechanism, and through the optimization of the architecture by self-supervised pre-training and search cooperation, the backbone network has the characteristics of lightness and high efficiency, can reduce the consumption of computing resources while ensuring the accuracy of feature extraction, and is suitable for deployment in conventional medical equipment or low-configuration computing environments. On the basis of the shared backbone network, a lightweight binary classification branch is additionally attached, such as an intermediate-scale feature map that is globally averaged and flattened into a feature vector, which is processed by a fully connected layer and then output by a Sigmoid activation function without target confidence to identify the confidence of no target in the image to be enhanced. If the confidence of no target is greater than a preset threshold (which can be adjusted according to actual clinical data and detection requirements, and is generally determined through multiple experimental verifications, for example, set to 0.8), the image to be enhanced is identified as negative, indicating that there is no target microorganism that needs to be enhanced in the image, and the original image can be directly output; otherwise, it is identified as positive and enters the subsequent target detection and enhancement process. Whether the image is identified as positive or negative, the shared backbone network will extract the image features of the image to be enhanced, which contain key information such as texture, contour, and color of the image, providing data support for subsequent target detection and enhancement. In the present application, the lightweight binary classification branch is used to quickly screen positive and negative images, avoid subsequent invalid processing of negative images without target, improve the overall process efficiency, and reduce unnecessary computational consumption. The global features of the image are extracted in advance, providing a high-quality data basis for subsequent target detection and enhancement, ensuring the processing accuracy of subsequent steps, and reducing resource waste caused by repeated feature extraction.

[0069] Step S102 in the present application is used for positive image target detection and enhancement parameter generation. For images identified as positive, the global feature pyramid network (GFPN) based on DAMO-YOLO fuses image features extracted by the shared backbone network to generate multi-scale feature maps, including a first scale feature map, a second scale feature map, and a third scale feature map. The multi-scale feature maps can capture the features of targets of different sizes in the image. The first scale feature map has a high resolution and is suitable for capturing detailed features of small-sized targets (such as cocci); the second scale feature map has a moderate resolution and can effectively capture features of medium-sized targets (such as short rod-shaped bacteria); and the third scale feature map has a low resolution but a large receptive field, which is helpful for capturing overall features of large-sized targets (such as clustered lactobacilli).

[0070] In the present application, the first decoupled branch of DAMO-YOLO is used for target detection, and regression, classification, and confidence detection tasks are performed based on the generated multi-scale feature maps. Specifically,

[0071] The regression task: the position and size of the target in the image are determined by predicting the coordinates of the bounding box of the target. The coordinates of the bounding box are usually in pixels, accurate to each pixel point of the image, ensuring that the target area can be accurately framed.

[0072] The classification task: the detected target is classified to distinguish different types of microorganisms such as lactobacilli, small short rods, and cocci. The classification result is output in the form of a class label, such as "lactobacilli-1", "small short rod-2", and "cocci-3".

[0073] Confidence detection: the confidence of each detection result is calculated to measure the reliability of target detection. The higher the confidence, the greater the likelihood that the detected target is a real target. A confidence threshold (such as 0.5) is usually set to filter out detection results with a confidence lower than the threshold, reducing false positives. The final output includes the detection results of each target, including the bounding box, the class, and the confidence.

[0074] In the present application, the second decoupled branch of DAMO-YOLO is used for generating an enhancement parameter vector, which is based on image features extracted by the shared backbone network in combination with multi-scale feature maps to generate an enhancement parameter vector corresponding to each target. The enhancement parameter vector contains key parameters for subsequent target region adaptive enhancement, such as contrast adjustment coefficients, brightness compensation values, and sharpening degree parameters. These parameters are dynamically generated based on the class, size, and morphological features of the target as well as the surrounding background information, ensuring that each target can be optimally enhanced. For example, for a cocci target with a blurred boundary and low contrast, the generated enhancement parameter vector will contain a higher contrast adjustment coefficient and an appropriate sharpening degree parameter to highlight its boundary and detailed features.

[0075] The present application uses the GFPN of DAMO-YOLO to perform multi-scale feature fusion, effectively captures the features of different size targets (such as coccus, short rod-shaped bacteria, lactobacillus), solves the problem of low detection accuracy of small targets, and improves the target positioning and classification accuracy. The first decoupling branch accurately outputs the target boundary box, class and confidence, provides a clear target area for subsequent enhancement, avoids indiscriminate enhancement, and ensures that the enhancement resources are concentrated on effective targets. The second decoupling branch dynamically generates an enhancement parameter vector adapted to each target without human intervention, solves the problem of parameter dependence on artificial setting and poor adaptability in traditional methods, and realizes individualized adaptation of the enhancement strategy.

[0076] Step S103 in the present application is used for target region adaptive enhancement and image output.

[0077] According to the position of each target on the feature map, a corresponding Gaussian weighted mask is generated. The role of the Gaussian weighted mask is to perform weighted processing on the target region, so that the target center region obtains a higher weight and the edge region obtains a lower weight. In this way, in the subsequent enhancement process, the core features of the target can be highlighted, and the artifact problem caused by excessive enhancement of the target edge region can be avoided. The generation of the Gaussian weighted mask is based on the Gaussian function, and the standard deviation can be adjusted according to the size of the target. The larger the target, the larger the standard deviation, and the wider the coverage of the mask.

[0078] Each target corresponds to an enhancement parameter vector, which is mapped to a dynamic convolution kernel weight. The dynamic convolution kernel weight is adaptively adjusted according to the specific features and enhancement requirements of the target. Different types and sizes of targets will correspond to different dynamic convolution kernel weights. For example, for lactobacillus, which has a relatively regular shape, the dynamic convolution kernel weight may focus more on smoothing processing to remove image noise; for small short rods, which have an elongated shape, the dynamic convolution kernel weight may focus more on edge enhancement to highlight their shape features.

[0079] In the present application, feature transformation is divided into general case and area consideration case.

[0080] In general, the feature transformation is performed on the region corresponding to each target on the feature map, and the formula is:

[0081] F mod =F o +∑ i M i ·(K i *F o )

[0082] Where F mod is the transformed feature map, F o is the feature map before transformation, M i is the Gaussian weighted mask of target i, Ki is the dynamic convolution kernel weight of the target i, i is the target identifier. The formula represents that the transformed feature map is obtained by performing convolution operation on the dynamic convolution kernel weight and the feature map before transformation, combining with the weighted processing of the Gaussian weighted mask, and finally adding the feature map before transformation, so as to realize the enhancement of the target region features.

[0083] Considering the area of the target, if the area of the target is greater than a preset area threshold A (the determination of the preset area threshold needs to be combined with the size distribution of the target in the actual image, and by statistically analyzing the area data of the target in a large number of clinical sample images, a critical value that can effectively distinguish large and small targets is selected, for example, 100x100 pixels), the second scale feature map is selected as the feature map before transformation; otherwise, the feature transformation is performed on the region corresponding to the target with an area not greater than the preset threshold on the first scale feature map, and the feature transformation is performed on the region corresponding to the target with an area greater than the preset threshold on the second scale feature map, and the feature fusion is performed between the first scale feature map after feature transformation and the second scale feature map after feature transformation, and the formula is:

[0084]

[0085] Wherein, i∈m, m is a target set with an area greater than a preset threshold, j∈s, s is a target set with an area not greater than a preset threshold; is the dynamic convolution kernel weight of the target i generated by the enhancement parameter vector P i corresponding to the second scale feature map F m ; is the dynamic convolution kernel weight of the target j generated by the enhancement parameter vector P j corresponding to the first scale feature map F s ; Upsample represents the up-sampling operation, which is used to improve the resolution of the first scale feature map to that of the second scale feature Figure 1 , so as to perform feature fusion. This way of selecting different scale feature maps for feature transformation and fusion according to the area of the target can better adapt to the enhancement needs of targets of different sizes and improve the enhancement effect of small targets.

[0086] In step S103, the image is reconstructed and output. Specifically, the feature map after regional feature transformation is reconstructed by U-Net decoding to generate a local enhanced image of the stained sample. The U-Net decoder has strong image reconstruction capability, which can restore the feature map after feature transformation to a high-resolution image while retaining the detailed features and spatial position information of the target. In the finally output local enhanced image, the boundary of the target microorganism is clear and has high contrast, which is convenient for clinicians to accurately identify and quantitatively analyze.

[0087] In the present application, the Gaussian weighting mask focuses the enhancement on the target core area, reduces the artifacts caused by excessive enhancement of the edge, and avoids background interference, further improving the distinction between the target and the background. According to the target area, the corresponding scale feature map is transformed and fused, and the enhancement effect of targets of different sizes is optimized, especially the presentation of small target details, solving the problem of limited enhancement effect of small targets. The U-Net decoding reconstruction guarantees the resolution and detail integrity of the enhanced image, and the output of the local enhancement image meets the clinical precise interpretation demand, directly improving the efficiency of microorganism identification and analysis.

[0088] In the present application, the DAMO-YOLO GFPN is used for multi-scale feature fusion and target detection, which can accurately identify the target microorganisms in the stained sample image, and combined with the dynamically generated enhancement parameter vector and the Gaussian weighting mask, the adaptive enhancement of the target area is realized, the visual distinguishability of the target is effectively improved, and the accuracy of clinical diagnosis is improved. The lightweight shared backbone network obtained by MAE-NAS search reduces the parameter quantity and calculation quantity of the model, and at the same time, according to the characteristics of sparse distribution of small targets, only the target area is enhanced, avoiding invalid calculation of the whole image, improving the calculation efficiency, and being suitable for real-time processing in medical field or resource-limited environment. Without manual setting of enhancement parameters, the enhancement parameter vector is dynamically generated according to the specific characteristics of the target, which can adapt to different types, different sizes and different imaging conditions of the stained sample image, and has strong adaptability and robustness.

[0089] Through the technical scheme of the present application, accurate targeted enhancement can be realized, the visual distinguishability of the target is significantly improved, clear image support is provided for accurate identification and quantitative analysis of gynecological reproductive tract microorganisms, and help is provided for diagnosis of vaginal microecological balance and judgment of infection type. The pain points of high computational complexity and dependence on a large amount of labeled data are solved, the lightweight network architecture and the synthetic data generation strategy are adopted, the computational resource consumption is reduced, the dependence on real labeled data is reduced, and the present application can be efficiently deployed in conventional medical equipment or resource-limited environment, improving the clinical detection practicability. Breaking the limitation of separation of target detection and image enhancement process, an integrated processing scheme is constructed, end-to-end optimization is realized, precision loss caused by process separation is avoided, and manual parameter setting is not required, which is self-adaptive to different imaging conditions and target types, and the adaptability and robustness of the enhancement scheme are improved.

[0090] In the present application, in order to optimize the target detection and target enhancement effect at the same time, the following composite loss function is used:

[0091]

[0092] Wherein, is the total loss, used to measure the performance of the whole model; is the detection loss, used to optimize the accuracy of target detection; For the enhancement loss, the enhancement effect for the optimization target area; λ damo For the detection loss weight, λ enh For the enhancement loss weight; the values of both need to be adjusted according to the actual training effect, and are generally determined through cross-validation, for example, set λ damo = 0.6, λ enh = 0.4 to balance the optimization objectives of the two tasks of detection and enhancement.

[0093] Detection loss

[0094] Wherein, Quality focus loss, Distribution focus loss, Intersection over union loss; λ qfl Quality focus loss weight, λ dfl Distribution focus loss weight, λ IoU Intersection over union loss weight, which needs to be adjusted according to the influence of different loss functions on the detection performance, for example, set λ qfl = 0.3, λ dfl = 0.4, λ IoU = 0.3 to ensure that each loss component can work together to improve the target detection effect.

[0095] Quality focus loss

[0096]

[0097] Wherein, x is the real label, if it is a target, x = 1, otherwise x = 0); σ is the class confidence; Intersection over union quality score, used to measure the overlap degree of the predicted bounding box and the real bounding box; β is an adjustment parameter, generally taking 2, used to adjust the punishment degree of the loss function to different confidence samples, so that the model pays more attention to high-quality detection samples.

[0098] Distribution focus loss

[0099]

[0100] Wherein, y is the real coordinate; y i , y i+1 are the two integer coordinates closest to the real coordinate; S i , S i+1 are the predicted distribution values, and the real coordinate falls in the interval [y i , y i+1the probability that the target is not in the bounding box; the loss function improves the positioning accuracy of the target bounding box by optimizing the coordinate distribution.

[0101] Intersection over Union loss

[0102]

[0103] where ρ is the Euclidean distance, used to calculate the distance between the center points of the predicted box and the real box; (b, b gt ) are the center points of the predicted box and the real box, respectively; c is the diagonal length of the minimum closed box covering the two boxes; w, h, w gt , h gt are the width and height of the predicted box and the real box, respectively;

[0104] is the aspect ratio consistency term, used to measure the difference between the predicted box and the real box in terms of aspect ratio; is used to adjust the weight of the aspect ratio consistency term in the loss function. The Intersection over Union loss can directly reflect the overlap between the predicted box and the real box, effectively optimizing the positioning accuracy of target detection.

[0105] Enhancement loss

[0106]

[0107] where, is the weighted pixel-level fidelity loss, is the perception loss, is the semantic contrast constraint loss, λ pixel-weighted is the fidelity loss weight, λ perceptual is the perception loss weight, λ semantic is the semantic contrast constraint loss weight; its value needs to be adjusted according to the evaluation index of the enhancement effect (such as peak signal-to-noise ratio, structural similarity index, etc.), for example, set λ pixel-weighted = 0.5, λ perceptual = 0.3, λ semantic = 0.2, to achieve comprehensive optimization of image pixel fidelity, visual perception quality, and semantic contrast.

[0108] Weighted pixel-level fidelity loss

[0109]

[0110] where N obj represents the number of targets in the image, and |B i | represents the pixels within target i. denotes the target weight of the corrected target i, η denotes the preset base weight (e.g., set to 0.5), A i denotes the area of the target i, A min denotes the preset area threshold (consistent with the target area judgment threshold described above); I' denotes the enhanced image, and I denotes the real image (i.e., the ideal enhanced effect image). The loss function assigns different weights to targets of different sizes, so that the model pays more attention to the pixel fidelity of small targets during the enhancement process and reduces the information loss of small targets during the enhancement process.

[0111] perception loss

[0112]

[0113] wherein, denotes the selected feature layer set, usually the middle feature layers of the pre-trained network, which can capture the high-level semantic information of the image; C l , H l , W l denote the channel number, height, and width of the l-th feature map, respectively; denotes the pre-trained feature extraction function for extracting the l-th layer features from the image; I gt denotes the real image. The perception loss compares the difference between the enhanced image and the real image in the high-level feature space, so that the enhanced image is closer to the real image in visual perception, thereby improving the image quality.

[0114] semantic contrast constraint loss

[0115]

[0116] wherein, denotes the feature vector extracted from the target i region of the enhanced image I', denotes the feature vector extracted from the target i region of the real image I, denotes the feature vector extracted from the background region k of the real image I bg , and K denotes the number of background regions, which can be set according to the complexity of the image background; sim(·, ·) denotes a similarity calculation function between feature vectors, such as cosine similarity; τ is a temperature coefficient, usually set to 0.1, for adjusting the steepness of the similarity distribution. The loss function constrains the similarity between the enhanced target region features and the real target region features, as well as the difference between the background region features, enhances the semantic contrast between the target and the background, and makes the target more prominent in the enhanced image.

[0117] The composite loss function optimizes target detection and enhancement effect at the same time, balances detection accuracy and enhancement quality, avoids overall performance imbalance caused by single task optimization, and improves comprehensive processing capability of the scheme.

[0118] Due to the scarcity of high-quality and completely labeled fluorescent image data of stained samples in actual clinical detection, and the high cost of obtaining the data, in order to meet the demand of model training for large-scale data, a conditional diffusion model is used to simulate fluorescent decay, optical blur and Poisson noise to generate synthetic data for model training.

[0119] Conditional information: the conditional information of the conditional diffusion model is the bounding box and the class label of the target. The bounding box information is used to determine the position and size of the target in the synthetic image, and the class label is used to specify the type of the target (such as lactobacillus, small short rod, coccus, etc.), so as to ensure that the synthetic data is consistent with the real clinical data in terms of target distribution and category.

[0120] Simulation of degradation process:

[0121] Fluorescent decay: according to the decay law of fluorescent signal, the decay process of fluorescent signal under different time and different excitation light intensity is simulated, so that the brightness of the target in the synthetic image presents a reasonable decay change, which is close to the characteristics of the fluorescent signal in the real fluorescent image.

[0122] Optical blur: considering the imaging characteristics of the microscope optical system, the image blur effect caused by factors such as lens focal length deviation and optical diffraction is simulated, and synthetic images with different blur degrees are generated by adjusting the size and shape of the blur kernel, so as to enhance the adaptability of the model to blurred images.

[0123] Poisson noise: due to the randomness of photon counting in the process of fluorescence imaging, Poisson noise will be generated. According to the statistical characteristics of Poisson distribution, Poisson noise is added to the synthetic image, so that the noise distribution of the synthetic data is consistent with that of the real fluorescent image, and the de-noising ability and robustness of the model are improved.

[0124] Data diversity: by adjusting the parameters of the conditional diffusion model (such as diffusion step number, noise intensity, etc.) and the conditional information of the target (such as bounding box size, position, category combination, etc.), a large number of synthetic data with different target morphologies, different background complexities and different degradation degrees are generated, which enriches the diversity of the training data set and improves the generalization ability of the model. The generated synthetic data needs to be quality inspected to ensure that it is similar to the real clinical data in terms of visual effect and statistical characteristics, and can be used for model training.

[0125] The conditional diffusion model generates synthetic training data, effectively solves the problem of scarcity of high-quality labeled clinical data, reduces the dependence of the model on real labeled data, reduces the cost of data acquisition, and facilitates the popularization and application of the scheme.

[0126] In addition, the target detection and the local enhancement are organically combined in the present application, end-to-end optimization is realized, the precision loss caused by the separation of the detection and the enhancement process is avoided, and the overall performance stability is improved.

[0127] In an optional implementation, the shared backbone network extracts a global semantic feature vector of an image at the same time of performing the Yin-Yang binary classification recognition. Specifically, a high-dimensional feature map is extracted from a last convolutional layer of the shared backbone network, an initial feature vector is obtained after global average pooling, and the initial feature vector is processed to obtain the global semantic feature vector F cls .

[0128] In the feature map after the regional feature transformation, the global semantic feature vector is used as conditional information for decoding reconstruction to guide the generation of the local enhanced image of the stained sample. Specifically, the global semantic feature vector is mapped into modulation parameters {γ l ,β l} of multiple scales, l represents a level of decoding, γ is a scaling parameter, and β is an offset parameter; in the up-sampling of the decoding reconstruction, the modulation parameters of the corresponding scale are fused with the decoding features in a global semantic feature vector instance normalization or feature splicing manner. The modulation of the input features is as follows: is a mean value of the feature map, and σ(·) is a standard deviation of the feature map. The global semantic feature vector guides the enhancement reconstruction of the local region of the image, and ensures that the enhanced image can maintain local details while being consistent with the overall style.

[0129] A classification feature consistency loss is added to the enhancement loss

[0130] Φ cls (I′) represents a global semantic feature vector extracted from the enhanced image I′ by the shared backbone network, ensuring that the images before and after enhancement are consistent in the feature space, and avoiding semantic distortion introduced in the enhancement process.

[0131] Embodiment 2

[0132] The present embodiment provides a target local enhancement device suitable for a stained sample, which is applied to the target local enhancement method suitable for a stained sample provided in Embodiment 1. As shown in FIG. 2, the target local enhancement device provided in an embodiment of the present application can include the following modules. Figure 2

[0133] ​The image input and feature extraction module is configured to obtain a to-be-enhanced staining sample image, perform positive-negative binary classification identification on the to-be-enhanced staining sample image by using a shared backbone network, and extract image features of the to-be-enhanced staining sample image.

[0134] The positive image target detection and enhancement parameter generation module is configured to, if the to-be-enhanced staining sample image is identified as positive, perform fusion on the image features based on a DAM0-YOL0 GFPN, detect targets in the to-be-enhanced staining sample image, and generate an enhancement parameter vector corresponding to each target.

[0135] The target region adaptive enhancement and image output module is configured to perform adaptive enhancement on a target region of the to-be-enhanced staining sample image according to each detected target and the corresponding enhancement parameter vector, and obtain a local enhancement image of the staining sample.

[0136] The target local enhancement device for staining samples provided by the embodiments of the present application can be applied to the target local enhancement method for staining samples provided in Embodiment 1 described above, and the related details are referred to the method embodiments described above, which have similar implementation principles and technical effects, and will not be described here.

[0137] Embodiment 3

[0138] Please refer to Figure 3 An embodiment of the present application further provides a computer device, which can be a desktop computer, a notebook computer, a palm computer, a cloud server or the like. The computer device can include, but is not limited to, a processor and a memory. The processor and the memory can be connected through a bus or other means.

[0139] An embodiment of the present application further provides a computer readable storage medium for storing a computer program, and the computer program is executed by a processor to implement the method in the method embodiments described above.

Claims

1. A method for targeted local enhancement of a sample suitable for staining, characterized in that, The method comprises the following steps: obtaining a to-be-enhanced staining sample image, performing positive and negative classification identification on the to-be-enhanced staining sample image by using a shared backbone network, and extracting image features of the to-be-enhanced staining sample image; if the to-be-enhanced staining sample image is identified as positive, fusing the image features by using a DAMO-YOLO-based GFPN, detecting targets in the to-be-enhanced staining sample image, and generating an enhanced parameter vector corresponding to each target; performing adaptive enhancement on a target region of the to-be-enhanced staining sample image according to each detected target and the corresponding enhanced parameter vector, and obtaining a local enhanced image of the staining sample.

2. The target local enhancement method for a staining sample according to claim 1, wherein: the Backbone searched by the DAMO-YOLO-based MAE-NAS is used as the shared backbone network; the shared backbone network comprises a lightweight binary classification branch; the lightweight binary classification branch identifies a confidence degree of a target-free region in the to-be-enhanced staining sample image; if the confidence degree of the target-free region is greater than a preset threshold, the to-be-enhanced staining sample image is identified as negative; otherwise, the to-be-enhanced staining sample image is identified as positive.

3. A method for targeted local enhancement of a sample suitable for staining according to claim 2, characterized in that: the DAMO-YOLO-based GFPN fuses the image features, detects the targets in the to-be-enhanced staining sample image, and generates the enhanced parameter vector corresponding to each target, which comprises the following steps: the DAMO-YOLO-based GFPN fuses the image features and generates a multi-scale feature map; a first decoupling branch of the DAMO-YOLO is used to perform regression, classification and confidence detection based on the multi-scale feature map, and output the detection result of each target, including a bounding box, a category and a confidence degree; a second decoupling branch of the DAMO-YOLO is used to generate the enhanced parameter vector corresponding to each target based on the image features in combination with the multi-scale feature map.

4. A method for targeted local enhancement of a sample suitable for staining according to claim 3, characterized in that, the adaptive enhancement on the target region of the to-be-enhanced staining sample image according to each detected target and the corresponding enhanced parameter vector to obtain the local enhanced image of the staining sample comprises the following steps: a corresponding Gaussian weighting mask is generated according to the position of each target on the feature map; the enhanced parameter vector corresponding to each target is mapped into a dynamic convolution kernel weight; feature transformation is performed on the region corresponding to each target on the feature map, and the formula is: F mod = F o +∑ i M i ·(K i *F o ) Wherein, F mod is the transformed feature map, F o is the feature map before transformation, M i is the Gaussian weighted mask of the target i, K i is the dynamic convolution kernel weight of the target i, and i is the target identifier. the feature map after the region feature transformation is reconstructed by using a U-Net decoding to generate the local enhanced image of the staining sample.

5. A method for targeted local enhancement of a sample suitable for staining according to claim 4, characterized in that, Further comprising: the multi-scale feature map comprises a first scale feature map, a second scale feature map and a third scale feature map; if the area of a target is greater than a preset area threshold A, the second scale feature map is selected as the feature map before transformation; otherwise, feature transformation is performed on the region corresponding to the target with an area not greater than the preset threshold on the first scale feature map, and feature transformation is performed on the region corresponding to the target with an area greater than the preset threshold on the second scale feature map; feature fusion is performed on the first scale feature map after the feature transformation and the second scale feature map after the feature transformation after upsampling, and the formula is: Wherein, i∈m, m is a target set with an area greater than a preset threshold, j∈s, s is a target set with an area not greater than a preset threshold; The dynamic convolution kernel weight of the corresponding first scale feature map F i generated for the target i from the enhanced parameter vector P m ; The dynamic convolution kernel weight of the corresponding second scale feature map F j generated for the target j from the enhanced parameter vector P s ; 6. A method for targeted local enhancement of a sample suitable for staining according to claim 5, characterized in that, The following formula is used to optimize target detection and target enhancement: wherein, is the total loss, is the detection loss, is the augmentation loss, λ damo is the detection loss weight, λ enh is the augmentation loss weight; is a quality focus loss, is a distribution focus loss, is an intersection over union loss, λ qfl is a quality focus loss weight, λ dfl is a distribution focus loss weight, λ IoU is an intersection over union loss weight; LwPF is a weighted pixel-level fidelity loss, Lpercept is a perceptual loss, Lsc is a semantic contrast constraint loss, λ pixell-weighted LwPF is a fidelity loss weight, λ perceptual Lpercept is a perceptual loss weight, λ semantic Lsc is a semantic contrast constraint loss weight; where x is the true label; σ is the class confidence; is the intersection over union quality score; β is a tuning parameter; where y is the real coordinate; y i , y i+1 are the two integer coordinates closest to the real coordinate; S i , S i+1 is the predicted distribution value, the probability that the real coordinate falls in the interval [y i , y i+1 ] after Softmax. where p is the Euclidean distance; (b, b gt ) are the center points of the predicted and ground-truth boxes, respectively; c is the diagonal length of the minimum enclosing box covering both boxes; w, h, w gt , h gt are the width and height of the predicted and ground-truth boxes, respectively; v is the aspect ratio consistent term; where N obj denotes the number of targets in the image, |B i denotes the pixels within target i, A i denotes the area of target i, A min denotes a preset area threshold; η denotes a preset base weight, ω i denotes the target weight of the corrected target i; I' denotes the enhanced image, and I denotes the real image. wherein, denotes a pre-trained feature extraction function; wherein, represents a feature vector extracted from the target i region of the real image I, represents a feature vector extracted from the target i region of the real image I, represents a feature vector extracted from the background k region of the real image I bg region k, and τ is a temperature coefficient.

7. A method for targeted local enhancement of a sample suitable for staining according to claim 6, characterized in that, Also includes: A conditional diffusion model is used to simulate fluorescence decay, optical blur and Poisson noise to generate synthetic data for model training; wherein the conditional information is the boundary box and the class label of the target.

8. A target local enhancement device adapted to stain a sample, characterized in that, Includes: An image input and feature extraction module is used to obtain a to-be-enhanced staining sample image, perform positive and negative binary classification recognition on the to-be-enhanced staining sample image using a shared backbone network, and extract image features of the to-be-enhanced staining sample image; A positive image target detection and enhancement parameter generation module is used to, if the to-be-enhanced staining sample image is identified as positive, fuse the image features based on DAM0-YOLO's GFPN, detect targets in the to-be-enhanced staining sample image, and generate an enhancement parameter vector corresponding to each target; A target region adaptive enhancement and image output module is used to, according to each detected target and the corresponding enhancement parameter vector, perform adaptive enhancement on the target region of the to-be-enhanced staining sample image to obtain a local enhancement image of the staining sample.

9. A computer device, comprising: Includes: A memory and a processor, which are communicatively connected between each other, the memory stores computer instructions, and the processor executes the computer instructions to implement the target local enhancement method for staining samples according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the target local enhancement method for staining samples according to any one of claims 1-7.

Images