Deep learning based slice defect re-scan control method

Abstract

This invention discloses a deep learning-based method for controlling the rescanning of slicing defects, belonging to the field of digital pathological scanning and medical image processing technology. To address the problem of time-consuming, missed detections, and high rescanning costs associated with manually reviewing and rescanning the entire slice for defects such as defocusing, local blurring, wrinkles, bubbles, and scalpel marks during scanning, this invention divides the pre-scanned image or online scanned image into blocks and inputs quality defects into a multi-task neural network to output defect detection results and quality assessment results, thereby determining the target region of interest. A gain prediction neural network outputs the quality gain and confidence lower bound of candidate control actions relative to the baseline action. A hierarchical model predictive controller then continuously optimizes the scanning control commands under scanning resource and risk constraints, executing local refocusing, local rescanning, and scanning parameter adjustments, and fusing these to generate an updated full-field-of-view image. This achieves the technical effect of increasing the proportion of usable full-field-of-view images and reducing the rescanning rate and rescanning time.

Description

Technical Field

[0001] This invention relates to the field of digital pathological scanning and medical image processing, and in particular to a method for controlling the rescanning of slide defects based on deep learning. Background Technology

[0002] With the development of digital pathology technology, full-field digital scanning of pathological slides has gradually become an important foundation for clinical diagnosis, remote consultation, and scientific research analysis. Existing digital pathology scanning systems typically employ a pre-scan followed by a fine scan workflow, where the scanner generates a full-field image through stage movement, autofocus, and exposure control. Simultaneously, to improve imaging reliability, the industry has gradually introduced automated quality control methods based on image quality evaluation, such as focus sharpness evaluation, local blur detection, and artifact detection (wrinkles, bubbles, etc.), prompting for review or triggering a rescan when abnormalities are detected. In recent years, deep learning methods, due to their advantages in defect identification and segmentation, have been used for defect detection, quality scoring, and usability assessment of scanned images.

[0003] However, existing technologies still have the following shortcomings:

[0004] 1. After a defect is detected, the common processing method is still manual review or full-area rescan, which makes it difficult to accurately locate the defect area and rescan or refocus only the local area, resulting in long rescan time and high resource consumption.

[0005] 2. Defect detection and scanning control are often isolated from each other, lacking a mechanism to further transform defect location results and quality assessment results into executable scanning control strategies, making it difficult to adaptively adjust the rescan range, focusing strategy and exposure parameters for different defect types.

[0006] 3. Rescanning decisions often rely on fixed thresholds or empirical rules, lacking quantitative assessment of whether performing a certain rescanning action is worthwhile. They also lack global optimization and risk control methods under resource constraints such as scanning time and rescanning area, which can easily lead to problems such as missed detections, over-rescanning, or uncertain rescanning benefits.

[0007] Therefore, there is a need for a method to control the rescanning of slab defects that can overcome the shortcomings of the existing technology. Summary of the Invention

[0008] One objective of this invention is to propose a deep learning-based method for controlling the rescanning of slide defects. Addressing the problems of existing technologies where defects such as defocusing, local blurring, wrinkling, and bubbles in digital pathology scanning often require manual review and full-scan rescanning, and where defect detection results are difficult to directly translate into scanning control strategies, and optimal rescanning decisions are difficult to achieve under resource constraints such as rescanning time and area, this invention proposes a method to acquire the initial scan image and scanning parameters and process them in blocks. A quality defect joint multi-task neural network is used to output defect detection results and quality assessment results to determine the target region of interest and construct state features. Gain prediction is performed on candidate control actions to obtain the expected quality gain and confidence lower bound relative to the baseline action. Then, a hierarchical model predicts the controller to continuously optimize and issue scanning control commands under resource and risk constraints, executing local refocusing, local rescanning, and parameter adjustment, and fusing them to generate an updated full-view image. This invention achieves the technical effects of increasing the proportion of usable full-view images, reducing the rescanning rate and rescanning time, and reducing missed detections.

[0009] This invention provides a deep learning-based method for controlling the rescanning of slice defects, comprising:

[0010] S1. Obtain the initial scan image of the slice to be scanned and the corresponding scan parameter information. Divide the initial scan image into blocks according to a preset block division rule to obtain an image block set. S2. Input the image block set and scan parameter information into a joint multi-task neural network for quality defects, and output the defect detection results and quality assessment results corresponding to each image block. S3. Based on the defect detection results and quality assessment results, determine at least one target region of interest (ROI) and construct the corresponding state features. S4. Generate a corresponding set of candidate control actions for each target ROI. Jointly encode the state features and candidate control actions and input them into a gain prediction neural network. Output the quality gain prediction results of each candidate control action relative to the baseline action, including... The expected value of quality gain and the confidence lower bound used for risk constraints are used as the baseline action, which is not to perform local rescanning and scanning parameter adjustment in the target region of interest; S5, the target region of interest, the candidate control action set, and the quality gain prediction results are input into the hierarchical model prediction controller, and rolling optimization is performed within the preset prediction time domain to determine the optimized control action under the premise of satisfying the preset scanning resource constraints, and risk constraints are applied based on the confidence lower bound; S6, the optimized control action is converted into a scanning control command and sent to the digital pathology scanning system, and local rescanning and scanning parameter adjustment corresponding to the optimized control action are performed in the target region of interest to obtain a local rescanning image, which is then fused with the initial scanning image to generate an updated full-field image.

[0011] Optionally, S1 includes:

[0012] Obtain a pre-scanned image or an online scanned image of the slice to be scanned as the initial image;

[0013] Obtain scanning parameter information corresponding to the initial image, the scanning parameter information including focal plane position parameters and exposure parameters;

[0014] The block positions are determined on the initial image according to the preset block division rules, and the initial image is cropped according to the block positions to obtain the image block set containing multiple image blocks;

[0015] Furthermore, the preset segmentation rules include: first, performing tissue region detection on the initial scan image to generate a tissue mask; determining segmentation positions only within the area covered by the tissue mask; and adaptively adjusting the image block size or image block overlap ratio according to the tissue proportion of the tissue mask.

[0016] Optionally, S2 includes:

[0017] Each image block in the image block set is jointly encoded with the scanning parameter information to form a network input, and the network input is input into the quality defect joint multi-task neural network.

[0018] The quality defect joint multi-task neural network outputs the defect detection result and the quality assessment result corresponding to each image block in the image block set. The defect detection result includes a defect localization map corresponding to each pixel of the image block, a defect category and a defect level determined based on the defect localization map, and the quality assessment result includes a quality heatmap corresponding to each pixel of the image block.

[0019] Optionally, S3 includes:

[0020] Map the defect location map and the quality heat map to the coordinate system of the initial image;

[0021] In the defect location map, pixel regions with defect confidence scores not less than a first threshold are selected, and in the quality heatmap, pixel regions with quality scores not greater than a second threshold are selected. The above two types of pixel regions are then combined and fused to obtain candidate regions.

[0022] Connectivity extraction is performed on the candidate regions to obtain at least one candidate region of interest, and candidate regions of interest that overlap or have a spacing less than a preset distance threshold are merged to obtain the target region of interest set;

[0023] For each target region of interest in the set of target regions of interest, a state feature is calculated based on the defect detection result, the quality assessment result, and the scanning parameter information within the target region of interest. The state feature includes the location range of the target region of interest, the area of ​​the target region of interest, the defect category, the defect level, and the quality heatmap statistics within the target region of interest, thus obtaining the set of state features.

[0024] Optionally, S4 includes:

[0025] For each target region of interest in the set of target regions of interest, the allowed action type is determined based on the state characteristics corresponding to the target region of interest, and at least one action parameter is generated within the preset parameter range corresponding to each action type to obtain a set of candidate control actions corresponding to the target region of interest, and further, a set of candidate control actions corresponding to the set of target regions of interest is obtained.

[0026] The action parameters include local rescan range parameters, refocus step size parameters, exposure adjustment amount parameters, focal plane adjustment amount parameters, and focal layer number parameters.

[0027] For each candidate control action in the candidate control action set corresponding to each target region of interest in the candidate control action set group, the action type and action parameters of the candidate control action are vectorized and encoded to obtain action features. The state features in the state feature set corresponding to the target region of interest are combined with the action features to form a gain prediction input.

[0028] The gain prediction input is fed into the gain prediction neural network, and the gain prediction result corresponding to the candidate control action is output. The gain prediction result includes the expected value of the quality gain and the standard deviation of the quality gain. The confidence lower bound of the quality gain is calculated based on the expected value of the quality gain and the standard deviation of the quality gain.

[0029] The confidence lower bound is equal to the expected value of the quality gain minus the product of the preset risk coefficient and the standard deviation of the quality gain.

[0030] Optionally, S5 includes:

[0031] The target region of interest set, the candidate control action set group, and the gain prediction result are input into the hierarchical model prediction controller.

[0032] The upper-level model predicts the expected value of the quality gain corresponding to each candidate control action in the gain prediction result. A discrete optimization problem is constructed with the upper-level objective function as maximizing the total quality gain and minimizing the rescan cost. Under the premise of satisfying the preset scan constraints, the action type corresponding to the target region of interest is determined from the candidate control action set group and the candidate control action set corresponding to each target region of interest.

[0033] The rescanning cost includes the rescanning area cost and the rescanning time cost, and the preset scanning constraints include the maximum rescanning area constraint and the maximum rescanning time constraint.

[0034] The lower-level model predicts and controls the action type corresponding to each target region of interest. The expected value of the quality gain is used as the optimization target to continuously optimize the action parameters corresponding to the action type. The risk constraint is applied based on the confidence lower bound of the quality gain in the gain prediction result so that the optimized action parameters meet the risk constraint. The risk constraint includes that the confidence lower bound is not less than a preset threshold.

[0035] Output optimized control actions that satisfy the preset scanning constraints and the risk constraints;

[0036] Furthermore, the rescanning cost also includes a displacement cost, which is used to characterize the distance or time the stage moves between different target regions of interest; the upper-level model predictive control further minimizes the displacement cost when determining the action type to reduce the total rescanning time.

[0037] Optionally, S6 includes:

[0038] For each target region of interest in the set of target regions of interest, the location range of the target region of interest, the action type and action parameters corresponding to the optimized control action are determined according to the optimized control action corresponding to the target region of interest, and the location range, the action type and the action parameters are encapsulated to generate the scanning control instruction;

[0039] The scanning control command is sent to the digital pathology scanning system, causing the digital pathology scanning system to perform local refocusing, local rescanning, and scanning parameter adjustment within the location range, corresponding to the action type and the action parameters, to obtain the local rescanning image.

[0040] The local rescanned image is aligned with the initial image according to the position range, and the pixel data of the aligned local rescanned image is written into the pixel position in the initial image corresponding to the target region of interest to generate the updated full-view image;

[0041] Furthermore, the coordinate alignment includes: establishing a mapping relationship between the image coordinate system and the stage coordinate system based on the scanning system calibration parameters, and performing affine transformation correction or lens distortion correction on the local rescanned image before pixel writing and fusion.

[0042] The beneficial effects of this invention are:

[0043] 1. By using a quality defect joint multi-task neural network to perform defect localization, defect classification and quality heatmap evaluation on the initial scan image, and determine the target region of interest accordingly, the defect region can be accurately located and structurally represented, reducing the need for manual review and lowering the risk of missed detection.

[0044] 2. By using a gain prediction neural network to perform counterfactual evaluation of the quality gain of candidate control actions relative to the baseline action, and outputting a confidence lower bound for risk constraint, the rescan decision is transformed from an empirical threshold into a quantifiable benefit assessment, avoiding ineffective rescans and improving the certainty of rescan benefits.

[0045] 3. By using a hierarchical model to predict the controller, local refocusing, local rescanning, and scanning parameter adjustment actions are continuously optimized under scanning resource constraints such as maximum rescanning area and maximum rescanning time. The local rescanned images are then fused with the initial images to generate an updated full-field-of-view image. This increases the proportion of available full-field-of-view images while ensuring scanning efficiency and reduces the rescanning rate and total rescanning time. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a flowchart of the deep learning-based slicing defect rescanning control method proposed in this invention.

[0048] Figure 2 This is a flowchart of the gain prediction and confidence lower bound calculation in step S4 of the present invention;

[0049] Figure 3 This is a flowchart of the hierarchical constraint optimization control process in step S5 of the present invention. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0051] refer to Figure 1-3 A deep learning-based method for controlling slice defect rescanning includes:

[0052] S1. Obtain the initial scan image of the slice to be scanned and the corresponding scan parameter information. Divide the initial scan image into blocks according to a preset block division rule to obtain an image block set. S2. Input the image block set and scan parameter information into a joint multi-task neural network for quality defects, and output the defect detection results and quality assessment results corresponding to each image block. S3. Based on the defect detection results and quality assessment results, determine at least one target region of interest (ROI) and construct the corresponding state features. S4. Generate a corresponding set of candidate control actions for each target ROI. Jointly encode the state features and candidate control actions and input them into a gain prediction neural network. Output the quality gain prediction results of each candidate control action relative to the baseline action, including... The expected value of quality gain and the confidence lower bound used for risk constraints are used as the baseline action, which is not to perform local rescanning and scanning parameter adjustment in the target region of interest; S5, the target region of interest, the candidate control action set, and the quality gain prediction results are input into the hierarchical model prediction controller, and rolling optimization is performed within the preset prediction time domain to determine the optimized control action under the premise of satisfying the preset scanning resource constraints, and risk constraints are applied based on the confidence lower bound; S6, the optimized control action is converted into a scanning control command and sent to the digital pathology scanning system, and local rescanning and scanning parameter adjustment corresponding to the optimized control action are performed in the target region of interest to obtain a local rescanning image, which is then fused with the initial scanning image to generate an updated full-field image.

[0053] In this specific embodiment, S1 includes:

[0054] The digital pathology scanning system performs a pre-scan on the slide to be scanned and obtains the pre-scan image as the initial scan image, denoted as . ,in It is a three-channel RGB image with a pixel width of Pixel height is Simultaneously read from scanner control and imaging metadata One-to-one correspondence of scanning parameter information, including focal plane position parameters. With exposure parameters ,in Indicates the pre-scan focusing position of the stage along the optical axis, and the unit is . , Indicates camera exposure time in units of ;

[0055] Subsequently Perform tissue region detection to generate a tissue mask. ,in and Aligned to the same size in pixel coordinate system and Represents pixel coordinates The area is the organizational region. The background area is represented, and the organization area detection is implemented according to the following process: The RGB image is converted to grayscale using the BT.601 standard coefficients and then subjected to Gaussian filtering with a standard deviation of 1.0 pixel to suppress noise. Otsu thresholding is then applied to the filtered grayscale image to obtain a binary foreground mask. Finally, the binary foreground mask is processed sequentially using the structuring element... Morphological closing operations are used to fill small pores and structural elements within tissues. Morphological opening operations are used to remove small-particle noise. Finally, area filtering is performed on connected components, and connected components with an area less than 5000 pixels are deleted, thus obtaining the desired result. Tissue mask;

[0056] In generation The organizational percentage was then calculated. Based on this, the image patch size and image patch overlap ratio are adaptively determined, specifically... The number of pixels with a value of 1 and The ratio as And according to the formula:

[0057] The side length of the image patch was calculated. With overlap ratio ;

[0058] in Indicates the organizational percentage and the range of values ​​is Represents the side length of a square image patch, in pixels. This represents the overlap ratio of adjacent image patches in the horizontal and vertical directions, and its value ranges from [value missing]. Pixels Pixels and the calculated Pixels rounded to a multiple of 32 Rounding to two decimal places is used to facilitate the planning of the platform grid.

[0059] Subsequently only in tissue mask Within the covered area, the block positions are determined. These block positions are generated using a grid sliding window, with horizontal and vertical step sizes set to [values ​​to be filled in]. Multiply The integer pixel step size is then rounded to the nearest whole number, and the percentage of organized pixels within the window is calculated for each candidate sliding window's corresponding pixel region. If and only if The candidate sliding window is then confirmed as a valid block position, and its upper left corner coordinates are recorded. and side length After filtering all valid block positions, based on each valid block position from... The corresponding image patch is obtained by cropping from the middle. and associate all image patches with their location information. and shared scan parameter information They are collectively organized into a set of image patches.

[0060] In this specific embodiment, S2 includes:

[0061] For a set of image patches, each image patch is denoted as... ,in The image patch index has a value range of 1. The total number of image blocks is determined by the number of valid block locations. The space dimensions are and First, matching the side length of the image patch, Perform normalized preprocessing consistent with the training phase, RGB pixel values ​​are linearly scaled by channel to The interval is recorded, and the scanning parameter information corresponding to the image block is recorded. and ,in The focal plane position parameter is in units of , Exposure parameters and units are Then and Dimensionless parameters were obtained by linear normalization according to a fixed range. and ,in By In the interval The linear mapping above yields and By In the interval The linear mapping above yields and and will and Replicating and expanding in the spatial dimension to... Parametric channel plots of the same size and ,in For size And the value of each pixel is always equal to A single-channel matrix, For size And the value of each pixel is always equal to The single-channel matrix is ​​then used to finally convert the three-channel image blocks. With parameter channel plot The network input of the joint multi-task neural network for quality defects is formed by splicing together channels. Its construction satisfies the formula:

[0062] ;

[0063] in For size The five-channel input tensor This represents the splicing operator along the channel dimension. Indicate size RGB image patch tensor, Indicate size The focal plane normalized parameter channel tensor, Indicate size Exposure normalization parameter channel tensor;

[0064] The quality defect joint multi-task neural network adopts a deterministic structure of a shared encoder and a dual-task decoder. The encoder contains four downsampling stages, and each stage consists of two layers. Convolution, batch normalization, and ReLU activation are combined and passed through a step size of 2 at the end of the stage. Max pooling is used to achieve downsampling, with the number of channels in the four stages as follows: The 512 dual-task decoder corresponds to the defect detection branch and the quality assessment branch, respectively. Both branches contain four upsampling stages. Each upsampling stage upsamples by a factor of 2 through bilinear interpolation and performs a skip connection with features of the same scale as the encoder before passing through two layers. Feature fusion is performed using convolution, batch normalization, and ReLU activation.

[0065] The output of the defect detection branch adopts Convolution produces a size of The pixel-level defect categories are logits, where 6 is the sum of defect category 1 (background defect) and defect category 5 (defect category 5, in order of appearance): out of focus, partial blur, wrinkle, bubble, and knife mark. Then, Softmax is applied to these logits along the category dimension to obtain a pixel-level defect probability tensor. The defect category at each pixel is then selected as the category with the highest probability to form a defect localization map corresponding to each pixel. Simultaneously, based on the defect localization map, the pixel proportion of each defect category within the image block is calculated, and the defect category with the largest pixel proportion is selected as the defect category for that image block. Finally, the defect level is determined by segmenting the defect pixel proportion corresponding to this defect category according to a threshold value, with the defect level set to a value. A defect level of 0 indicates that the percentage of defective pixels is less than 0.01, while a defect level of 1 indicates that the percentage of defective pixels is within the range of 0.01. A defect level of 2 indicates that the percentage of defective pixels is located at [a certain level]. A defect level of 3 indicates that the percentage of defective pixels is not less than 0.15.

[0066] The output of the quality assessment branch uses Convolution produces a size of The quality logits are then mapped using a Sigmoid algorithm to create a quality heatmap that corresponds one-to-one with each pixel. The pixel values ​​in the quality heatmap range from [value range missing]. Furthermore, the smaller the value, the worse the image quality of that pixel;

[0067] The network parameters of the joint multi-task neural network for quality defects were determined through offline training before deployment. The training data consisted of image patch samples with pixel-level defect annotations and pixel-level quality annotations. The training loss was calculated by weighting the pixel-level cross-entropy loss of the defect branch and the pixel-level mean square error loss of the quality branch. Weighted composition, the optimizer uses Adam and the learning rate is The batch size is 16, and the number of training rounds is 100, so that the network input for each image patch is... The output then includes the defect detection results and quality assessment results corresponding to the image patch, along with the image patch index. Associated storage.

[0068] In this specific embodiment, S3 includes:

[0069] For each image block The output defect detection results and quality assessment results are first mapped from the defect location map and quality heatmap corresponding to each image block to the initial scan image. The coordinate system is used to form a full-scale defect confidence map and a quality heatmap;

[0070] Image patch index is Image blocks in The top left pixel coordinates are The side length of the image block is The pixel coordinates inside the image block are And satisfy The mapping rule is to map the pixels inside the image block. Corresponding to the total number of pixels in the image ;

[0071] During full-image fusion, when the same full-image pixel... When a pixel is covered by multiple image patches, the defect confidence score is the maximum of the defect confidence scores of all image patches covering that pixel at that pixel, forming a full-image defect confidence map. The quality heatmap is formed by taking the minimum quality score of all image patches covering a pixel at that pixel. ,in Furthermore, the larger the value, the higher the confidence level that the pixel is a defective pixel. Furthermore, the smaller the value, the worse the image quality of that pixel;

[0072] Subsequently, candidate regions were selected from the full-map defect confidence map and the full-map quality heatmap based on threshold rules, with the first threshold set as follows: Furthermore, it is used for defect confidence screening, with the second threshold set as follows: It is used for quality scoring and screening, and a binary mask for the candidate region is generated using the following formula. ,Right now:

[0073] ;

[0074] in and They represent Horizontal and vertical pixel coordinates in a coordinate system and Represents pixels It belongs to the candidate region. This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Represents the logical OR operator. This indicates the confidence map of defects in the entire image at the pixel level. The value at that location, This indicates the overall image quality heatmap at the pixel level. The value at that location, Indicates the first threshold. Indicates the second threshold;

[0075] After obtaining the binary mask of the candidate region Afterwards, Perform 8-neighborhood connected component extraction to obtain a set of candidate regions of interest and remove regions with areas smaller than 8. A connected region of a pixel, where the area is determined by the condition within the connected region. Pixel count calculation;

[0076] Subsequently, a merging process is performed on the candidate regions of interest to obtain a set of target regions of interest. The merging rule is that when the bounding rectangles of two candidate regions of interest overlap, they are merged into the same target region of interest, or when the minimum boundary distance between the bounding rectangles of two candidate regions of interest is less than a distance threshold. When pixels are merged into the same target region of interest, the merging implementation uses a disjoint-set data structure to establish equivalence relations between candidate regions of interest that meet the merging rules and outputs the merged connected component as the target region of interest.

[0077] For each target region of interest Construct its state feature vector ,in Index the target region of interest. Defined by its pixel set and contained in Minimum bounding rectangle range in coordinate system and area according to The number of pixels within the cell is calculated and expressed in pixels.

[0078] In determining the defect category and defect level, firstly... The pixel count of each defect category in the pixel-level defect category label is counted, and the defect category with the highest pixel count is selected as the defect category of the target region of interest. When the statistical results show When all pixels within the range are of the background category, the defect category will be... Set to local fuzziness to ensure that subsequent control action type selection has a definite input;

[0079] Recalculate This belongs to the defect category. The pixel ratio is used to determine the defect level. The defect level is set to a value of That is, when the percentage of defective pixels is less than 0.01. The percentage of defective pixels is located in hour The percentage of defective pixels is located in hour When the percentage of defective pixels is not less than 0.15 ;

[0080] In the calculation of mass heatmap statistics, based on the full-map mass heatmap... exist Calculate the average quality value of pixels within the range. Quality standard deviation Minimum mass and low-quality pixel ratio The proportion of low-quality pixels Defined as Internal satisfaction The number of pixels and The ratio;

[0081] Finally, the location range of the target region of interest will be determined. Area Defect Category Defect level Statistical measures of quality heatmap , , And the scanning parameter information corresponding to the slice. and The state feature vector is constructed by concatenating the fields in a fixed order. and all target regions of interest The set of state features is composed of these features.

[0082] In this specific embodiment, S4 includes:

[0083] For each target region of interest and its state feature vector ,in Index the target region of interest and To determine the total number of regions of interest, first, based on... Defect categories included With defect level Determine the set of allowed action types in the region of interest of the target. ,in The value set is {out of focus, partial blur, wrinkles, bubbles, knife marks}. Action types use deterministic coding ,in Indicates the reference action and is incorrect. Perform local rescan and scan parameter adjustment actions. This indicates a partial rescan and allows for exposure and focal plane adjustments. This indicates that after local refocusing, a local rescan is performed, and exposure adjustment is allowed. This indicates local multi-focal layer rescanning and allows for exposure and focal plane adjustment;

[0084] when Out of focus, partially blurred Time setting ,when Wrinkles, bubbles, knife marks Time setting ;

[0085] Subsequently, a set of candidate control actions is generated within the preset parameter range corresponding to each action type. These candidate control actions are then used... It means that, among them Index the candidate control actions and For the target region of interest The corresponding number of candidate control actions, and each candidate control action Based on action type With action parameter vector To be determined jointly;

[0086] Action parameter vector Defined according to a fixed field order:

[0087] ;

[0088] in The local rescan range parameter is expressed in The minimum bounding rectangle's outward expansion boundary width, expressed in pixels. This is the refocusing step size parameter, representing the search step size of local autofocus along the optical axis, and expressed as follows: As a unit, This is a focal plane adjustment parameter and represents the focal plane position parameter in the scanning parameter information. The bias applied on the basis and with As a unit, This refers to the exposure adjustment parameter and represents the exposure parameter in the scan parameter information. Increment applied on the basis and with As a unit, The number of focal layers is a parameter representing the number of focal layers acquired through local multi-focal-layer rescanning, and it is an odd number.

[0089] The parameter values ​​are determined using a discrete set, and are set accordingly. , And according to the action type, the parameters are enabled or zeroed to ensure that the action definition is uniquely determined, where when season ,when season and ,when season and ,when season and ;

[0090] To ensure that candidate actions are executable within the physical constraints of the scanning system, parameter pruning is performed on each candidate control action during generation to optimize the actual exposure parameters. Cut to Within the range and actual focal plane position parameters Cut to Within the range;

[0091] After forming a set of candidate control actions, the state features and candidate control actions are jointly encoded to form the gain prediction input. Specifically, the state features and candidate control actions are then jointly encoded. The position range in the initial scan image width With height Normalization, dividing the area according to Normalization, classifying defects Perform 5D one-hot encoding according to a fixed order [out of focus, partial blur, wrinkles, bubbles, knife marks] to classify the defect levels. Normalized to 3, the statistical measures of the quality heatmap are directly adopted. and scan parameter information and After obtaining dimensionless parameters by mapping the same range as S2, they are concatenated into a normalized state vector.

[0092] Action type Perform 4D one-hot encoding and convert the action parameter vector After linearly normalizing the values ​​of each candidate control action to their respective minimum and maximum values, concatenate them with the normalized state vector to obtain the corresponding candidate control action. Gain prediction input vector ;

[0093] Will The input gain prediction neural network outputs a quality gain prediction result of the candidate control action relative to the reference action. The gain prediction neural network adopts a deterministic multilayer perceptron structure with the following parameters: the input layer dimension equals... The dimension is defined by three fully connected hidden layers with 128, 64, and 32 neurons respectively, all using ReLU activation functions. The output layer consists of two scalar outputs, each representing the expected value of the mass gain. Standard deviation of mass gain ,in Activation guaranteed via Softplus ;

[0094] The gain prediction neural network determines its weights through offline training before deployment. The training samples are generated from historical scan logs, and each sample contains... With supervision label ,in The quality gain relative to the baseline action is represented by the average pixel quality of the local rescanned image within the region after performing candidate control actions on the same target region of interest, minus the average pixel quality of the initial scan image within the same region. The training loss uses... and The negative log-likelihood loss characterizes the Gaussian distribution, and the Adam optimizer is used with a learning rate set to 1. Batch size is set to 256, and number of training rounds is set to 50;

[0095] In obtaining and The confidence lower bound of the quality gain used for risk constraints is then calculated. The calculation formula is as follows:

[0096] ;

[0097] in Indicates candidate control actions The lower bound of the quality gain relative to the reference action. This represents the expected value of the mass gain. The standard deviation of the quality gain is represented by... This indicates a preset risk coefficient with a value of 1.645 to correspond to one-sided risk. Confidence level;

[0098] Region of interest for each target Candidate control action set and its corresponding Common output and region index The data is stored in association to form a set of candidate control actions and a quality gain prediction result.

[0099] In this specific embodiment, S5 includes:

[0100] Set the target region of interest and candidate control action set group The corresponding mass gain prediction results , Input the hierarchical model predictive controller and set the prediction time domain as follows: ,in Indexing the target region of interest The total number of regions of interest. Indexing candidate control actions For the region The corresponding number of candidate control actions, Candidate control actions The expected mass gain relative to the baseline motion. The standard deviation of the quality gain. Provide a confidence lower bound for the mass gain;

[0101] The hierarchical model prediction controller operates in a rolling optimization manner, specifically by re-executing S1 through S5 to update the full-view image after each execution of S6 to generate an updated full-view image. and And output the next optimized control action;

[0102] To ensure that the constrained optimization is computable and the displacement cost can be precisely minimized, the number of regions of interest participating in the optimization is limited to a certain limit. When the number of regions of interest is greater than 12, the region area is used as the criterion. With defect level The products are sorted from largest to smallest, and the top 12 regions of interest are selected for S5. The area of ​​the target region of interest is expressed in pixels. Defect level;

[0103] In cost modeling, for each candidate control action Calculate the cost of rescanning area, rescanning time, and displacement. The cost of rescanning area is determined by the local rescanning range parameter. Determine and pass the target area of ​​interest The minimum bounding rectangle expands outwards on all four sides. The pixels are used to obtain a local rescan rectangle, which is then cropped to the initial scan image. pixel boundaries The rescan area is then obtained by calculating its pixel area. ,in The width of the boundary is expressed in pixels. for Pixel width, for Pixel height, rescan area cost is taken as Divide by the maximum rescan area constraint The obtained normalized value and ;

[0104] The cost of rescanning time is determined by the area of ​​the local rescanning rectangle and the number of focal layers. The scanning imaging time is determined by the scanning throughput and the type of action. Estimate and Pixels per second, the scanning imaging time for a single candidate control action is taken as... and product divided by When the action type is partial refocusing followed by partial rescanning, the partial autofocus time is included in the rescanning time, and the autofocus is within... Within the focal length range, in steps Sharpness evaluation frames were acquired point by point, and the autofocus time was obtained by timing each frame for 0.02 seconds. The refocusing step size and the unit is When the action type is the baseline action, the rescan time cost is 0. The rescan time cost is taken as the total rescan time of the candidate control action divided by the maximum rescan time constraint. The obtained normalized value and Second;

[0105] The displacement cost is used to characterize the stage movement time between different regions of interest (ROIs) and is further minimized by prediction control from the upper-level model. Its calculation is based on the mapping relationship between the image coordinate system and the stage coordinate system established by the scanning system calibration parameters, dividing each ROI into... The pixel center point is mapped to the stage plane coordinates. and at the maximum moving speed of the platform millimeters per second and time to settling Estimate the distance between any two regions in seconds. Move to The movement time is calculated, and then the selected target region of interest that needs to perform non-baseline actions is taken as a node set, with the current position of the platform as the starting node. Held-Karp dynamic programming is used to solve for the visiting sequence that minimizes all nodes and the total movement time, obtaining the minimum displacement cost and dividing it by... Obtain the normalized displacement cost;

[0106] The upper-level model predictive control constructs a discrete optimization problem with the objective of maximizing total mass gain and minimizing rescan cost, and solves it under scan resource constraints, where the discrete decision variables are: and Represented as the target region of interest Select candidate control actions and for each Forced satisfaction To ensure that each target's region of interest selects exactly one action and the baseline action is used as one of the candidate control actions in the selection, while also... Not less than the threshold As a feasibility condition, risk constraints are imposed, and the total rescanned area shall not exceed [a certain limit]. And the total rescan time plus the minimum displacement cost does not exceed As a constraint on scanning resources;

[0107] The upper-level objective function is defined as:

[0108] ;

[0109] in To optimize the target value for the upper layer, For the expected mass gain, For the rescanned area according to Normalized rescan area cost For candidate control actions, total scan duration is pressed. The cost of rescanning time obtained by normalization To be based on the minimum displacement cost The displacement cost obtained by normalization These are the rescan area cost weight, rescan duration cost weight, and displacement cost weight, respectively.

[0110] The upper-level discrete optimization employs a branch-and-bound method for solution, performing feasibility pruning based on resource constraints and boundary pruning based on the upper bound of the maximum achievable quality gain of the remaining region during the search process. This results in the output of the selected candidate control action index for each target's region of interest. And the minimum displacement access order corresponding to the selected region set;

[0111] The lower-level model predictive control continuously optimizes motion parameters under the condition that the motion type output by the upper level is fixed. Specifically, for each non-baseline motion, it uses the selected motion parameter vector. As initial values ​​and in parameter boundaries The internal solution employs sequential quadratic programming to maximize the expected value of the quality gain. And simultaneously satisfy the confidence lower bound not less than Risk constraints, and after each iteration, the exposure and focal plane parameters are cropped back to the physical range of the scanning system:

[0112] , ;

[0113] in This is the continuously optimized action parameter vector. These are the initial exposure parameters. These are the initial focal plane position parameters;

[0114] When continuous optimization at the lower level causes any resource constraint to be violated, the action in the region of interest of the target is reverted to the discrete candidate control action selected by the upper level. It maintains its original parameters, thereby outputting a set of optimized control actions that simultaneously satisfy the maximum rescan area constraint, the maximum rescan duration constraint, and the risk constraint.

[0115] In this specific embodiment, S6 includes:

[0116] For each target region of interest The corresponding optimized control action, read the action type of the optimized control action. With action parameter vector:

[0117] ;

[0118] in Index the target region of interest and The total number of regions of interest. These represent the baseline action, local rescan and parameter adjustment, local refocusing and rescan, and local multi-focal layer rescan and parameter adjustment, respectively. This refers to the local rescan range parameters, with units in pixels. This is the refocusing step size parameter, and the unit is... , The focal plane adjustment parameter is in units of , The exposure adjustment parameter is in units of The focal layer number parameter must be an odd number;

[0119] according to In the initial scan image Minimum bounding rectangle range in coordinate system Generate local rescan pixel range ,in Depend on Cut to boundary Later obtained and for Pixel width, for Pixel height;

[0120] Will The scan control commands are encapsulated as scan control instructions and sent to the digital pathology scanning system. The scan control instructions use a fixed set of fields. ,in For regional identification, for Top left pixel coordinates and They are respectively The pixel width and pixel height;

[0121] To enable the scanning system to perform local scanning within the stage coordinate system, a mapping relationship between the image coordinate system and the stage coordinate system is established based on the scanning system calibration parameters, and then... Pixel coordinates of the four corner points Perform mapping to obtain the stage coordinates The mapping uses an affine model:

[0122] ;

[0123] in and These are the horizontal and vertical pixel coordinates in the image coordinate system, respectively. and These represent the horizontal and vertical positions in the stage coordinate system, with units of . for calibration matrix and express The directional pixel size is , express The directional pixel size is and This represents the rotational coupling coefficient of the image coordinate system relative to the stage coordinate system, and its unit is 1. , and This represents the stage offset corresponding to the image origin, in units of . ;

[0124] After receiving the scan control command, if the digital pathology scanning system... Then skip this And it does not produce local rescan images, if Then according to The stage bounding box obtained from corner mapping is used to plan stage movement and scanning grid, and the actual exposure parameters are set to... And will Crop to Set the actual focal plane position parameters to And will Crop to ,in and These are the exposure parameters and focal plane position parameters read in step S1, respectively;

[0125] when At that time Perform local autofocus within the corresponding stage area and use Centered on within the range by step size Focus evaluation frames are acquired. Sharpness evaluation uses Tenengrad gradient energy, and the sum of squared gradients is calculated on each frame. The focal plane with the highest evaluation value is selected as the locally optimal focal plane. Subsequently Local rescanning of a single focal layer is performed to obtain a local rescanned image. ;

[0126] when At that time The local rescan of the single focal layer was completed to obtain ;when Time Collection by center Layer local rescan image stack The interlayer spacing is fixed as And collect data in ascending order of focal plane, and... according to The pixel grid is divided into multiple sub-blocks. The Tenengrad gradient energy of each focal layer is calculated for each sub-block, and the focal layer sub-block with the largest evaluation value is selected as the output at that location. These sub-blocks are then stitched together to generate a single local multiscan image. ;

[0127] In Compared with the initial scan image Coordinate alignment is performed before fusion. The alignment process is as follows: first, based on the calibration matrix... Will The pixel range is converted into a local rescan image. The theoretical landing point range in the middle, and then... Lens distortion correction was performed using the Brown-Conrady model with the distortion parameters fixed as radial distortion coefficients. , With tangential distortion coefficient The principal point coordinates are fixed as the center pixel coordinates of the locally scanned image. After correction, the enhanced correlation coefficient alignment algorithm is used to perform the alignment under the affine transformation model. and exist Iterative registration and affine transformation parameters are updated on the overlapping band formed 32 pixels inward from the boundary. The number of iterations is fixed at 50, and the termination threshold is fixed at [value missing]. This yields the aligned local rescan image. ;

[0128] Will according to and Perform pixel write fusion to generate an updated full-view image During fusion Set a linear transition band with a width of 16 pixels at the boundary and within the transition band... and Weighted linear blending is used to eliminate stitching edges, while the remaining areas are treated with... Pixel direct overlay For corresponding pixel positions, when multiple target regions of interest need to be fused, alignment and fusion are performed sequentially according to the minimum displacement access order, and after each fusion, [the following is applied]. This serves as the base map for the next fusion until all regions are processed, resulting in the final updated full-view image. .

[0129] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0130] This invention forms an end-to-end closed loop of "joint perception of quality defects, evaluation of rescan benefits, and constrained control decision-making": First, a joint multi-task neural network for quality defects is used to simultaneously output defect localization and quality heatmaps on pre-scanned or online scanned images, thereby transforming problems such as defocusing, local blurring, wrinkles, and bubbles into calculable regions of interest (ROIs) and state features. Then, using "no rescanning and parameter adjustment" as the baseline action, quality gain prediction is performed on candidate local rescanning, refocusing, exposure, and focal plane adjustment actions. Under a hierarchical model predictive control framework, rolling optimization is performed by comprehensively considering scanning resource constraints such as rescanning area, rescanning time, and displacement costs, outputting control commands that can be directly sent to the scanner to achieve local intervention in defective areas and image fusion updates. This process transforms rescanning from primarily "whole-area rescanning and manual verification" to "precise rescanning by ROI and adaptive parameter tuning," thereby increasing the proportion of usable full-field-of-view images and reducing rescanning rate and time consumption.

[0131] Compared to existing rescanning schemes based on threshold rules or single quality judgment, this invention improves the algorithm structure to address technical issues in the following ways: First, it introduces a counterfactual evaluation mechanism based on gain prediction to directly estimate the quality improvement of each candidate action relative to the baseline action, providing a quantitative basis for whether a rescan is worthwhile and reducing ineffective rescanning. Second, it adopts a hierarchical model prediction control that combines upper-level discrete action type selection with lower-level continuous action parameter optimization, enabling different defect types to be matched with more suitable intervention strategies and achieving overall optimization within the resource budget. Third, it calculates the confidence lower bound based on the uncertainty of gain prediction and applies risk constraints, which can still suppress high-risk decisions when there are biases or noise in the model prediction, further improving the stability and reliability of rescanning benefits.

Claims

1. A deep learning-based method for controlling the rescanning of slab defects, characterized in that, include: S1. Obtain the initial scan image of the slice to be scanned and the corresponding scan parameter information, and divide the initial scan image into blocks according to the preset block division rules to obtain a set of image blocks. S2. Input the image patch set and scanning parameter information into the quality defect joint multi-task neural network, and output the defect detection results and quality assessment results corresponding to each image patch; S3. Based on the defect detection results and quality assessment results, determine at least one target region of interest and construct the corresponding state features; S4. Generate a set of candidate control actions for each target region of interest. After jointly encoding the state features and candidate control actions, input them into the gain prediction neural network. Output the quality gain prediction results of each candidate control action relative to the baseline action, including the expected value of the quality gain and the confidence lower bound for risk constraints. The baseline action is the action that does not perform local rescanning and scan parameter adjustment in the target region of interest. S5. Input the target region of interest, the set of candidate control actions, and the quality gain prediction results into the hierarchical model prediction controller. Perform rolling optimization within the preset prediction time domain to determine the optimized control action under the premise of satisfying the preset scan resource constraints, and apply risk constraints based on the confidence lower bound. S6. Convert the optimized control action into a scan control command and send it to the digital pathology scanning system. Perform local rescanning and scan parameter adjustment corresponding to the optimized control action in the target region of interest to obtain local rescanning images, and fuse them with the initial scan images to generate an updated full-field image.

2. The deep learning-based method for controlling slice defect rescanning according to claim 1, characterized in that, S1 includes: Obtain a pre-scanned image or an online scanned image of the slice to be scanned as the initial image; Obtain scanning parameter information corresponding to the initial image, the scanning parameter information including focal plane position parameters and exposure parameters; The block positions are determined on the initial image according to the preset block division rules, and the initial image is cropped according to the block positions to obtain the image block set containing multiple image blocks.

3. The deep learning-based slice defect rescan control method according to claim 1, characterized in that, S2 include: Each image block in the image block set is jointly encoded with the scanning parameter information to form a network input, and the network input is input into the quality defect joint multi-task neural network. The quality defect joint multi-task neural network outputs the defect detection result and the quality assessment result corresponding to each image block in the image block set. The defect detection result includes a defect localization map corresponding to each pixel of the image block, a defect category and a defect level determined based on the defect localization map, and the quality assessment result includes a quality heatmap corresponding to each pixel of the image block.

4. The deep learning-based method for controlling slice defect rescanning according to claim 1, characterized in that, S3 includes: Map the defect location map and the quality heat map to the coordinate system of the initial image; In the defect location map, pixel regions with defect confidence scores not less than a first threshold are selected, and in the quality heatmap, pixel regions with quality scores not greater than a second threshold are selected. The above two types of pixel regions are then combined and fused to obtain candidate regions. Connectivity extraction is performed on the candidate regions to obtain at least one candidate region of interest, and candidate regions of interest that overlap or have a spacing less than a preset distance threshold are merged to obtain the target region of interest set; For each target region of interest in the set of target regions of interest, a state feature is calculated based on the defect detection result, the quality assessment result, and the scanning parameter information within the target region of interest. The state feature includes the location range of the target region of interest, the area of ​​the target region of interest, the defect category, the defect level, and the quality heatmap statistics within the target region of interest, thus obtaining the set of state features.

5. The deep learning-based slice defect rescan control method according to claim 1, characterized in that, S4 include: For each target region of interest in the set of target regions of interest, the allowed action type is determined based on the state characteristics corresponding to the target region of interest, and at least one action parameter is generated within the preset parameter range corresponding to each action type to obtain a set of candidate control actions corresponding to the target region of interest, and further, a set of candidate control actions corresponding to the set of target regions of interest is obtained. The action parameters include local rescan range parameters, refocus step size parameters, exposure adjustment amount parameters, focal plane adjustment amount parameters, and focal layer number parameters. For each candidate control action in the candidate control action set corresponding to each target region of interest in the candidate control action set group, the action type and action parameters of the candidate control action are vectorized and encoded to obtain action features. The state features in the state feature set corresponding to the target region of interest are combined with the action features to form a gain prediction input. The gain prediction input is fed into the gain prediction neural network, and the gain prediction result corresponding to the candidate control action is output. The gain prediction result includes the expected value of the quality gain and the standard deviation of the quality gain. The confidence lower bound of the quality gain is calculated based on the expected value of the quality gain and the standard deviation of the quality gain. The confidence lower bound is equal to the expected value of the quality gain minus the product of the preset risk coefficient and the standard deviation of the quality gain.

6. The deep learning-based method for controlling slice defect rescanning according to claim 1, characterized in that, S5 include: The target region of interest set, the candidate control action set group, and the gain prediction result are input into the hierarchical model prediction controller. The upper-level model predicts the expected value of the quality gain corresponding to each candidate control action in the gain prediction result. A discrete optimization problem is constructed with the upper-level objective function as maximizing the total quality gain and minimizing the rescan cost. Under the premise of satisfying the preset scan constraints, the action type corresponding to the target region of interest is determined from the candidate control action set group and the candidate control action set corresponding to each target region of interest. The rescanning cost includes the rescanning area cost and the rescanning time cost, and the preset scanning constraints include the maximum rescanning area constraint and the maximum rescanning time constraint. The lower-level model predicts and controls the action type corresponding to each target region of interest. The expected value of the quality gain is used as the optimization target to continuously optimize the action parameters corresponding to the action type. The risk constraint is applied based on the confidence lower bound of the quality gain in the gain prediction result so that the optimized action parameters meet the risk constraint. The risk constraint includes that the confidence lower bound is not less than a preset threshold. Output optimized control actions that satisfy the preset scanning constraints and the risk constraints.

7. The deep learning-based slice defect rescan control method according to claim 1, characterized in that, S6 include: For each target region of interest in the set of target regions of interest, the location range of the target region of interest, the action type and action parameters corresponding to the optimized control action are determined according to the optimized control action corresponding to the target region of interest, and the location range, the action type and the action parameters are encapsulated to generate the scanning control instruction; The scanning control command is sent to the digital pathology scanning system, causing the digital pathology scanning system to perform local refocusing, local rescanning, and scanning parameter adjustment within the location range, corresponding to the action type and the action parameters, to obtain the local rescanning image. The local rescanned image is aligned with the initial image according to the position range, and the pixel data of the aligned local rescanned image is written into the pixel position in the initial image corresponding to the target region of interest to generate the updated full-view image.

8. The deep learning-based method for controlling slice defect rescanning according to claim 2, characterized in that, The preset segmentation rules include: first, performing tissue region detection on the initial scan image to generate a tissue mask; determining segmentation positions only within the area covered by the tissue mask; and adaptively adjusting the image block size or image block overlap ratio according to the tissue proportion of the tissue mask.

9. The deep learning-based method for controlling slice defect rescanning according to claim 6, characterized in that, The rescanning cost also includes a displacement cost, which is used to characterize the distance or time the stage moves between different target regions of interest; the upper-level model predictive control further minimizes the displacement cost when determining the action type to reduce the total rescanning time.

10. The deep learning-based method for controlling slice defect rescanning according to claim 7, characterized in that, The coordinate alignment includes: establishing a mapping relationship between the image coordinate system and the stage coordinate system based on the scanning system calibration parameters, and performing affine transformation correction or lens distortion correction on the local rescanned image before pixel writing and fusion.

Images