Tissue slice image automatic registration method and device and electronic equipment

Abstract

The embodiment of the invention discloses an automatic image registration method and device for tissue slices and electronic equipment. The method comprises the following steps: acquiring a target dyed image obtained by preprocessing a tissue slice, and generating a honeycomb image corresponding to the tissue slice; constructing a multi-layer Gaussian image pyramid for the target dyed image and the cellular image; and carrying out angle and position search layer by layer from a low-resolution layer to a high-resolution layer in the multi-layer Gaussian image pyramid, determining a target registration parameter corresponding to the highest-resolution layer, and carrying out registration on the target dyed image and the honeycomb image according to the target registration parameter. In the embodiment of the invention, the optimal registration parameter can be comprehensively evaluated layer by layer through a multi-dimensional scoring mode, the limitation of a traditional correlation method is overcome, the algorithm adaptability is better, and the precision of automatic registration is higher.

Description

Technical Field

[0001] The embodiments in this specification pertain to the field of spatial transcriptome data analysis, and specifically relate to an automatic image registration method, apparatus, and electronic device for tissue sections. Background Technology

[0002] Spatial transcriptomics obtains gene expression data with spatial location information by simultaneously performing microscopic imaging (e.g., hematoxylin-eosin staining, H&E) and in situ RNA capture on tissue sections. In this process, accurately registering and aligning the morphological images of the tissue sections (H&E maps) with the probe array images (honeycomb maps) on the sequencing chip is fundamental for subsequent data analysis. However, current spatial transcriptomics data processing suffers from inefficiencies, poor adaptability of automated algorithms, lack of interactive correction mechanisms, and missing image flip detection, resulting in registration results that fail to meet expectations and insufficient registration accuracy. Summary of the Invention

[0003] Embodiments of this disclosure provide an automatic image registration method, apparatus, and electronic device for tissue sections, aimed at solving one or more of the above-mentioned problems and other potential problems.

[0004] According to a first aspect of this disclosure, an automatic image registration method for tissue slices is provided. The method includes acquiring a target stained image of the tissue slice after preprocessing and generating a honeycomb image corresponding to the tissue slice; constructing a multi-layer Gaussian image pyramid for both the target stained image and the honeycomb image, wherein the search range of the non-initial layers of the multi-layer Gaussian image pyramid is determined based on the optimal registration parameters of the previous layer. The optimal registration parameters are the registration parameters that yield the highest weighted value among the normalized mutual information score, edge matching score, and spatial correspondence score of the target stained image and the honeycomb image at the current layer. The normalized mutual information score is used to characterize the statistical correlation of the image in grayscale distribution, the edge matching score is used to characterize the geometric alignment of the image on the edge of the tissue structure and the boundary of the honeycomb grid, and the spatial correspondence score is used to characterize the degree of spatial overlap between the image's expression signal region and the tissue region; and performing angle and position searches layer by layer from low-resolution to high-resolution layers in the multi-layer Gaussian image pyramid to determine the target registration parameters corresponding to the highest resolution layer, so as to register the target stained image and the honeycomb image according to the target registration parameters.

[0005] According to a second aspect of this disclosure, an automatic image registration apparatus for tissue slices is provided. The apparatus includes an image acquisition module configured to acquire a target stained image obtained after preprocessing of the tissue slice and generate a honeycomb image corresponding to the tissue slice; a Gaussian image pyramid construction module configured to construct multi-layer Gaussian image pyramids for the target stained image and the honeycomb image respectively. The search range of the non-initial layers of the multi-layer Gaussian image pyramid is determined based on the optimal registration parameters of the previous layer. The optimal registration parameters are the registration parameters that have the highest weighted value among the normalized mutual information score, edge matching score, and spatial correspondence score of the target stained image and the honeycomb image in the current layer. The normalized mutual information score is used to characterize the statistical correlation of the image in grayscale distribution, the edge matching score is used to characterize the geometric alignment of the image on the edge of the tissue structure and the boundary of the honeycomb grid, and the spatial correspondence score is used to characterize the degree of spatial overlap between the expression signal region of the image and the tissue region; and an image registration module configured to perform angle and position searches layer by layer in the multi-layer Gaussian image pyramid from low-resolution layer to high-resolution layer to determine the target registration parameters corresponding to the highest resolution layer, so as to register the target stained image and the honeycomb image according to the target registration parameters.

[0006] According to a third aspect of this disclosure, an electronic device is provided, including one or more processors and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform a method provided according to a first scheme.

[0007] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided according to the first aspect.

[0008] The scheme provided in the embodiments of this specification can construct a multi-layer Gaussian image pyramid, determine the optimal registration parameters for each layer based on a weighted value calculated from the normalized mutual information score, edge matching score, and spatial correspondence score, and then determine the search range for the next layer based on the optimal registration parameters until the target registration parameters are obtained and registration is completed. In this way, the optimal registration parameters can be comprehensively evaluated layer by layer through multi-dimensional scoring, overcoming the limitations of traditional correlation methods, resulting in better algorithm adaptability and higher accuracy in automatic registration. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A flowchart illustrating an automatic image registration method for tissue slices according to some embodiments of the present disclosure is shown.

[0011] Figure 2 A schematic diagram of the structure of an automatic image registration apparatus for tissue slices according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] The terms “comprising” and “having”, and any variations thereof, in this specification, claims, and the foregoing drawings are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. Depending on the context, the word “if” as it applies herein may be interpreted as “when”, “in response to determination”, or “in response to detection”.

[0015] As mentioned earlier, taking spatial transcriptome data processing on specific chip platforms like Agilent as an example, the following problems exist: 1. Low efficiency: Traditional tools (such as some open-source viewers like Loupe Browser and ImageJ) often rely on researchers manually dragging and rotating images to align fiducials on the chip. For high-throughput data, a single slice can take tens of minutes. 2. Poor adaptability of automated algorithms: When existing general medical image registration algorithms are directly applied to spatial transcriptome scenarios, feature extraction often fails due to inconsistent H&E staining depths, tissue damage, or folding, making fully automated and accurate alignment difficult. 3. Lack of interactive correction mechanisms: The process is essentially "black box," and once automatic registration fails, users cannot easily make fine-tuning corrections, rendering valuable sample data unusable (due to the lack of transcriptome expression data from the tissue region). 4. Lack of image flip detection: Due to the uncertainty of tissue slicing operations, H&E images may be horizontally or vertically flipped, and existing tools often cannot automatically detect and correct this. 5. Traditional alignment / registration of idle chip data and H&E images requires specific chip identifiers in a specific format to assist in identifying orientation and position. Often, image registration algorithms are limited to specific H&E images and idle chip formats. For example, the image registration algorithm involved in the 10XGenomics VisiumHD product does not take full advantage of the algorithm and is limited to a single product form (data format).

[0016] Figure 1 A flowchart illustrating an automatic image registration method 100 for tissue slices according to some embodiments of this disclosure is shown. Method 100 can be executed by a terminal, which may include, but is not limited to, a mobile phone, tablet computer, desktop computer, server, etc. Figure 1 As shown, in method 100, step 102 can obtain the target staining image obtained after the tissue section has been preprocessed, and generate the honeycomb image corresponding to the tissue section.

[0017] In this embodiment, after obtaining the original stained images of the tissue sections (i.e., hematoxylin-eosin stained histopathological images), preprocessing is required to improve image quality, highlight tissue texture, and accurately identify tissue regions to obtain the target stained image. The target stained image can be, for example, an image in binary mask form. As an example, preprocessing can include image enhancement and tissue region segmentation. Image enhancement can include, for example, global or local histogram equalization, contrast linear stretching, gamma correction, etc. Tissue region segmentation can include, for example, threshold-based segmentation (such as Otsu's method, adaptive thresholding, etc.), clustering-based segmentation (such as K-means, mean shift, etc.), and machine learning-based segmentation (such as random forest, support vector machine, etc.). Furthermore, based on the spatial transcriptome chip design file and gene expression data, a virtual honeycomb image of the same size as the target stained image needs to be generated to simulate the physical arrangement of probes on the chip and their corresponding expression abundance, serving as a floating image during the registration process. In the process of drawing the probe geometry, the size of the geometry can be determined by the probe spacing and the preset visual density of the honeycomb pattern. For example, the side length of the hexagon can be calculated by the probe spacing and the geometric relationship of the grid. Vector graphics rendering or rasterized filling can be used when drawing.

[0018] In method 100, step 104 can construct a multi-layer Gaussian image pyramid for the target stained image and the honeycomb image respectively. The search range of the non-initial layer of the multi-layer Gaussian image pyramid is determined based on the optimal registration parameter of the previous layer. The optimal registration parameter is the registration parameter that has the highest weighted value among the normalized mutual information score, edge matching score and spatial correspondence score of the target stained image and the honeycomb image in the current layer.

[0019] In this embodiment, a 5-layer image pyramid will be constructed for the target stained image and the honeycomb image respectively through iterative downsampling:

[0020]

[0021] in, For the original image (i.e., the image with the highest resolution), the downsampling factor can be 2.

[0022] The hierarchical resolution relationship of a multi-level Gaussian image pyramid is as follows:

[0023]

[0024] in, This represents the total number of levels. Let be the width of the i-th pyramid image. Let represent the height of the i-th pyramid image. Level 0 represents the lowest resolution, and Level 4 represents the highest resolution.

[0025] The formula for calculating the optimal number of levels in a multi-level Gaussian image pyramid is:

[0026]

[0027] in, For the minimum level size, This represents the maximum number of levels.

[0028] When constructing a multi-layer Gaussian image pyramid for angle and position searches, the search range of each layer, except the initial layer, is determined based on the optimal registration parameters identified in the previous layer. The optimal registration parameters are obtained by weighting the normalized mutual information score, edge matching score, and spatial correspondence score calculated based on the target stained image and the honeycomb image at that layer according to a set synthesis function, and determining the maximum weighted value. The weighting coefficients in the weighting process can be adaptively adjusted based on the actual application scenario (such as the actual staining quality of different species and tissues) through grid search or Bayesian optimization.

[0029] As an example, normalized mutual information scoring can be used to measure the statistical correlation between a target stained image and a cellular image in terms of grayscale distribution. For example, it can be achieved by calculating the joint histogram, information entropy, and mutual information of the two images in the overlapping region through geometric mean normalization or symmetric normalization, and then normalizing the values ​​to stabilize them within the range of [0, 1]. Edge matching scoring measures the degree of geometric alignment between two images on the edges of tissue structures and the boundaries of cellular grids. For example, it can be achieved by extracting edge features from the two images (e.g., through gradient operators, Canny detectors, or deep learning edge detection models), then evaluating the overlap ratio of edge pixels and the consistency of edge directions, and quantifying it through the product of overlap rate and direction consistency. Edge features have high discriminative power for geometric information such as tissue contours and chip grid structures, effectively supplementing the shortcomings of mutual information scoring in single-texture regions. Spatial correspondence scoring measures whether regions in the cellular image with expressed signals (UMI count > 0) spatially overlap with tissue regions in the HE image. For example, it can be achieved by calculating the overlap between the tissue mask and the cellular signal mask and weighting it with a signal intensity contrast factor.

[0030] In other possible implementations, a boundary gradient score can be introduced for weighting. The boundary gradient score is used to evaluate the alignment quality of tissue slices and microarray cell images at tissue boundary regions. For example, using the tissue boundary extracted from the tissue segmentation mask of an HE image as the region of interest, the similarity of the gradient magnitude or gradient direction field in this region between the two images is compared. Specifically, the boundary gradient similarity can be calculated using methods such as normalized cross-correlation, gradient structure similarity, or the reciprocal of the mean square error.

[0031] In method 100, step 106 can perform angle and position search layer by layer from low resolution layer to high resolution layer in the multi-layer Gaussian image pyramid to determine the target registration parameters corresponding to the highest resolution layer, so as to register the target stained image and the honeycomb image according to the target registration parameters.

[0032] In this embodiment, a coarse-to-fine search strategy can be adopted to perform angle and position searches layer by layer in the multi-layer Gaussian image pyramid. The angle search range of the initial layer (i.e., Level 0) can be... , angle step size is The position search step size is adaptively adjusted according to the search area. For non-initial layers (Levels 1-4), the angle search range is the optimal angle of the upper layer. , angle step size is The position search range is the neighborhood of the optimal position in the upper layer. The dynamic calculation process of the angle search range is as follows:

[0033]

[0034]

[0035] in, For hierarchical indexes, For the maximum angle, For the first The search range of the layer For the first Layer step size.

[0036] The initial search area for location search is:

[0037]

[0038] in, The width of the current level of the stained image. The width of the current layer of the cellular image. The height of the current level of the stained image. This represents the height of the current layer of the cellular image.

[0039] The search region for the non-initial layer based on the results of the upper layer is:

[0040]

[0041]

[0042] in, For the first The center of the layer.

[0043] The final target registration parameters can include position coordinates (i.e., the pixel position of the center of the cell pattern in the stained image), rotation angle (i.e., the rotation angle of the cell pattern relative to the stained image), flip state (which can be a Boolean value used to indicate whether the cell pattern needs to be flipped horizontally), scaling factor (i.e., the scaling ratio of the cell pattern), transparency (i.e., the transparency parameter when overlaid), etc. By adjusting the cell pattern according to the target registration parameters, the automatic registration of the target stained image and the cell pattern can be completed.

[0044] In one possible implementation, acquiring a target stained image obtained after preprocessing a tissue section includes:

[0045] Obtain the original stained image of the tissue section, and perform contrast-limited adaptive histogram equalization on the original stained image based on a preset cropping threshold to obtain an enhanced grayscale image;

[0046] The enhanced grayscale image is segmented into organizational regions to obtain a binarized result.

[0047] Morphological optimization is performed on the binarization result to obtain the target stained image.

[0048] In this embodiment, after obtaining the original stained image of the tissue section, contrast-limited adaptive histogram equalization can be performed on the original stained image. The purpose is to divide the image into multiple small tiles, perform histogram equalization on each tile independently, and simultaneously limit the contrast amplification factor to prevent the number of pixels at certain gray levels from far exceeding the average, leading to excessive contrast amplification and noise amplification. Specifically, for the original image gray-level histogram corresponding to the original stained image... The normalized cumulative distribution function is:

[0049]

[0050] in, This represents the total number of pixels in the image.

[0051] Next, you can set the cropping threshold. Contrast limitation is achieved by cropping the histogram:

[0052]

[0053] Pixels exceeding the threshold are redistributed uniformly:

[0054]

[0055] in, The grayscale level is usually 256.

[0056] After completing the above processing, an enhanced grayscale image corresponding to the original stained image can be obtained. Next, the tissue region can be automatically segmented using the Segment Anything Model (SAM), or segmented using Otsu thresholding, yielding a binarized result. Taking Otsu thresholding as an example, let the threshold be... The foreground pixel ratio is The background pixel ratio is Then the variance between classes is:

[0057]

[0058] in, and These are the average grayscale values ​​for the background and foreground, respectively. Next, an optimal threshold is applied. Maximize the inter-class variance to complete the segmentation:

[0059]

[0060] Finally, morphological optimization is performed on the binarized result to eliminate noise and fill holes. Specifically, this can be done by first performing an opening operation (erosion followed by dilation):

[0061]

[0062] Then perform the closing operation (dilation followed by erosion):

[0063]

[0064] in, For the input image, For structural elements (e.g., a 5x5 elliptical kernel). Indicates corrosion. It indicates expansion.

[0065] In one possible implementation, generating a honeycomb image corresponding to a tissue slice includes:

[0066] Obtain the chip design file for the spatial transcriptome and convert the physical coordinates of each probe in the chip design file into the first pixel coordinates;

[0067] The hexagonal contour coordinate set of each probe is calculated based on the first pixel coordinate, and color mapping is performed according to the hexagonal contour coordinate set and the gene expression level of each probe to generate a honeycomb image corresponding to the tissue slice.

[0068] In this embodiment, the physical coordinates of each probe in the chip design file are first converted into the coordinates of the first pixel in the image:

[0069]

[0070]

[0071] in, , For chip physical dimensions, , The pixel size of the target colored image. , These are the horizontal and vertical coordinates of the physical coordinates.

[0072] Next, taking an Agilent chip as an example, a hexagon is drawn at each probe position, constructing a hexagonal contour coordinate set. The formula for calculating vertex coordinates is:

[0073] For a vertical hexagonal arrangement:

[0074]

[0075] For a horizontal hexagonal arrangement:

[0076]

[0077] in, , , The side length of the hexagon is... These are the coordinates of the probe center.

[0078] The side length of the hexagon is determined by the probe spacing. calculate:

[0079]

[0080] Finally, a piecewise normalization function will be used to map gene expression levels (i.e., UMI counts) to a color space to obtain a cellular image. Let the UMI threshold (tissue / background boundary point) be... x is the UMI count of a single probe, and the background color range percentage is... :

[0081]

[0082] Among them, threshold Determined by percentiles:

[0083]

[0084] Where X is the set of all probe UMI count values, and tissue_ratio is the tissue proportion parameter (the proportion of tissue in the HE diagram).

[0085] In one possible implementation, the method further includes:

[0086] Affine transformation of the honeycomb image based on the target staining image;

[0087] Calculate the joint histogram of the target stained image and the transformed honeycomb image within the effective region, calculate the information entropy and mutual information based on the joint histogram, and calculate the normalized mutual information score corresponding to the information entropy and mutual information based on the geometric mean method.

[0088] The gradient magnitude and gradient direction between the target stained image and the transformed honeycomb image are calculated based on the Sobel operator. The edge overlap rate and directional consistency in the edge overlap region are calculated based on the gradient magnitude and gradient direction, and the edge matching score is calculated based on the edge overlap rate and directional consistency.

[0089] The overlap ratio of the tissue mask region and the honeycomb signal region in the target stained image and the transformed honeycomb image is calculated, and the signal contrast is calculated based on the average honeycomb pixel value in the tissue mask region and the average honeycomb pixel value in the background region, so as to calculate the spatial correspondence score according to the overlap ratio and the signal contrast.

[0090] The normalized mutual information score, edge matching score, and spatial correspondence score are weighted to obtain a weighted value.

[0091] In this embodiment, the cellular image is first subjected to an affine transformation, which includes rotation, translation, scaling, and flipping. The rotation matrix is:

[0092]

[0093] in, The angle is the rotation angle.

[0094] The scaling matrix is:

[0095]

[0096] in, This is the scaling ratio.

[0097] The flip matrix (horizontal flip) is:

[0098]

[0099] The complete affine transformation matrix is:

[0100]

[0101] in, Indicates flipping. Indicates no flipping. The translation amount, Let (1, 1) be the (1, 1)th element of the rotation matrix. Let (1, 2) be the (1, 2)th element of the rotation matrix. Let (2, 1) be the element of the rotation matrix. It is the (2,2)th element of the rotation matrix.

[0102] Pixel coordinate transformation can be expressed as:

[0103]

[0104] in, This is the center point of the cellular diagram.

[0105] The calculation process for the normalized mutual information score is as follows: the grayscale value of 0-255 is divided into 32 intervals (bins). Let the H&E grayscale image corresponding to the target stained image be... The honeycomb grayscale image is , This represents the grayscale index of the H&E grayscale image. This represents the grayscale index of the cellular grayscale image, within the valid region. Within, the joint histogram of the two images (i.e., simultaneously satisfying the condition that the H&E image has a gray level of 1) And the grayscale level of the honeycomb pattern is )for:

[0106]

[0107] Next, the counts are converted into normalized joint probabilities:

[0108]

[0109] Without considering the cellular pattern, H&E image pixels fall into the grayscale level. The edge probability, and when H&E images are not considered, the number of cellular map pixels falling into the gray level. The marginal probability is:

[0110]

[0111] Based on the different probabilities mentioned above, different information entropies can be calculated. Higher entropy indicates a more uniform grayscale distribution (greater information content); lower entropy indicates a more concentrated distribution. The general formula for calculating information entropy is:

[0112]

[0113]

[0114] The formula for calculating mutual information is:

[0115]

[0116] Finally, the normalized mutual information score will be calculated using the geometric mean method:

[0117]

[0118] The calculation process for the edge matching score is as follows: Let... Represents the set of edge pixels in the HE image. This represents the set of edge pixels in a cellular map. The Sobel gradient calculation formula is:

[0119]

[0120] in, Let be the gray level (0-31) of the image at (x, y).

[0121] gradient magnitude With gradient direction for:

[0122]

[0123] During edge detection, only the top 25% of pixels by gradient magnitude are retained as edge points:

[0124]

[0125] The edge direction consistency is:

[0126]

[0127] Edge matching score:

[0128]

[0129] The spatial correspondence score is calculated by first measuring how much of the gene expression signal in the cellular image is located within the tissue region identified by the H&E staining image, i.e., calculating the spatial overlap. The formula is as follows:

[0130]

[0131] in, To organize the mask area, This represents the signal region in the cellular map.

[0132] Next, the signal contrast is calculated:

[0133]

[0134] in, The average signal strength in the tissue region. This represents the average signal intensity of the background region. A signal contrast greater than 0 indicates that the signal within the tissue region is stronger than the background (normal, as genes are expressed within the tissue). A signal contrast equal to 0 indicates that the signal intensity inside and outside the tissue is the same (indistinguishable). A signal contrast less than 0 indicates that the background signal is stronger than the signal within the tissue (abnormal).

[0135] The formula for calculating spatial correspondence score is:

[0136]

[0137] This calculation method results in a minimum value of 0.7, where overlap contributes 70% even when the contrast is 0, and a maximum value of 1.0, where overlap contributes completely when the contrast is 1. Furthermore, contrast provides a maximum bonus of 30%.

[0138] The formula for calculating the weighted value is:

[0139]

[0140] in, That is , , , For weighted calculations, the weights can be set to 1.0, 0.2, and 0.2 respectively, so that the normalized mutual information dominates the calculation.

[0141] In one possible implementation, the target stained image and the honeycomb image are registered according to target registration parameters, including:

[0142] Based on the target registration parameters, the physical coordinates of each probe in the spatial transcriptome are converted into the second pixel coordinates on the target staining image.

[0143] In this embodiment, based on the target registration parameters The physical coordinates of each probe can be mapped to the second pixel coordinates of the target stained image. Specifically, a coordinate transformation is first performed:

[0144]

[0145]

[0146] if Then flip and adjust:

[0147]

[0148] Next, centralization will be implemented:

[0149]

[0150]

[0151] Next, perform a rotation transformation:

[0152]

[0153] Finally, translate to the coordinate system of the stained image:

[0154]

[0155]

[0156] In one possible implementation, the method further includes:

[0157] In the lowest resolution layer of the multi-layer Gaussian image pyramid, the weighted values ​​of the original honeycomb pattern and the horizontally mirrored honeycomb pattern under the optimal registration parameters of the current layer are calculated respectively, and the flip state corresponding to the honeycomb pattern with the higher weighted value is locked.

[0158] In this embodiment, at the lowest resolution layer (i.e., Level 0), the original state of the cellular map (i.e., ) and horizontal flip state (i.e. The test is conducted to determine the optimal flip state based on the state with the higher weighting value, and this flip state is locked so that subsequent layers use the locked flip state for fine registration, avoiding registration failure caused by inconsistent tissue section orientation.

[0159] In one possible implementation, after registering the target stained image and the honeycomb image according to the target registration parameters, the method further includes:

[0160] Abnormal probes not located within the tissue region are identified based on the target stained image, and the probe data corresponding to the abnormal probes are filtered.

[0161] In this embodiment, the stained tissue mask corresponding to the target stained image can also be used to determine whether each probe is located within the tissue region. The calculation formula is as follows:

[0162]

[0163] in, This is a binary tissue mask. Probes outside the tissue region will be treated as anomalous probes, and their collected probe data will be filtered to avoid affecting accuracy.

[0164] Figure 2This diagram illustrates the structure of an automatic image registration apparatus 200 for tissue sections according to some embodiments of the present disclosure. The various embodiments in this specification are described in a progressive manner; similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments. Figure 2 As shown, the device 200 includes an image acquisition module 201, configured to acquire a target stained image obtained after preprocessing a tissue slice, and generate a honeycomb image corresponding to the tissue slice; a Gaussian image pyramid construction module 202, configured to construct multi-layer Gaussian image pyramids for the target stained image and the honeycomb image respectively. The search range of the non-initial layer of the multi-layer Gaussian image pyramid is determined based on the optimal registration parameter of the previous layer. The optimal registration parameter is the registration parameter with the highest weighted value among the normalized mutual information score, edge matching score, and spatial correspondence score of the target stained image and the honeycomb image in the current layer. The normalized mutual information score is used to characterize the statistical correlation of the image in grayscale distribution, the edge matching score is used to characterize the geometric alignment of the image on the edge of the tissue structure and the boundary of the honeycomb grid, and the spatial correspondence score is used to characterize the degree of spatial overlap between the expression signal region of the image and the tissue region; and an image registration module 203, configured to perform angle and position searches layer by layer from low resolution layer to high resolution layer in the multi-layer Gaussian image pyramid to determine the target registration parameter corresponding to the highest resolution layer, so as to register the target stained image and the honeycomb image according to the target registration parameter.

[0165] In one possible implementation, the image acquisition module 201 is further configured to acquire the original stained image of the tissue section, perform contrast-limited adaptive histogram equalization on the original stained image based on a preset cropping threshold to obtain an enhanced grayscale image; perform tissue region segmentation on the enhanced grayscale image to obtain a binarization result; and perform morphological optimization on the binarization result to obtain the target stained image.

[0166] In one possible implementation, the image acquisition module 201 is further configured to acquire the chip design file of the spatial transcriptome, convert the physical coordinates of each probe in the chip design file into first pixel coordinates, calculate the hexagonal contour coordinate set of each probe based on the first pixel coordinates, and perform color mapping according to the hexagonal contour coordinate set and the gene expression level of each probe to generate a honeycomb image corresponding to the tissue slice.

[0167] In one possible implementation, the apparatus further includes a weighted calculation module configured to perform an affine transformation on the cellular image based on the target stained image; calculate a joint histogram of the target stained image and the transformed cellular image within the effective region to calculate information entropy and mutual information based on the joint histogram, and calculate a normalized mutual information score corresponding to the information entropy and mutual information based on the geometric mean method; calculate the gradient magnitude and gradient direction between the target stained image and the transformed cellular image based on the Sobel operator to calculate the edge overlap rate and directional consistency in the edge overlap region based on the gradient magnitude and gradient direction, and calculate an edge matching score based on the edge overlap rate and directional consistency; calculate the overlap area ratio of the tissue mask region and the cellular signal region in the target stained image and the transformed cellular image, and calculate the signal contrast based on the mean cellular pixel value in the tissue mask region and the mean cellular pixel value in the background region, to calculate a spatial correspondence score based on the overlap area ratio and signal contrast; and weight the normalized mutual information score, the edge matching score, and the spatial correspondence score to obtain a weighted value.

[0168] In one possible implementation, the image registration module 203 is further configured to convert the physical coordinates of each probe of the spatial transcriptome into second pixel coordinates on the target staining image based on the target registration parameters.

[0169] In one possible implementation, the device further includes a state locking module configured in the lowest resolution layer of the multi-layer Gaussian image pyramid, which calculates the weighted values ​​of the original cell image and the horizontally mirrored cell image under the optimal registration parameters of the current layer, and locks the flipped state corresponding to the cell image with the higher weighted value.

[0170] In one possible implementation, the image registration module 203 is further configured to identify anomalous probes that are not located within the tissue region based on the target stained image, and to filter the probe data corresponding to the anomalous probes.

[0171] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this specification is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0172] Figure 3 A block diagram of an electronic device 300 that can implement various embodiments of the present disclosure is shown. For example... Figure 3 As shown, the electronic device 300 includes a processor 310, a disk drive 320, an input / output interface 330, a network interface 340, and a memory 350. The processor 310, disk drive 320, input / output interface 330, network interface 340, and memory 350 can communicate with each other via a communication bus 360.

[0173] The processor 310 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided in this application.

[0174] The memory 350 can be implemented in the form of ROM (Read Only Memory), RAM (Read Access Memory), static memory, dynamic storage devices, etc. The memory 350 can store the operating system 351 used to control the operation of the electronic device 300, and the basic input / output system (BIOS) 352 used to control the low-level operations of the electronic device 300. Additionally, it can store a web browser 353, a data storage management system 354, etc. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 350 and is called and executed by the processor 310.

[0175] Input / output interface 330 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, vibrators, indicator lights, etc.

[0176] Network interface 340 is used to connect a communication module (not shown in the figure) to enable communication and interaction between the device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0177] Bus 360 includes a pathway for transmitting information between various components of the device, such as processor 310, disk drive 320, input / output interface 330, network interface 340, and memory 350.

[0178] It should be noted that although the above-described device only shows the processor 310, disk drive 320, input / output interface 330, network interface 340, memory 350, bus 360, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the method of this application, and does not necessarily include all the components shown in the figures.

[0179] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0181] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An automatic image registration method for tissue sections, characterized in that, The method includes: Obtain the target stained image of the tissue section after preprocessing, and generate the corresponding honeycomb image of the tissue section; A multi-layer Gaussian image pyramid is constructed for the target stained image and the honeycomb image respectively. The search range of the non-initial layer of the multi-layer Gaussian image pyramid is determined based on the optimal registration parameter of the previous layer. The optimal registration parameter is the registration parameter that has the highest weighted value among the normalized mutual information score, edge matching score and spatial correspondence score of the target stained image and the honeycomb image in the current layer. The normalized mutual information score is used to characterize the statistical correlation of the image in gray-level distribution, the edge matching score is used to characterize the geometric alignment of the image on the edge of the tissue structure and the boundary of the honeycomb grid, and the spatial correspondence score is used to characterize the degree of spatial overlap between the expression signal region and the tissue region of the image. In a multi-layer Gaussian image pyramid, angle and position searches are performed layer by layer from low-resolution to high-resolution layers to determine the target registration parameters corresponding to the highest resolution layer, so as to register the target stained image and the honeycomb image according to the target registration parameters.

2. The method for automatic image registration of tissue sections according to claim 1, characterized in that, The acquisition of the target stained image obtained after preprocessing the tissue sections includes: Obtain the original stained image of the tissue section, and perform contrast-limited adaptive histogram equalization on the original stained image based on a preset cropping threshold to obtain an enhanced grayscale image. The enhanced grayscale image is segmented into organizational regions to obtain a binarized result; Morphological optimization is performed on the binarization result to obtain the target stained image.

3. The automatic image registration method for tissue sections according to claim 1, characterized in that, Generating the honeycomb image corresponding to the tissue slice includes: Obtain the chip design file for the spatial transcriptome and convert the physical coordinates of each probe in the chip design file into the first pixel coordinates; Based on the first pixel coordinates, calculate the hexagonal contour coordinate set of each probe, and perform color mapping according to the hexagonal contour coordinate set and the gene expression level of each probe to generate a honeycomb image corresponding to the tissue slice.

4. The automatic image registration method for tissue sections according to claim 1, characterized in that, The method further includes: Affine transformation of the honeycomb image based on the target staining image; Calculate the joint histogram of the target stained image and the transformed honeycomb image within the effective region, calculate the information entropy and mutual information based on the joint histogram, and calculate the normalized mutual information score corresponding to the information entropy and mutual information based on the geometric mean method. The gradient magnitude and gradient direction between the target stained image and the transformed cellular image are calculated based on the Sobel operator, so as to calculate the edge overlap rate and directional consistency in the edge overlap region based on the gradient magnitude and gradient direction, and to calculate the edge matching score based on the edge overlap rate and directional consistency. The overlap ratio of the tissue mask region and the honeycomb signal region in the target stained image and the transformed honeycomb image is calculated, and the signal contrast is calculated based on the average value of the honeycomb pixels in the tissue mask region and the average value of the honeycomb pixels in the background region, so as to calculate the spatial correspondence score according to the overlap ratio and the signal contrast. The normalized mutual information score, edge matching score, and spatial correspondence score are weighted to obtain a weighted value.

5. The automatic image registration method for tissue sections according to claim 1, characterized in that, The registration of the target stained image and the honeycomb image according to the target registration parameters includes: Based on the target registration parameters, the physical coordinates of each probe in the spatial transcriptome are converted into the second pixel coordinates on the target staining image.

6. The method for automatic image registration of tissue sections according to claim 1, characterized in that, The method further includes: In the lowest resolution layer of the multi-layer Gaussian image pyramid, the weighted values ​​of the original honeycomb pattern and the horizontally mirrored honeycomb pattern under the optimal registration parameters of the current layer are calculated respectively, and the flip state corresponding to the honeycomb pattern with the higher weighted value is locked.

7. The automatic image registration method for tissue sections according to claim 1, characterized in that, After registering the target stained image and the honeycomb image according to the target registration parameters, the process further includes: Abnormal probes that are not located within the tissue region are identified based on the target stained image, and the probe data corresponding to the abnormal probes are filtered.

8. An automatic image registration device for tissue sections, characterized in that, The device includes: The image acquisition module is configured to acquire the target stained image obtained after preprocessing the tissue slices, and generate a honeycomb image corresponding to the tissue slices; The Gaussian image pyramid construction module is configured to construct multi-layer Gaussian image pyramids for the target stained image and the honeycomb image respectively. The search range of the non-initial layer of the multi-layer Gaussian image pyramid is determined based on the optimal registration parameter of the previous layer. The optimal registration parameter is the registration parameter that has the highest weighted value among the normalized mutual information score, edge matching score and spatial correspondence score of the target stained image and the honeycomb image in the current layer. The normalized mutual information score is used to characterize the statistical correlation of the image in gray-level distribution, the edge matching score is used to characterize the geometric alignment of the image on the edge of the tissue structure and the boundary of the honeycomb grid, and the spatial correspondence score is used to characterize the degree of spatial overlap between the expression signal region and the tissue region of the image. The image registration module is configured to perform angle and position searches layer by layer in a multi-layer Gaussian image pyramid, from low-resolution layers to high-resolution layers, to determine the target registration parameters corresponding to the highest resolution layer, so as to register the target stained image and the honeycomb image according to the target registration parameters.

9. An electronic device, characterized in that, include: One or more processors, and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the automatic image registration method for tissue slices according to any one of claims 1-7.

10. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements an automatic image registration method for tissue slices according to any one of claims 1-7.

Images