Multi-modal cell image feature extraction method, electronic device and program product

Abstract

The invention provides a multi-modal cell image feature extraction method, electronic equipment and a program product, and relates to the technical field of image recognition. The method comprises the following steps: acquiring a multi-modal image set obtained by shooting the same pathological cell sample under different microscopes; pre-processing the multi-modal image set to obtain a pre-processed multi-modal image set, and taking the pre-processed multi-modal image set as input data; performing spatial registration on each cell image in the input data by using a preset spatial registration strategy to obtain registered input data; performing feature fusion on each cell image in the registered input data by using a preset feature fusion strategy to obtain a fusion feature; and according to the fusion features, carrying out feature extraction on an abnormal region in the pathological cell sample to generate an abnormal region thermodynamic diagram. Thus, the problems that in a traditional cell anomaly recognition task, cell feature extraction depends on a single-mode cell image, feature loss is likely to be caused, and the cell anomaly recognition result is inaccurate are solved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a multimodal cell image feature extraction method, electronic equipment, and program product. Background Art

[0002] With the development and popularization of AI (Artificial Intelligence) technology, medical image analysis technology based on image recognition has become an indispensable auxiliary means in disease diagnosis and treatment.

[0003] In the diagnosis and treatment of hematological diseases, it is often necessary to detect abnormalities in blood cell images based on image recognition technology to assist doctors in making pathological judgments. In traditional cell abnormality detection, image feature extraction typically relies on single-modality cell images.

[0004] However, due to factors such as light and shooting angle, single-modality cell images may not be fully imaged under the microscope, and some features may be lost during the image feature extraction process, ultimately leading to inaccurate cell abnormality identification results. Summary of the Invention

[0005] In view of this, the purpose of the embodiments of the present application is to provide a multimodal cell image feature extraction method, electronic device and program product, which can improve the problem of traditional cell abnormality identification tasks in which cell feature extraction relies on a single modality of cell images, which easily leads to feature loss and inaccurate cell abnormality identification results.

[0006] To achieve the above technical objectives, the technical solutions adopted in this application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a multimodal cell image feature extraction method, the method comprising:

[0008] Acquire a multimodal image set obtained by photographing the same pathological cell sample under different microscopes;

[0009] Preprocessing the multimodal image set to obtain a preprocessed multimodal image set as input data;

[0010] Using a preset spatial registration strategy, spatially register each cell image in the input data to obtain registered input data;

[0011] Using a preset feature fusion strategy, feature fusion is performed on each cell image in the registered input data to obtain a fusion feature;

[0012] Based on the fusion features, feature extraction is performed on the abnormal region in the pathological cell sample to generate a heat map of the abnormal region.

[0013] In conjunction with the first aspect, in some optional implementations, preprocessing the multimodal image set to obtain a preprocessed multimodal image set as input data includes:

[0014] According to the resolution of each cell image in the multimodal image set, bilinear interpolation is performed on each cell image to obtain multiple cell images with the same resolution;

[0015] performing grayscale normalization on the multiple cell images with the same resolution to obtain multiple normalized cell images;

[0016] The normalized multiple cell images are filtered to suppress noise interference in the normalized multiple cell images, and the preprocessed multimodal image set consisting of the multiple cell images after noise reduction is obtained as the input data.

[0017] In conjunction with the first aspect, in some optional embodiments, using a preset spatial registration strategy to perform spatial registration on each cell image in the input data to obtain the registered input data includes:

[0018] Selecting a plurality of nonlinear control points from each cell image in the input data;

[0019] Performing a radial transformation on the input data according to the plurality of nonlinear control points to obtain transformed data;

[0020] Perform spatial registration on each cell image in the transformed data to obtain the registered input data.

[0021] In conjunction with the first aspect, in some optional implementations, performing a radial transformation on the input data according to the multiple nonlinear control points to obtain transformed data includes:

[0022] For each modality cell image in the input data, any modality cell image is used as a reference modality image, and a transformation matrix between each modality cell image and the reference modality image is determined according to the nonlinear control points:

[0023]

[0024] Where, T k represents the transformation matrix, M represents the number of modes, represents the mth control point in the cell image of k mode, represents the mth control point in the reference modality image,

[0025] According to the transformation matrix, the input data is affine transformed to obtain the transformed data:

[0026]

[0027] Where G k represents the cell image of the kth mode in the transformed data, R k represents the cell image of the kth mode in the input data, and δ represents the preset sampling function.

[0028] In conjunction with the first aspect, in some optional embodiments, performing spatial registration on each cell image in the transformed data to obtain the registered input data includes:

[0029] Using a preset scale-invariant feature exchange matching strategy, feature extraction is performed on each cell image in the exchange data to obtain a description subset of key points of each cell image in the exchange data, where the key points are pixels representing corner points, spots, or edge endpoints in each cell image in the exchange data;

[0030] According to the Euclidean distance between each descriptor in the descriptor subset, the key points corresponding to each descriptor in the descriptor subset are screened using a preset random sampling consistency strategy to obtain a reliable matching point set corresponding to each cell image in the exchange data;

[0031] Using a preset thin plate spline transformation strategy, aligning the reliable matching point sets between different cell images in the transformation data to obtain an aligned reliable matching point set;

[0032] An interpolation operation is performed on the aligned reliable matching point set corresponding to each cell image in the exchange data to obtain a registered pixel point set corresponding to each cell image in the exchange data as the registered input data.

[0033] In conjunction with the first aspect, in some optional embodiments, a preset feature fusion strategy is used to perform feature fusion on each cell image in the registered input data to obtain a fusion feature, including:

[0034] For each modality of cell image in the input data, performing multi-scale feature extraction on the cell image using a preset feature extraction network to obtain a first target feature map;

[0035] Performing global average pooling on the first target feature map to obtain a feature vector corresponding to the cell image of each modality in the input data;

[0036] Obtaining a dynamic weight corresponding to each eigenvector in the eigenvector corresponding to the cell image of each modality in the input data;

[0037] The feature vectors corresponding to the cell images of each modality in the input data are weightedly fused using the dynamic weights to obtain a fused feature vector as the fused feature.

[0038] In conjunction with the first aspect, in some optional embodiments, feature extraction is performed on the abnormal region in the pathological cell sample based on the fusion feature to generate a heat map of the abnormal region, including:

[0039] Concatenating the fusion feature and the first target feature map to obtain a second target feature map;

[0040] Determining an anomaly score for each pixel in the second target feature map according to the second target feature map;

[0041] Normalizing the anomaly score to obtain an anomaly probability for each pixel in the second target feature map;

[0042] The abnormal area heat map is generated according to the abnormal probability and the registered input data.

[0043] In conjunction with the first aspect, in some optional implementations, generating the abnormal area heat map according to the abnormal probability and the registered input data includes:

[0044] Based on a preset mask strategy, performing image segmentation on the cell image with the highest contrast in the registered input data to obtain a cell mask representing the cell area;

[0045] Performing connected domain analysis on the cell mask to obtain multiple cell regions;

[0046] For each of the plurality of cell regions, determining an average abnormality probability of each cell region according to the abnormality probability;

[0047] The abnormal area heat map is generated according to the average abnormal probability.

[0048] In a second aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory coupled to each other, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the electronic device executes the above-mentioned method.

[0049] In a third aspect, an embodiment of the present application further provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0050] The invention adopting the above technical solution has the following advantages:

[0051] In the technical solution provided in the present application, a multimodal image set is first obtained, and the multimodal image set is preprocessed to obtain input data. Then, a preset spatial registration strategy is used to perform spatial registration on each cell image in the input data to obtain the registered input data. Then, a preset feature fusion strategy is used to perform feature fusion on each cell image in the registered input data to obtain fusion features. Finally, based on the fusion features, feature extraction is performed on the abnormal area in the pathological cell sample to generate a heat map of the abnormal area. In this way, by performing spatial registration and feature extraction on cell images under multiple modalities, the ability of cell images to express cell features is improved, the robustness of cell image feature extraction is enhanced, the accuracy of subsequent cell abnormality recognition is improved, and the problem that cell feature extraction relies on cell images of a single modality in traditional cell abnormality recognition tasks is improved, which easily causes feature loss and inaccurate cell abnormality recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present application may be further illustrated by the non-limiting embodiments provided in the accompanying drawings. It should be understood that the following drawings illustrate only certain embodiments of the present application and are therefore not to be construed as limiting the scope of the present application. It is understood that a person skilled in the art can derive other relevant drawings from these drawings without inventive effort.

[0053] Figure 1 This is a structural block diagram of the electronic device provided in an embodiment of the present application.

[0054] Figure 2 A schematic flow chart of the multimodal cell image feature extraction method provided in an embodiment of the present application.

[0055] Icon: 100-electronic device; 101-processor; 102-memory. DETAILED DESCRIPTION

[0056] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts in the drawings or descriptions are numbered the same. Implementations not shown or described in the drawings are known to those of ordinary skill in the art. In the description of this application, the terms "first," "second," etc. are used solely to distinguish descriptions and are not to be construed as indicating or implying relative importance.

[0057] Please refer to Figure 1 In an embodiment of the present application, an electronic device 100 may include a processor 101 and a memory 102. The memory 102 stores a computer program, and when the computer program is executed by the processor 101, the electronic device 100 can perform the corresponding steps in the following multimodal cell image feature extraction method.

[0058] In this embodiment, the processor 101 may be an integrated circuit chip having signal processing capabilities. The processor 101 may be a general-purpose processor 101. For example, the processor 101 may be a central processing unit (CPU), a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0059] The memory 102 may be, but is not limited to, a random access memory 102, a read-only memory 102, a programmable read-only memory 102, an erasable programmable read-only memory 102, an electrically erasable programmable read-only memory 102, etc. In this embodiment, the memory 102 may be used to store a multimodal image set, input data, a preset spatial registration strategy, registered input data, a preset feature fusion strategy, fusion features, abnormal region heat maps, etc. Of course, the memory 102 may also be used to store programs, and the processor 101 executes the program after receiving an execution instruction.

[0060] It is understandable that Figure 1 The structure of the electronic device 100 shown in FIG is only a schematic diagram of a structure. The electronic device 100 may also include Figure 1 More components shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0061] In this embodiment, the electronic device 100 can be a personal computer, laptop computer, cloud server, etc., and is used to acquire and preprocess a multimodal image set to obtain input data. A preset spatial registration strategy is then used to spatially register the cell images in the input data to obtain registered input data. A preset feature fusion strategy is then used to perform feature fusion on the cell images in the registered input data to obtain fused features. Finally, based on the fused features, features are extracted from abnormal regions in the pathological cell sample to generate a heat map of the abnormal regions.

[0062] Please refer to Figure 2The present application also provides a multimodal cell image feature extraction method, which can be applied to the above-mentioned electronic device 100, and each step of the method is executed or implemented by the electronic device 100. The multimodal cell image feature extraction method may include the following steps:

[0063] Step 210 , obtaining a multimodal image set obtained by photographing the same pathological cell sample under different microscopes;

[0064] Step 220 , preprocessing the multimodal image set to obtain a preprocessed multimodal image set as input data;

[0065] Step 230, using a preset spatial registration strategy, spatially registering each cell image in the input data to obtain registered input data;

[0066] Step 240, using a preset feature fusion strategy, performing feature fusion on each cell image in the registered input data to obtain a fusion feature;

[0067] Step 250 : Extract features of the abnormal region in the pathological cell sample based on the fusion features to generate a heat map of the abnormal region.

[0068] In the above-mentioned embodiment, a multimodal image set is first acquired and preprocessed to obtain input data. Then, a preset spatial registration strategy is used to spatially register each cell image in the input data to obtain the registered input data. Then, a preset feature fusion strategy is used to perform feature fusion on each cell image in the registered input data to obtain fusion features. Finally, based on the fusion features, feature extraction is performed on the abnormal areas in the pathological cell sample to generate a heat map of the abnormal areas. In this way, by spatially registering and extracting features from cell images under multiple modalities, the ability of cell images to express cell features is improved, the robustness of cell image feature extraction is enhanced, and the accuracy of subsequent cell abnormality recognition is improved. This improves the problem in traditional cell abnormality recognition tasks where cell feature extraction relies on cell images of a single modality, which easily leads to feature loss and inaccurate cell abnormality recognition results.

[0069] The following is a detailed description of the steps of the multimodal cell image feature extraction method:

[0070] In step 210, the multimodal image set can be acquired by photographing the same pathological cell sample under different microscopes in real time to obtain multiple cell images, with each cell image photographed under a different microscope being considered a cell image of a modality. These multiple cell images are then sent in real time as the multimodal image set to the processor 101 of the electronic device 100 for subsequent feature extraction. Alternatively, the multimodal image set can be acquired by pre-photographing the same pathological cell sample under different microscopes during development testing to obtain multiple cell images, storing the collection of these multiple cell images as a multimodal image set in the memory 102 of the electronic device 100, and then accessing the multimodal image set based on instructions issued by the user through the processor 101 during the subsequent cell image feature extraction process. The method for acquiring the multimodal image set is not specifically limited herein.

[0071] In this embodiment, the specific type of microscope can be flexibly set according to user needs. This embodiment uses a transmitted light microscope, a fluorescence microscope, a phase contrast microscope, and a differential interference contrast microscope. That is, the multimodal image set proposed in this embodiment includes cell images of four modalities.

[0072] In step 220, the method according to claim 1 is characterized in that preprocessing the multimodal image set to obtain a preprocessed multimodal image set as input data may include:

[0073] According to the resolution of each cell image in the multimodal image set, bilinear interpolation is performed on each cell image to obtain multiple cell images with the same resolution:

[0074]

[0075] Where, I k represents the cell image of the kth modality in the multimodal image set, represents the cell image of the kth mode after bilinear interpolation, represents the scaling factor, d k represents the resolution of the cell image of the kth modality, d min Indicates the minimum resolution of all cell images in the multimodal image set;

[0076] Grayscale normalization is performed on the multiple cell images of the same resolution to obtain multiple normalized cell images:

[0077]

[0078] Where, Represents the cell image of the kth mode after grayscale normalization of channel c, satisfying μc , σ c They represent the mean and standard deviation of the cell image of the kth modality in channel c, respectively. γ represents the contrast enhancement factor. In this embodiment, γ=1.2. After completing the grayscale normalization of channel c, the normalized cell image of the kth modality is obtained.

[0079] Filtering the normalized multiple cell images to suppress noise interference in the normalized multiple cell images, and obtaining the preprocessed multimodal image set consisting of the multiple denoised cell images as the input data:

[0080]

[0081] Where, represents the spatial gradient of the normalized k-th modality cell image in the x-axis direction, represents the spatial gradient of the normalized cell image of the kth mode in the y-axis direction, represents the result of filtering the normalized cell image of the kth modality after the tth iteration, g x 、g y Represent the conduction coefficients of the cell image of the kth mode on the x and y axes, Indicates the gradient blur threshold that controls the smoothing strength of the iterative filter. div stands for divergence;

[0082] Then, the normalized cell image of the kth modality is iteratively filtered using the following formula until the number of iterations is greater than or equal to the preset number, and the result obtained from the last iterative filtering is used as the preprocessed cell image of the kth modality. The collection of multiple modalities (four modalities in this embodiment) after iterative filtering is the input data:

[0083]

[0084] Wherein, λ represents the iterative step length that affects the convergence speed, λ≤0.25, and in this embodiment, λ=0.1.

[0085] In this way, through the above preprocessing, the feature deviation caused by scale differences in the feature extraction process of each cell image in the multimodal image set is avoided, and the robustness of the image and feature is enhanced.

[0086] In step 230, spatial registration is performed on each cell image in the input data using a preset spatial registration strategy to obtain registered input data, which may include:

[0087] Selecting a plurality of nonlinear control points from each cell image in the input data;

[0088] Performing a radial transformation on the input data according to the plurality of nonlinear control points to obtain transformed data;

[0089] Perform spatial registration on each cell image in the transformed data to obtain the registered input data.

[0090] In this embodiment, the control points may be selected by performing Harris corner point detection or SIFT feature point detection on the cell image in the input data, and using at least three corner points or feature points obtained by the detection as control points.

[0091] In this embodiment, performing a radial transformation on the input data according to the multiple nonlinear control points to obtain transformed data may include:

[0092] For each modality cell image in the input data, any modality cell image is used as a reference modality image, and a transformation matrix between each modality cell image and the reference modality image is determined according to the nonlinear control points:

[0093]

[0094] Where, T k represents the transformation matrix, M represents the number of modes, represents the mth control point in the cell image of k mode, represents the mth control point in the reference modality image,

[0095] Where Affine(2) represents the two-dimensional affine transformation group:

[0096]

[0097] Wherein, the submatrix composed of the upper left 2×2 elements is used to control the rotation, scaling and shearing in the affine transformation, and the submatrix composed of the upper right 2×1 elements is used to represent the translation along the x-axis and y-axis in the affine transformation.

[0098] According to the transformation matrix, the input data is affine transformed to obtain the transformed data:

[0099]

[0100] Where G k represents the cell image of the kth mode in the transformed data, R krepresents the cell image of the kth mode in the input data, and δ represents a preset sampling function. The preset sampling function may be a Dirac sampling function.

[0101] In this embodiment, performing spatial registration on each cell image in the transformed data to obtain the registered input data may include:

[0102] Using a preset scale-invariant feature exchange matching strategy, feature extraction is performed on each cell image in the exchange data to obtain a description subset of key points of each cell image in the exchange data, where the key points are pixels representing corner points, spots, or edge endpoints in each cell image in the exchange data;

[0103] According to the Euclidean distance between each descriptor in the descriptor subset, the key points corresponding to each descriptor in the descriptor subset are screened using a preset random sampling consistency strategy to obtain a reliable matching point set corresponding to each cell image in the exchange data;

[0104] Using a preset thin plate spline transformation strategy, aligning the reliable matching point sets between different cell images in the transformation data to obtain an aligned reliable matching point set;

[0105] An interpolation operation is performed on the aligned reliable matching point set corresponding to each cell image in the exchange data to obtain a registered pixel point set corresponding to each cell image in the exchange data as the registered input data.

[0106] In this embodiment, feature extraction is performed on each cell image in the exchange data using a preset scale-invariant feature exchange matching strategy (i.e., a scale-invariant feature exchange matching algorithm, Scale Invariant Feature Transform, SIFT), as specifically expressed as follows:

[0107] For the cell image G of the kth modality in the exchange data k Perform multi-scale Gaussian blur:

[0108]

[0109] Where, L k (x,y,ε) represents G k The result after Gaussian blurring, where ε represents the initial scale of Gaussian blurring. In this embodiment, ε=1.6;

[0110] According to G k The result after Gaussian blurring generates a Gaussian difference image:

[0111] D k (x,y,ε)=Lk (x,y,k s ε)-L k (x,y,ε) (15)

[0112] Where k s represents the multiplication factor of adjacent scales, S represents the number of scale space layers, in this embodiment S=3;

[0113] The pixel points that meet the preset conditions in the Gaussian difference image are used as key points. The preset conditions indicate that the current pixel point is an extreme point (including maximum and minimum values) among the adjacent 26 pixels.

[0114] The result L of the key point after Gaussian blur is calculated by the following formula k The gradient magnitude and direction of the neighborhood in (x,y,ε) (i.e., Gaussian image):

[0115]

[0116] Where m represents the neighborhood gradient amplitude of the current key point, θ represents the neighborhood direction of the current key point, and the neighborhood refers to the area with a radius of 3×1.5ε centered on the current key point.

[0117] Finally, according to the gradient magnitude and direction of the neighborhood of each key point, a descriptor matrix is ​​generated as a descriptor set, and the matrix shape is N k ×128, N k Indicates the number of key points.

[0118] In this embodiment, based on the Euclidean distance between each descriptor in the descriptor subset, a preset random sampling consistency strategy is used to screen the key points corresponding to each descriptor in the descriptor subset to obtain a reliable matching point set corresponding to each cell image in the exchange data, including:

[0119] The cell images of different modalities in the registered input data are paired without duplication. For each pair of cell images, the Euclidean distance between the key points of the two cell images in each pair of cell images is calculated according to the description subset corresponding to each pair of cell images, and the key points with Euclidean distance less than the preset threshold are retained as the initial reliable matching point set:

[0120] Determine each descriptor in the cell image of the kth modality in each pair of cell images Each descriptor in the cell image of l modalities The Euclidean distance between:

[0121]

[0122] Where, Representation descriptor and descriptors The Euclidean distance between

[0123] Traverse each pair of cell images, each descriptor in the cell image of the lth modality and The Euclidean distance of and determine the minimum and second minimum values;

[0124] Determine the descriptor based on the minimum value and the second minimum value and The nearest neighbor ratio of , and retain the key points whose nearest neighbor ratio is less than the preset threshold as the initial reliable matching point set:

[0125]

[0126] In the formula, Matches k→l represents the initial reliable matching point set, represents the nearest neighbor ratio, j and j′ represent the indexes of the minimum value and the second minimum value respectively, and represents the preset threshold. In this embodiment, 0.8 is used as an example;

[0127] Then, the initial reliable matching point set is screened again by the RANSAC (RANdom SAmple Consensus) algorithm to obtain the above reliable matching point set.

[0128] In this embodiment, a preset thin plate spline transformation strategy is used to align the reliable matching point sets between different cell images in the transformation data to obtain an aligned reliable matching point set, including:

[0129] The Thin Plate Spline (TPS) algorithm is used to align the reliable matching point sets between different cell images in the transformed data to obtain the aligned reliable matching point set. The aligned reliable matching point set is then bilinearly interpolated to obtain the registered input data.

[0130] In this way, the interference of local cell deformation and outliers on subsequent feature extraction is reduced, and the robustness of image registration is enhanced.

[0131] In step 240, a preset feature fusion strategy is used to perform feature fusion on each cell image in the registered input data to obtain a fusion feature, which may include:

[0132] For each modality of cell image in the input data, performing multi-scale feature extraction on the cell image using a preset feature extraction network to obtain a first target feature map;

[0133] Performing global average pooling on the first target feature map to obtain a feature vector corresponding to the cell image of each modality in the input data;

[0134] Obtaining a dynamic weight corresponding to each eigenvector in the eigenvector corresponding to the cell image of each modality in the input data;

[0135] The feature vectors corresponding to the cell images of each modality in the input data are weightedly fused using the dynamic weights to obtain a fused feature vector as the fused feature.

[0136] In this embodiment, the preset feature extraction network may be ResNet-50. In this embodiment, ResNet-50 is used to extract a first target feature map of the cell image of each modality in the input data, and then global average pooling is performed on the first target feature map to obtain a feature vector corresponding to the cell image of each modality in the input data.

[0137] After determining the eigenvectors corresponding to the cell images of each modality in the input data, the dynamic weights of the eigenvectors corresponding to the cell images of different modalities are determined by the genetic algorithm (GA). The fitness function of the genetic algorithm is as follows:

[0138]

[0139] Where w (g) represents the weight group in the iteration, O represents the number of samples, w prior represents the initial weight group, Represents the feature vector corresponding to the cell image of the kth mode in the input data, y i represents the preset true label, Indicates the preset regularization strength. In this embodiment

[0140] In this embodiment, after determining the dynamic weights of the feature vectors corresponding to cell images of different modalities, the feature vectors are weighted fused using the dynamic weights to obtain the fused features:

[0141]

[0142] In the formula, H represents the fusion feature, represents the dynamic weight corresponding to the cell image of the kth mode, F k represents the eigenvector corresponding to the cell image of the kth mode, ⊙ represents element-wise multiplication, W a Represents a trainable weight matrix.

[0143] In this way, by fusing image features of different modalities, cross-modal complementary information is retained, and the sensitivity and reliability of subsequent abnormal feature extraction are improved.

[0144] In step 250, feature extraction is performed on the abnormal region in the pathological cell sample based on the fusion feature to generate a heat map of the abnormal region, which may include:

[0145] Concatenating the fusion feature and the first target feature map to obtain a second target feature map;

[0146] Determining an anomaly score for each pixel in the second target feature map according to the second target feature map;

[0147] Normalizing the anomaly score to obtain an anomaly probability for each pixel in the second target feature map;

[0148] The abnormal area heat map is generated according to the abnormal probability and the registered input data.

[0149] In this embodiment, the fusion feature and the first target feature map are spliced ​​to obtain the second target feature map:

[0150]

[0151] Where, Represents the concatenation operation, Conv 1×1 Represents a convolution operation with a convolution kernel size of 1×1, H spatial Represents the second target feature map, F (l) represents the first target feature map;

[0152] After obtaining the second target feature map, determine the anomaly score of each pixel in the second target feature map:

[0153]

[0154] Where, Σ represents the mean and covariance matrix of the predefined normal samples;

[0155] Then the anomaly score is normalized to obtain the anomaly probability of each pixel in the second woody feature map:

[0156]

[0157] Where, P anomaly represents the abnormal probability, S max 、S min represent the maximum and minimum values ​​in the anomaly score, respectively.

[0158] In this embodiment, generating the abnormal area heat map according to the abnormal probability and the registered input data may include:

[0159] Based on a preset mask strategy, performing image segmentation on the cell image with the highest contrast in the registered input data to obtain a cell mask representing the cell area;

[0160] Performing connected domain analysis on the cell mask to obtain multiple cell regions;

[0161] For each of the plurality of cell regions, determining an average abnormality probability of each cell region according to the abnormality probability;

[0162] The abnormal area heat map is generated according to the average abnormal probability.

[0163] The preset mask strategy includes extracting the cell mask through threshold segmentation and morphological operations:

[0164]

[0165] Wherein, Threshold represents the threshold segmentation, MorphClose represents the morphological operation, r represents the radius of the closing operation kernel in the morphological operation, in this embodiment r = 5, ν represents the adaptive threshold, M cell represents the cell mask;

[0166] Then, the cell mask is subjected to connected domain analysis to obtain multiple cell regions:

[0167]

[0168] Where ConnectedComponents represents the connected domain analysis, and U represents the number of cell regions;

[0169] For each cell region, determine the average abnormal probability of the current cell region:

[0170]

[0171] Where, P cell (i) represents the average abnormal probability of the i-th cell region;

[0172] Then render a heatmap based on the average anomaly probability of each cell region:

[0173]

[0174] Where M heat represents the preliminary rendered heat map, Ⅱ(·) represents the indicator function, i.e., it takes 1 when (x, y) belongs to the cell region and 0 otherwise;

[0175] After the preliminary rendering of the heat map is completed, Gaussian smoothing is performed on the preliminary rendered heat map to obtain the above-mentioned abnormal area heat map.

[0176] In this way, the normalization of abnormal probability usually quantifies the degree of cell pathology, reducing the probability of misidentification. Then, through cell mask segmentation combined with connected domain analysis, abnormal cell clusters are accurately located and thermal map rendering and Gaussian smoothing of abnormal cell clusters are performed to achieve feature extraction of cell images, that is, to obtain the thermal map of abnormal cells, which is convenient for providing medical staff with accurate reference materials when making pathological judgments, thereby improving the efficiency and accuracy of pathological judgments.

[0177] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the electronic device 100 described above can refer to the corresponding processes of each step in the aforementioned method, and will not be elaborated here.

[0178] An embodiment of the present application further provides a computer program product, including a computer program, which implements the above-mentioned multimodal cell image feature extraction method when executed by the processor 101.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented through hardware or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0180] In summary, the embodiments of the present application provide a multimodal cell image feature extraction method, electronic device and program product. In this technical solution, a multimodal image set is first acquired, and the multimodal image set is preprocessed to obtain input data. Then, a preset spatial registration strategy is used to spatially register each cell image in the input data to obtain the registered input data. Then, a preset feature fusion strategy is used to perform feature fusion on each cell image in the registered input data to obtain a fusion feature. Finally, based on the fusion feature, feature extraction is performed on the abnormal area in the pathological cell sample to generate a heat map of the abnormal area. In this way, by spatially registering and extracting features of cell images under multiple modalities, the ability of cell images to express cell features is improved, the robustness of cell image feature extraction is enhanced, the accuracy of subsequent cell abnormality recognition is improved, and the problem that cell feature extraction relies on cell images of a single modality in traditional cell abnormality recognition tasks, which easily causes feature loss and inaccurate cell abnormality recognition results, is improved.

[0181] In the embodiments provided in the present application, it should be understood that the disclosed method can also be implemented in other ways. The method embodiments described above are merely schematic. For example, the flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of code, and a part of the module, program segment or code includes one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0182] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal cell image feature extraction method, characterized in that: The method comprises: Acquire a multimodal image set obtained by photographing the same pathological cell sample under different microscopes; Preprocessing the multimodal image set to obtain a preprocessed multimodal image set as input data; Using a preset spatial registration strategy, spatially register each cell image in the input data to obtain registered input data; Using a preset feature fusion strategy, feature fusion is performed on each cell image in the registered input data to obtain a fusion feature; Based on the fusion features, feature extraction is performed on the abnormal region in the pathological cell sample to generate a heat map of the abnormal region.

2. The method according to claim 1, characterized in that Preprocessing the multimodal image set to obtain a preprocessed multimodal image set as input data includes: According to the resolution of each cell image in the multimodal image set, bilinear interpolation is performed on each cell image to obtain multiple cell images with the same resolution; performing grayscale normalization on the multiple cell images with the same resolution to obtain multiple normalized cell images; The normalized multiple cell images are filtered to suppress noise interference in the normalized multiple cell images, and the preprocessed multimodal image set consisting of the multiple cell images after noise reduction is obtained as the input data.

3. The method according to claim 1, characterized in that Using a preset spatial registration strategy, spatially register each cell image in the input data to obtain registered input data, including: Selecting a plurality of nonlinear control points from each cell image in the input data; Performing a radial transformation on the input data according to the plurality of nonlinear control points to obtain transformed data; Perform spatial registration on each cell image in the transformed data to obtain the registered input data.

4. The method according to claim 3, characterized in that Performing a radial transformation on the input data according to the plurality of nonlinear control points to obtain transformed data includes: For each modality cell image in the input data, any modality cell image is used as a reference modality image, and a transformation matrix between each modality cell image and the reference modality image is determined according to the nonlinear control points: Where, T k represents the transformation matrix, M represents the number of modes, represents the mth control point in the cell image of k mode, represents the mth control point in the reference modality image, According to the transformation matrix, the input data is affine transformed to obtain the transformed data: Where G k represents the cell image of the kth mode in the transformed data, R k represents the cell image of the kth mode in the input data, and δ represents the preset sampling function.

5. The method according to claim 3, characterized in that Performing spatial registration on each cell image in the transformed data to obtain the registered input data includes: Using a preset scale-invariant feature exchange matching strategy, feature extraction is performed on each cell image in the exchange data to obtain a description subset of key points of each cell image in the exchange data, where the key points are pixels representing corner points, spots, or edge endpoints in each cell image in the exchange data; According to the Euclidean distance between each descriptor in the descriptor subset, the key points corresponding to each descriptor in the descriptor subset are screened using a preset random sampling consistency strategy to obtain a reliable matching point set corresponding to each cell image in the exchange data; Using a preset thin plate spline transformation strategy, aligning the reliable matching point sets between different cell images in the transformation data to obtain an aligned reliable matching point set; An interpolation operation is performed on the aligned reliable matching point set corresponding to each cell image in the exchange data to obtain a registered pixel point set corresponding to each cell image in the exchange data as the registered input data.

6. The method according to claim 1, characterized in that Using a preset feature fusion strategy, feature fusion is performed on each cell image in the registered input data to obtain fusion features, including: For each modality of cell image in the input data, performing multi-scale feature extraction on the cell image using a preset feature extraction network to obtain a first target feature map; Performing global average pooling on the first target feature map to obtain a feature vector corresponding to the cell image of each modality in the input data; Obtaining a dynamic weight corresponding to each eigenvector in the eigenvector corresponding to the cell image of each modality in the input data; The feature vectors corresponding to the cell images of each modality in the input data are weightedly fused using the dynamic weights to obtain a fused feature vector as the fused feature.

7. The method according to claim 6, characterized in that Extracting features of abnormal regions in the pathological cell sample based on the fusion features to generate a heat map of the abnormal regions includes: Concatenating the fusion feature and the first target feature map to obtain a second target feature map; Determining an anomaly score for each pixel in the second target feature map according to the second target feature map; Normalizing the anomaly score to obtain an anomaly probability for each pixel in the second target feature map; The abnormal area heat map is generated according to the abnormal probability and the registered input data.

8. The method according to claim 6, characterized in that Generating the abnormal area heat map according to the abnormal probability and the registered input data, including: Based on a preset mask strategy, performing image segmentation on the cell image with the highest contrast in the registered input data to obtain a cell mask representing the cell area; Performing connected domain analysis on the cell mask to obtain multiple cell regions; For each of the plurality of cell regions, determining an average abnormality probability of each cell region according to the abnormality probability; The abnormal area heat map is generated according to the average abnormal probability.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory coupled to each other, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images