A method for information fusion between fluorescence images and pathological sections for lymphoma

Abstract

This invention relates to the field of image data processing technology, specifically to a method for information fusion between fluorescence images and pathological sections of lymphoma. The method involves acquiring a first image and a second image of a lymphoma tissue sample; the first image is a bright-field image, and the second image is a FISH image. In the first image, the degree of difference between each cell and normal cells, as well as the degree of difference between each tissue boundary and normal tissue boundaries, are analyzed to obtain cell abnormality indices and tissue abnormality indices, respectively. Based on the cell abnormality indices and tissue abnormality indices, a section abnormality index is determined, and the initial block size is adjusted based on the section abnormality index to obtain a target block size. Using the target block size as the block size, a two-stream convolutional network is used to fuse the first image and the second image to obtain a fused image. This invention improves the image fusion effect between FISH images and bright-field images of lymphoma pathological sections.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and specifically to a method for fusing information from fluorescence images and pathological sections of lymphoma. Background Technology

[0002] Integrating information from fluorescence images and pathological sections to aid diagnosis is a diagnostic approach that combines fluorescence in situ hybridization (FISH) technology with morphological analysis of pathological sections. This method can improve the accuracy and reliability of lymphoma diagnosis for healthcare professionals. FISH technology is a fluorescently labeled gene detection method that uses fluorescently labeled probes to hybridize with nucleic acids in patient pathological tissue samples. The fluorescent signal emitted when the probe binds to specific gene sequences in lymphoma cells indicates gene alterations. Pathological sections, on the other hand, involve sectioning and staining tissue to observe the morphology and structure of cells under a microscope. Integrating these two methods combines the genetic information from FISH images with the morphological information from pathological sections, providing a more comprehensive basis for diagnosis.

[0003] During the application, it was found that when fusing FISH and bright-field images of lymphoma pathological sections using a dual-stream convolutional network, the image fusion effect was poor. This made it difficult for medical personnel to accurately classify lymphoma subtypes by referring to the fused image in subsequent lymphoma pathological sections. Summary of the Invention

[0004] The purpose of this invention is to provide a method for information fusion of fluorescence images and pathological sections for lymphoma, in order to solve the technical problem of poor image fusion effect of FISH images and bright field images of lymphoma pathological sections in the prior art.

[0005] In a first aspect, one embodiment of this application provides a method for fusing information from fluorescence images and pathological sections for lymphoma, the method comprising:

[0006] Acquire a first image and a second image of a target pathological sample, wherein the target pathological sample is a lymphoma tissue sample, the first image is a bright field image acquired based on the target pathological sample, and the second image is a FISH image acquired based on the target pathological sample;

[0007] In the first image, the degree of difference between each cell and normal cells is analyzed to obtain the cell abnormality index;

[0008] In the first image, the degree of difference between the boundaries of each tissue and the boundaries of normal tissue is analyzed to obtain the tissue abnormality index;

[0009] The slice abnormality index is determined based on the cell abnormality index and the tissue abnormality index, and the initial segment size is adjusted based on the slice abnormality index to obtain the target segment size;

[0010] Using the target block size as the block size, a two-stream convolutional network is used to fuse the first image and the second image to obtain a fused image.

[0011] In one embodiment, analyzing the degree of difference between each cell and normal cells in the first image to obtain a cell abnormality index includes:

[0012] In the first image, the degree of pixel disorder in each cell nucleus is analyzed to obtain the nuclear staining entropy value;

[0013] In the first image, the degree of morphological distortion of each cell nucleus is analyzed to obtain the nuclear deformation value;

[0014] In the first image, the degree of size difference of each cell and the nuclear size fluctuation value are analyzed;

[0015] The cell abnormality index is obtained based on the nuclear staining entropy value, the nuclear deformation value, and the nuclear size fluctuation value.

[0016] In one embodiment, analyzing the pixel disorder of each cell nucleus in the first image to obtain the nuclear staining entropy value includes:

[0017] The first image is processed to obtain a grayscale image;

[0018] In the grayscale image, the connected regions corresponding to the cell nuclei are identified to obtain multiple nuclear connected domains;

[0019] In the multiple kernel connected domains, the grayscale differences of pixels located within the kernel connected domains are analyzed to obtain the kernel staining entropy value.

[0020] In one embodiment, the step of analyzing the degree of morphological distortion of each cell nucleus in the first image to obtain nuclear deformation values ​​includes:

[0021] Based on the multiple kernel connected components, the grayscale image is binarized to obtain a binarized image, wherein the binarized image is used to distinguish between pixels located within the kernel connected components and pixels located outside the kernel connected components.

[0022] The binarized image is subjected to a closing operation to obtain a shape-restored image;

[0023] The kernel deformation value is obtained by analyzing the difference between the pixel distribution of the shape-restored image and the pixel distribution of the binarized image.

[0024] In one embodiment, analyzing the degree of size difference and nuclear size fluctuation value of each cell in the first image includes:

[0025] In the multiple nuclear connected domains, the area of ​​each nuclear connected domain and the area of ​​the corresponding cell connected domain are obtained to obtain the areas of multiple nuclear domains and multiple cell domains.

[0026] The dispersion of the areas of the multiple kernel regions is analyzed to obtain the discrete values ​​of the kernel region areas;

[0027] The dispersion of the areas of the multiple cell domains is analyzed to obtain the discrete values ​​of the cell domain areas;

[0028] The nuclear size fluctuation value is obtained based on the discrete values ​​of the nuclear domain area and the discrete values ​​of the cell domain area.

[0029] In one embodiment, in the first image, the degree of difference between each tissue boundary and the normal tissue boundary is analyzed to obtain a tissue abnormality index, including:

[0030] Edge detection is performed on the first image to obtain multiple edge lines and the gradient magnitude of each pixel in the first image;

[0031] Among the multiple edge lines, the gradient magnitude difference between the boundary points and their adjacent pixels on each edge line, as well as the length of each edge line, are analyzed to determine the boundary blur index of each edge line.

[0032] Among the multiple edge lines, the degree of boundary morphology anomaly of each edge line is analyzed to obtain the boundary morphology anomaly index of each edge line;

[0033] The boundary distortion value of each edge line is determined based on the boundary fuzziness index and the boundary morphology anomaly index of each edge line.

[0034] The tissue abnormality index is obtained based on the boundary distortion values ​​of multiple edge lines.

[0035] In one embodiment, the step of analyzing the gradient magnitude difference between boundary points and their adjacent pixels on each edge line, as well as the length of each edge line, to determine the boundary blur index of each edge line includes:

[0036] In the plurality of edge lines, the gradient magnitude difference between the boundary points and their adjacent pixels on each edge line is analyzed to determine the point blur index of each boundary point on each edge line;

[0037] Among all the boundary points included by the plurality of edge lines, the boundary points with a boundary fuzziness index greater than the fuzziness threshold are determined as the fuzzy boundary points of the corresponding edge lines;

[0038] Among the plurality of edge lines, the boundary fuzziness index of each edge line is obtained based on the length of each edge line, the total number of fuzzy boundary points included in each edge line, and the average value of the point fuzziness index corresponding to all fuzzy boundary points included in each edge line.

[0039] In one embodiment, analyzing the gradient magnitude difference between boundary points and their adjacent pixels on each of the plurality of edge lines to determine the point blur index of each boundary point on each edge line includes:

[0040] Multiple target line segments are determined based on all boundary points included by the multiple edge lines, wherein the center point of the target line segment is a boundary point included by one of the edge lines, the normal of the edge line where the center point of the target line segment is located is the extension direction of the target line segment, and the length of the target line segment is less than or equal to a length threshold.

[0041] In the plurality of target line segments, the average gradient magnitude of multiple non-center pixels of each target line segment is calculated to obtain multiple first gradient abrupt change values;

[0042] In the multiple target line segments, the difference between the maximum non-center gradient magnitude of each target line segment and the gradient magnitude of its center point is analyzed to obtain multiple second gradient mutation values, wherein the maximum non-center gradient magnitude is: the largest gradient magnitude among the gradient magnitudes of multiple non-center pixels of the corresponding target line segment;

[0043] Based on the plurality of first gradient mutation values ​​and the plurality of second gradient mutation values, a plurality of point fuzziness indices corresponding one-to-one with the plurality of target line segments are obtained.

[0044] In one embodiment, analyzing the degree of boundary morphology anomaly of each of the plurality of edge lines to obtain a boundary morphology anomaly index for each edge line includes:

[0045] Among the multiple edge lines, the chain code value difference between adjacent boundary points in each edge line is analyzed to obtain multiple chain code differences for each edge line;

[0046] Among the multiple edge lines, the average value of the multiple chain code differences for each edge line is calculated to obtain the boundary morphology anomaly index for each edge line.

[0047] In one embodiment, adjusting the initial block size based on the slice anomaly index to obtain the target block size includes:

[0048] Based on the slice anomaly index and the initial correction coefficient, a target correction coefficient is obtained, wherein the slice anomaly index and the target correction coefficient are negatively correlated.

[0049] The target block size is determined by multiplying the target correction coefficient by the initial block size.

[0050] Secondly, another embodiment of the present invention provides an information fusion system for fluorescence images and pathological sections of lymphoma, the system comprising:

[0051] The image acquisition module is used to acquire a first image and a second image of the target pathological sample, wherein the target pathological sample is a lymphoma tissue sample, the first image is a bright field image acquired based on the target pathological sample, and the second image is a FISH image acquired based on the target pathological sample;

[0052] The cell analysis module is used to analyze the degree of difference between each cell and normal cells in the first image to obtain a cell abnormality index;

[0053] The tissue analysis module is used to analyze the degree of difference between the boundaries of each tissue and the boundaries of normal tissue in the first image to obtain the tissue abnormality index;

[0054] The segment size determination module is used to determine the slice abnormality index based on the cell abnormality index and the tissue abnormality index, and adjust the initial segment size based on the slice abnormality index to obtain the target segment size;

[0055] The image fusion module is used to fuse the first image and the second image using the target block size as the block size and a two-stream convolutional network to obtain a fused image.

[0056] Thirdly, in another embodiment of the present invention, an electronic device is provided, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect.

[0057] Fourthly, in another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0058] The present invention has the following beneficial effects:

[0059] This invention analyzes the degree to which each cell in the first image differs from normal cells, effectively assessing the severity of lymphoma lesions at the cellular level; and analyzes the degree to which the boundaries of each tissue in the first image differ from normal tissue boundaries, effectively assessing the severity of lymphoma lesions at the tissue level. Combining these two analyses allows for an accurate assessment of the overall severity of lymphoma lesions in the first image. Based on this, the initial block size is dynamically adjusted, and the adjusted block size is used for image fusion, ensuring that the block size better matches the severity of lymphoma lesions observed in the corresponding pathological sections. When the lymphoma lesion is mild, the size of the segment used can be appropriately increased to avoid over-processing of image details during fusion, while ensuring that the network's cross-modal attention can fully associate image information between different modalities. When the lymphoma lesion is severe, the size of the segment used can be appropriately decreased to ensure that the network's cross-modal attention accurately associates image information between different modalities, while ensuring that key image details are fully preserved during fusion. This improves the image fusion effect of FISH and bright-field images of lymphoma pathological sections, providing more effective data support for subsequent diagnosis and treatment by medical personnel. Attached Figure Description

[0060] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating a method for fusing information from fluorescence images and pathological sections for lymphoma, as provided in an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram of the structure of an information fusion system for fluorescence images and pathological sections of lymphoma provided in an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0064] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a method for information fusion of fluorescence images and pathological sections for lymphoma proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0066] The following describes in detail, with reference to the accompanying drawings, a specific scheme of the information fusion method for fluorescence images and pathological sections for lymphoma provided by the present invention.

[0067] In one embodiment, this invention provides a method for fusing information from fluorescence images and pathological sections for lymphoma, such as... Figure 1 As shown, the method includes:

[0068] Step S1: Obtain the first and second images of the target pathological sample.

[0069] The target pathological sample is a lymphoma tissue sample, the first image is a bright field image acquired based on the target pathological sample, and the second image is a FISH image acquired based on the target pathological sample.

[0070] The aforementioned lymphoma tissue sample can be understood as lymphoma tissue obtained from a patient (through lymph node puncture sampling or surgical resection, etc.).

[0071] In this invention, a serial slicing method is specifically used to obtain a first image corresponding to HE technology and a second image corresponding to FISH technology. That is, the target pathological sample is serially sliced ​​(the slice thickness can be 3μm to 5μm), and two consecutive slices are selected. One slice is stained with HE and a corresponding bright-field image (i.e., the first image) is acquired, while the other slice is FISH processed and a corresponding FISH image (i.e., the second image) is acquired. This is to ensure that the sample details shown in the first image and the sample details shown in the second image are as consistent as possible, while ensuring that both HE and FISH technologies can be effectively applied.

[0072] For example, the process of acquiring the first image and the second image can be as follows:

[0073] After obtaining the lymphoma tissue sample, it was sequentially sliced, and any two adjacent slices were selected, which are here defined as the first slice and the second slice.

[0074] The first section undergoes fixation (using formaldehyde to preserve tissue morphology), dehydration (using alcohol to remove moisture), embedding (using paraffin to solidify the tissue), and sectioning (using a microtome to cut thin slices 3-5 micrometers thick). Afterward, it is mounted on a glass slide and stained with hematoxylin and eosin (hematoxylin staining of cell nuclei appears blue-purple, eosin staining of cytoplasm appears pink). Finally, it is sealed with a coverslip to prepare a long-term preserved hematoxylin and eosin pathological section. A bright-field microscope is then prepared, and the light source brightness, focus, and white balance are adjusted to ensure uniform light in the field of view and avoid reflections or brightness deviations affecting image quality. The aforementioned first image is obtained by photographing the hematoxylin and eosin pathological section.

[0075] The second section undergoes dewaxing, hydration, protease digestion (to remove tissue proteins and allow the probe to penetrate cells), and denaturation (to unwind the DNA within the cells, facilitating probe binding) to prepare for probe hybridization. Then, specific fluorescent probes targeting lymphoma-related genes (such as red fluorescent probes for detecting the MYC gene and green fluorescent probes for detecting the BCL6 gene) are selected and added to the pretreated sample. The sample is then placed in a hybridization instrument and incubated for 12-24 hours at a specific temperature (usually 37°C) and humidity to allow the probes to specifically bind to the target gene sequences within the cells of the second section. Unbound free probes are then removed with washing buffer to avoid non-specific fluorescence interference. Finally, the cell nuclei are counterstained with DAPI dye (blue fluorescence) to visualize them, facilitating the location of the fluorescent signal in the cells. The resulting second image is obtained using a fluorescence microscope (equipped with a specific wavelength excitation light source and filter; for example, a 550nm light source is needed to excite red fluorescence, and a 488nm light source is needed for green fluorescence).

[0076] In this invention, after obtaining the first image and the second image, the first image and the second image are further standardized (e.g., image size unification) and aligned using image registration technology (using the SIFT algorithm based on feature points) to compensate for cell position differences between adjacent slices as much as possible. It should be understood that all subsequent processing of the first image and the second image in this invention is based on the first image and the second image after standardization and image registration.

[0077] It should be noted that in the diagnosis of lymphoma subtypes, the first image is more important than the second image (which is only used to indicate cells that may have gene alterations in the sample and their distribution). Therefore, in the process of fusing the first and second images, more reliance should be placed on the image information shown by the first image to assess the degree of lymphoma lesions in the corresponding sample, and the block size should be dynamically adjusted during image fusion accordingly.

[0078] Step S2: In the first image, analyze the degree of difference between each cell and normal cells to obtain the cell abnormality index.

[0079] Specifically, the step of analyzing the degree of difference between each cell and normal cells in the first image to obtain a cell abnormality index includes:

[0080] In the first image, the degree of pixel disorder in each cell nucleus is analyzed to obtain the nuclear staining entropy value;

[0081] In the first image, the degree of morphological distortion of each cell nucleus is analyzed to obtain the nuclear deformation value;

[0082] In the first image, the degree of size difference of each cell and the nuclear size fluctuation value are analyzed;

[0083] The cell abnormality index is obtained based on the nuclear staining entropy value, the nuclear deformation value, and the nuclear size fluctuation value.

[0084] Because the nuclei of normal lymphocytes are mostly round or oval, uniform in size, with dense nuclear chromatin and non-protruding nucleoli, while the nuclei of lymphoma cells show significant morphological distortions, and the distortion characteristics of different subtypes are significantly different. For example, the nuclei of lymphoma cells are extremely heterogeneous in size, and the nuclei can be polygonal, irregular, lobed, or even exhibit "nuclear folding" or "nuclear twisting", resulting in sparse nuclear chromatin and prominent nucleoli.

[0085] Therefore, this invention selects to accurately quantify the degree of abnormality of lymphoma cells reflected in the first image from multiple dimensions such as pixel distribution, morphology and size of cell nuclei, so as to accurately assess the degree of lesion of lymphoma tissue in the corresponding sample of the first image from a cellular perspective.

[0086] It should be understood that the nuclear staining entropy value is used to represent the degree of pixel disorder in each cell nucleus in the first image. The nuclear deformation value is used to represent the degree of morphological distortion in each cell nucleus in the first image. The nuclear size fluctuation value is used to represent the degree of size difference among cells in the first image.

[0087] In one example, the cell abnormality index can be the average of the nuclear staining entropy value, the nuclear deformation value, and the nuclear size fluctuation value.

[0088] Further, the step of analyzing the pixel disorder of each cell nucleus in the first image to obtain the nuclear staining entropy value includes:

[0089] The first image is processed to obtain a grayscale image;

[0090] In the grayscale image, the connected regions corresponding to the cell nuclei are identified to obtain multiple nuclear connected domains;

[0091] In the multiple kernel connected domains, the grayscale differences of pixels located within the kernel connected domains are analyzed to obtain the kernel staining entropy value.

[0092] After HE staining, hematoxylin staining of cell nuclei results in a deep blue-purple color with the lowest gray value; eosin staining of cytoplasm results in a medium pink color with a moderate gray value; and the background area (without tissue components or minimal staining residue) stains the lightest with the highest gray value. Therefore, in one example, the identification process for multiple nuclear connected regions is as follows:

[0093] The K-means clustering algorithm (K value set to 3) is used to cluster all pixels in the grayscale image to obtain 3 point clusters. Then, the average gray value of all pixels in each point cluster is calculated and used as the gray value feature value of the corresponding point cluster. Among the 3 point clusters, the point cluster with the largest gray value feature value is determined as the background point cluster of the corresponding background region, the point cluster with the middle gray value feature value is determined as the particle point cluster of the corresponding cytoplasm, and the point cluster with the smallest gray value feature value is determined as the nuclear point cluster of the corresponding cell nucleus.

[0094] All pixels included in the kernel cluster are identified as kernel pixels, and the regions containing adjacent kernel pixels in the grayscale image are identified as kernel connected regions, thus obtaining the multiple kernel connected regions. It is important to emphasize that kernel connected regions can be regions with irregular boundaries or even internal splits in the image, and each kernel connected region consists only of kernel pixels.

[0095] In one example, the process of analyzing the grayscale differences of pixels located within the kernel connected regions to obtain the kernel staining entropy value is as follows:

[0096] In the plurality of kernel connected regions, the gray-level range of kernel pixels located within the kernel connected regions is analyzed to determine the first gray-level difference factor corresponding to each kernel connected region. The gray-level range is positively correlated with the corresponding first gray-level difference factor. The gray-level range is used to represent the difference between the maximum gray-level value and the minimum gray-level value among the gray-level values ​​of the plurality of kernel pixels included in the kernel connected region.

[0097] In the plurality of kernel connected regions, the kurtosis of multiple gray values ​​of kernel pixels located within the kernel connected regions is analyzed to determine the second gray difference factor corresponding to each kernel connected region. The kurtosis of multiple gray values ​​of kernel pixels within the kernel connected regions is negatively correlated with the corresponding second gray difference factor.

[0098] The mean of the first gray-level difference factor and the second gray-level difference factor corresponding to each kernel connected region is determined as the domain gray-level difference value of each kernel connected region.

[0099] The nuclear staining entropy value can be obtained by calculating the average of the domain grayscale differences of multiple kernel connected domains.

[0100] For example, the difference in grayscale values ​​within a kernel-connected region. It can be represented as:

[0101]

[0102] in, This represents the maximum gray value in the corresponding kernel connected component. This represents the minimum gray value in the corresponding kernel connected component. This represents the first gray-level difference factor corresponding to the kernel connected component. This represents the kurtosis of multiple grayscale values ​​of the kernel pixel within the corresponding kernel connected region. Represents the normalization function. This represents the second grayscale difference factor corresponding to the kernel connected region. It should be noted that the value 30 in the above formula can be adaptively adjusted according to actual needs, such as adjusting it to 25 or 35, to ensure effective normalization of the grayscale difference of kernel pixels within the kernel connected region.

[0103] Furthermore, the step of analyzing the degree of morphological distortion of each cell nucleus in the first image to obtain nuclear deformation values ​​includes:

[0104] Based on the multiple kernel connected components, the grayscale image is binarized to obtain a binarized image, wherein the binarized image is used to distinguish between pixels located within the kernel connected components and pixels located outside the kernel connected components.

[0105] The binarized image is subjected to a closing operation to obtain a shape-restored image;

[0106] The kernel deformation value is obtained by analyzing the difference between the pixel distribution of the shape-restored image and the pixel distribution of the binarized image.

[0107] Specifically, the kernel deformation value is the ratio of the number of changed pixels to the total number of pixels in the binarized image. The changed pixels are those located outside the kernel connected region in the binarized image, but inside the kernel connected region in the shape-modified image.

[0108] The larger the nuclear deformation value, the greater the degree of deformation exhibited by each cell as a whole in the first image.

[0109] For normal round or oval cell nuclei, since their boundaries are regular and they do not divide internally, the nuclear connected domains indicating the cell nucleus change little before and after the closing operation. However, for the nuclei of lymphoma cells with nuclear morphological distortions and nuclear division, the closing operation fills in the division regions within the nuclear connected domains and smooths the morphological distortions of the nuclear connected domain boundaries, resulting in significant changes in the nuclear connected domains of lymphoma cells before and after the closing operation.

[0110] Based on this, by analyzing the difference between the pixel distribution of the shape-restored image and the pixel distribution of the binarized image to obtain the nuclear deformation value, the overall deformation degree of each cell in the first image can be accurately assessed, so as to assess the lesion degree of lymphoma in the corresponding sample of the first image.

[0111] Furthermore, the analysis of the degree of size difference and nuclear size fluctuation value of each cell in the first image includes:

[0112] In the multiple nuclear connected domains, the area of ​​each nuclear connected domain and the area of ​​the corresponding cell connected domain are obtained to obtain the areas of multiple nuclear domains and multiple cell domains.

[0113] The dispersion of the areas of the multiple kernel regions is analyzed to obtain the discrete values ​​of the kernel region areas;

[0114] The dispersion of the areas of the multiple cell domains is analyzed to obtain the discrete values ​​of the cell domain areas;

[0115] The nuclear size fluctuation value is obtained based on the discrete values ​​of the nuclear domain area and the discrete values ​​of the cell domain area.

[0116] As mentioned earlier, normal lymphocytes are usually of a consistent size, while lymphoma, as a malignant tumor of the lymphohematopoietic system, has tumor cells with "clonal abnormal proliferation" that can lead to significant size differences. In the same field of view, some lymphoma cells may be significantly enlarged (e.g., the cell diameter of large B-cell lymphoma can reach 15-25 μm), while others are close to the size of normal lymphocytes (about 8-10 μm).

[0117] Based on this, the present invention selects to evaluate the size consistency of each cell in the first image by analyzing the differences in nuclear area and cell area among different cells in the first image.

[0118] It should be understood that the larger the dispersion value of the nuclear domain area, the greater the nuclear size fluctuation value; the larger the dispersion value of the cellular domain area, the greater the nuclear size fluctuation value.

[0119] The larger the nuclear size fluctuation value, the more significant the size difference between cells in the first image, the weaker the size consistency of cells in the first image, and the higher the degree of lymphoma lesion in the sample corresponding to the first image.

[0120] In applications, after identifying multiple kernel connected regions, these kernel connected regions can be used as seed points, and the watershed algorithm can be used for image processing to segment and obtain multiple cell connected regions.

[0121] The discrete value of the kernel area is specifically the normalized value of the variance of multiple kernel areas.

[0122] The discrete value of cell domain area is specifically the normalized value of the variance of the areas of multiple cell domains.

[0123] The nuclear size fluctuation value is specifically the average of the nuclear domain area dispersion value and the cell domain area dispersion value.

[0124] It should be noted that in this invention, the normalized numerical range is (0,1).

[0125] Step S3: In the first image, analyze the degree of difference between the boundaries of each tissue and the boundaries of normal tissue to obtain the tissue abnormality index.

[0126] Specifically, in the first image, the degree of difference between the boundaries of each tissue and the boundaries of normal tissue is analyzed to obtain a tissue abnormality index, including:

[0127] Edge detection is performed on the first image to obtain multiple edge lines and the gradient magnitude of each pixel in the first image;

[0128] Among the multiple edge lines, the gradient magnitude difference between the boundary points and their adjacent pixels on each edge line, as well as the length of each edge line, are analyzed to determine the boundary blur index of each edge line.

[0129] Among the multiple edge lines, the degree of boundary morphology anomaly of each edge line is analyzed to obtain the boundary morphology anomaly index of each edge line;

[0130] The boundary distortion value of each edge line is determined based on the boundary fuzziness index and the boundary morphology anomaly index of each edge line.

[0131] The tissue abnormality index is obtained based on the boundary distortion values ​​of multiple edge lines.

[0132] Lymphoid tissue composed of normal cells (such as the cortex, medulla, and lymphoid follicles) usually shows clear structural features in images. However, tumor cells can break through the capsule of normal lymphoid tissue and infiltrate the surrounding adipose tissue, blood vessels, or stroma, resulting in blurred boundaries between the diseased tissue and normal tissue. The more blurred the boundaries between tissues, the more severe the infiltration of normal lymphoid tissue by tumor cells, which in turn indicates that the degree of lymphoma may be higher.

[0133] Specifically, the shorter the length of each edge line in the first image and the smaller the difference in gradient magnitude between the boundary points on the edge lines and their adjacent pixels, the higher the degree of blurring of the boundary between tissues from the perspective of pixels; as for the more abnormal the shape of each edge line in the first image (deviating from the standard shape of a circle or ellipse), the deeper the boundary between tissues is infiltrated by tumor cells from the perspective of edge shape.

[0134] For example, edge detection processing of the first image can be performed based on the Sobel operator.

[0135] Further, the step of analyzing the gradient magnitude difference between the boundary points and their adjacent pixels on each edge line, as well as the length of each edge line, to determine the boundary blur index of each edge line includes:

[0136] In the plurality of edge lines, the gradient magnitude difference between the boundary points and their adjacent pixels on each edge line is analyzed to determine the point blur index of each boundary point on each edge line;

[0137] Among all the boundary points included by the plurality of edge lines, the boundary points with a boundary fuzziness index greater than the fuzziness threshold are determined as the fuzzy boundary points of the corresponding edge lines;

[0138] Among the plurality of edge lines, the boundary fuzziness index of each edge line is obtained based on the length of each edge line, the total number of fuzzy boundary points included in each edge line, and the average value of the point fuzziness index corresponding to all fuzzy boundary points included in each edge line.

[0139] The step of analyzing the gradient magnitude difference between boundary points and their adjacent pixels on each of the plurality of edge lines to determine the point blur index of each boundary point on each edge line includes:

[0140] Multiple target line segments are determined based on all boundary points included by the multiple edge lines, wherein the center point of the target line segment is a boundary point included by one of the edge lines, the normal of the edge line where the center point of the target line segment is located is the extension direction of the target line segment, and the length of the target line segment is less than or equal to a length threshold.

[0141] In the plurality of target line segments, the average gradient magnitude of multiple non-center pixels of each target line segment is calculated to obtain multiple first gradient abrupt change values;

[0142] In the multiple target line segments, the difference between the maximum non-center gradient magnitude of each target line segment and the gradient magnitude of its center point is analyzed to obtain multiple second gradient mutation values, wherein the maximum non-center gradient magnitude is: the largest gradient magnitude among the gradient magnitudes of multiple non-center pixels of the corresponding target line segment;

[0143] Based on the plurality of first gradient mutation values ​​and the plurality of second gradient mutation values, a plurality of point fuzziness indices corresponding one-to-one with the plurality of target line segments are obtained.

[0144] For example, the target line segment can be obtained by traversing all boundary points included in the edge line. During the traversal, the currently traversed boundary point is used as the starting point, and the line extends to both sides along the normal of the corresponding edge line until it encounters an image boundary or other boundary point or the extension length exceeds a predetermined length (such as 5). The final line segment obtained is the target line segment corresponding to the currently traversed boundary point. The above length threshold can be set to 11 based on experience.

[0145] Each edge line includes all boundary points that correspond one-to-one with multiple target line segments. In this invention, the point fuzziness index corresponding to a target line segment is specifically the normalized value of the product of the first gradient abrupt change value and the second gradient abrupt change value corresponding to the target line segment.

[0146] The point fuzziness index is used to represent the degree of fuzziness of the boundary point (i.e., the center point of the target line segment).

[0147] For example, among multiple target line segments, the first... Point fuzziness index of a target line segment It can be represented as:

[0148]

[0149] in, Represents the normalization function. Indicates the first The average gradient magnitude of multiple non-center pixels of the target line segment (i.e., the average gradient magnitude of the i-th target line segment) (the first gradient mutation value of each target line segment). Indicates the first The gradient magnitude at the center point of each target line segment Indicates the first The maximum non-central gradient magnitude of each target line segment This represents an exponential function with base e. Indicates the first The standard deviation of the gradient magnitude of each pixel in the target line segment Indicates the first The second gradient mutation value of each target line segment.

[0150] The larger the first gradient mutation value, the more significant the gray-level change on both sides of the corresponding boundary point. That is, the gray-level values ​​on both sides of the corresponding boundary point show a gradual trend, and therefore the higher the point blur index of the corresponding boundary point.

[0151] The larger the second gradient mutation value, the smaller the difference in gradient magnitude between the corresponding boundary point and the pixels on both sides. In other words, the corresponding boundary point cannot effectively separate the two sides, and therefore the higher the point blur index of the corresponding boundary point.

[0152] In this invention, the fuzzy threshold can be set to 0.5 or 0.6 based on experience.

[0153] Furthermore, the boundary fuzziness index of the edge line It can be represented as:

[0154]

[0155] in, Indicates the number of fuzzy boundary points included in the corresponding edge line. This indicates the number of boundary points included in the corresponding edge line. This represents the average point blur factor of all blurry boundary points included in the corresponding edge line.

[0156] Furthermore, the step of analyzing the degree of boundary morphology anomaly of each of the plurality of edge lines to obtain a boundary morphology anomaly index for each edge line includes:

[0157] Among the multiple edge lines, the chain code value difference between adjacent boundary points in each edge line is analyzed to obtain multiple chain code differences for each edge line;

[0158] Among the multiple edge lines, the average value of the multiple chain code differences for each edge line is calculated to obtain the boundary morphology anomaly index for each edge line.

[0159] In this invention, the Freeman chain code technique with 8 neighborhoods is used to process multiple edge lines to obtain the chain code values ​​of all boundary points included in each edge line.

[0160] The chain code difference can be understood as the absolute difference between the chain code values ​​of the corresponding boundary point in the corresponding edge line and its previous (or next) boundary point.

[0161] Since the chain code value reflects the direction of the corresponding boundary point, the greater the difference in direction between adjacent boundary points in the edge line, the rougher and more irregular the outline of the corresponding edge line is. This indicates that the probability of the corresponding edge line corresponding to the edge of tissue infiltrated by tumor cells is higher, which in turn reflects the higher degree of lymphoma lesion.

[0162] Specifically, the aforementioned organizational abnormality index is the average value of the boundary distortion values ​​of multiple edge lines, and the boundary distortion value of the edge line is the normalized value of the product of the boundary fuzziness index and the boundary morphology abnormality index of the edge line.

[0163] For example, organizational anomaly index It can be represented as:

[0164]

[0165] in, Indicates the number of edge lines. Indicates the first The boundary fuzziness index of the edge line, Indicates the first Boundary morphology anomaly index of each edge line This represents the normalization function.

[0166] Step S4: Determine the slice abnormality index based on the cell abnormality index and the tissue abnormality index, and adjust the initial segment size based on the slice abnormality index to obtain the target segment size.

[0167] In this invention, the slice abnormality index is specifically the normalized value of the average of the cell abnormality index and the tissue abnormality index.

[0168] For example, slice anomaly index It can be represented as:

[0169]

[0170] in, Represents the hyperbolic tangent function. Indicates an organizational abnormality index. This indicates a cell abnormality index.

[0171] The higher the slice abnormality index, the higher the degree of lymphoma lesion, and the smaller the slice size needs to retain more image features for use. Therefore, in this invention, the slice abnormality index and the target slice size are negatively correlated.

[0172] In one embodiment, adjusting the initial block size based on the slice anomaly index to obtain the target block size includes:

[0173] Based on the slice anomaly index and the initial correction coefficient, a target correction coefficient is obtained, wherein the slice anomaly index and the target correction coefficient are negatively correlated.

[0174] The target block size is determined by multiplying the target correction coefficient by the initial block size.

[0175] For example, target block size It can be represented as:

[0176]

[0177] Where 240 represents the initial block size, and 0.3 represents the initial correction factor. Indicates the slice abnormality index. This represents the floor function. This represents the target correction factor.

[0178] Step S5: Using the target block size as the block size, and using a two-stream convolutional network to fuse the first image and the second image to obtain a fused image.

[0179] In the application, multiple dual-stream convolutional networks with different block sizes can be selected in advance. After the target block size is determined, the dual-stream convolutional network that matches the target block size (meaning that the block size used during network training is the target block size) is selected from the multiple pre-configured dual-stream convolutional networks to complete the image fusion operation, so as to ensure the accuracy of the image fusion result.

[0180] It should be understood that the aforementioned fused images not only include information such as cell morphology and tissue structure provided by the first image, but also molecular genetic information on gene mutations provided by the second image. This allows clinicians to better observe the cell morphology, tissue structure, and gene expression of the target pathological sample through the fused images, thereby better completing the subtype classification of lymphoma and laying a solid foundation for the subsequent treatment plan.

[0181] In summary, this invention effectively assesses the degree of lymphoma lesions in the first image at a fine-grained level by analyzing the degree to which each cell differs from a normal cell. Conversely, by analyzing the degree to which the boundaries of each tissue in the first image differ from normal tissue boundaries, it effectively assesses the degree of lymphoma lesions at a coarse-grained level. Combining these two methods allows for an accurate assessment of the overall degree of lymphoma lesions presented in the first image. Based on this, the initial block size is dynamically adjusted, and the adjusted block size is used for image fusion, ensuring that the block size is well-suited to the lymphoma lesions shown in the corresponding pathological sections. The algorithm adjusts the size of the image segments used to improve the image fusion effect. When the lymphoma lesion is mild, the segment size is increased appropriately to avoid over-processing of image details during fusion while ensuring that the network's cross-modal attention can comprehensively associate image information between different modalities. When the lymphoma lesion is severe, the segment size is decreased appropriately to ensure that the network's cross-modal attention accurately associates image information between different modalities while preserving key image details during fusion. This improves the image fusion effect of FISH and bright-field images of lymphoma pathology sections, providing more effective data support for subsequent diagnosis and treatment by medical personnel.

[0182] In one embodiment, the present invention also provides an information fusion system 200 for fluorescence images and pathological sections of lymphoma, such as... Figure 2 As shown, the system 200 includes:

[0183] Image acquisition module 201 is used to acquire a first image and a second image of a target pathological sample, wherein the target pathological sample is a lymphoma tissue sample, the first image is a bright field image acquired based on the target pathological sample, and the second image is a FISH image acquired based on the target pathological sample;

[0184] The cell analysis module 202 is used to analyze the degree of difference between each cell and normal cells in the first image to obtain a cell abnormality index;

[0185] The tissue analysis module 203 is used to analyze the degree of difference between the boundaries of each tissue and the boundaries of normal tissue in the first image to obtain a tissue abnormality index;

[0186] The block size determination module 204 is used to determine the slice abnormality index based on the cell abnormality index and the tissue abnormality index, and adjust the initial block size based on the slice abnormality index to obtain the target block size;

[0187] The image fusion module 205 is used to fuse the first image and the second image using the target block size as the block size and a two-stream convolutional network to obtain a fused image.

[0188] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the information fusion system for fluorescence images and pathological sections for lymphoma and the information fusion method for fluorescence images and pathological sections for lymphoma provided in the above embodiments belong to the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0189] This invention also provides an electronic device. Please refer to [link to relevant documentation]. Figure 3 The electronic device may include a processor 301, a memory 302, and a program 3021 stored in the memory 302 and capable of running on the processor 301.

[0190] When program 3021 is executed by processor 301, it can achieve the following: Figure 1 Any steps in the corresponding method embodiments and the achievement of the same beneficial effects will not be repeated here.

[0191] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0192] This invention also provides a readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described functions. Figure 1 Any step in the corresponding method embodiment can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0193] The computer-readable storage medium of this invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0194] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0195] The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0196] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or terminal. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0197] This invention also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to achieve the information fusion method for fluorescence images and pathological sections of lymphoma provided in the above embodiments.

[0198] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0199] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for information fusion of fluorescent images and pathological sections for lymphoma, characterized in that, The method comprises: acquiring a first image and a second image of a target pathological sample, wherein the target pathological sample is a lymphoma tissue sample, the first image is a bright field image collected based on the target pathological sample, and the second image is a FISH image collected based on the target pathological sample; in the first image, analyzing a difference degree of each cell from a normal cell to obtain a cell abnormality index; in the first image, analyzing a difference degree of each tissue boundary from a normal tissue boundary to obtain a tissue abnormality index; determining a slice abnormality index according to the cell abnormality index and the tissue abnormality index, and adjusting an initial block size based on the slice abnormality index to obtain a target block size; using the target block size as a block size, and using a double-flow convolutional network to fuse the first image and the second image to obtain a fused image.

2. The method of fusing information of fluorescent images and pathological sections for lymphoma according to claim 1, wherein, The method comprises: in the first image, analyzing a pixel confusion degree of each nucleus to obtain a nuclear staining entropy value; in the first image, analyzing a morphological distortion degree of each nucleus to obtain a nuclear deformation value; in the first image, analyzing a size difference degree of each cell to obtain a nuclear size fluctuation value; obtaining the cell abnormality index according to the nuclear staining entropy value, the nuclear deformation value, and the nuclear size fluctuation value.

3. The method of fusing information of fluorescent images and pathological sections for lymphoma according to claim 2, wherein, The method comprises: performing gray processing on the first image to obtain a gray image; identifying a connected region corresponding to a nucleus in the gray image to obtain a plurality of nuclear connected domains; in the plurality of nuclear connected domains, analyzing a gray difference of a pixel point located in a nuclear connected domain to obtain a nuclear staining entropy value.

4. The method of fusing information of a lymphoma-oriented fluorescent image and a pathological section according to claim 3, characterized in that, The method comprises: based on the plurality of nuclear connected domains, performing binaryzation processing on the gray image to obtain a binaryzation image, wherein the binaryzation image is used to distinguish a pixel point located in a nuclear connected domain from a pixel point located outside the nuclear connected domain; performing closed operation processing on the binaryzation image to obtain a shape repair image; analyzing a difference between a pixel point distribution of the shape repair image and a pixel point distribution of the binaryzation image to obtain a nuclear deformation value.

5. The method of claim 3, wherein the method is a method of information fusion of fluorescent images and pathological sections for lymphoma, characterized by, The method comprises: in the plurality of nuclear connected domains, acquiring an area of each nuclear connected domain and an area of a cell connected domain corresponding to each nuclear connected domain to obtain a plurality of nuclear domain areas and a plurality of cell domain areas; analyzing a discrete degree of the plurality of nuclear domain areas to obtain a nuclear domain area discrete value; analyzing a discrete degree of the plurality of cell domain areas to obtain a cell domain area discrete value; obtaining a nuclear size fluctuation value according to the nuclear domain area discrete value and the cell domain area discrete value.

6. The method of fusing information of fluorescent images and pathological sections for lymphoma according to claim 1, wherein, The method comprises: performing edge detection on the first image to obtain a plurality of edge lines and a gradient amplitude of each pixel point in the first image; determining a boundary blur index of each edge line by analyzing a gradient amplitude difference between a boundary point on each edge line and its adjacent pixel points and a length of each edge line in the plurality of edge lines; determining a boundary abnormality index of each edge line by analyzing a boundary shape abnormality degree of each edge line in the plurality of edge lines; determining a boundary distortion value of each edge line according to the boundary blur index and the boundary abnormality index of each edge line; obtaining a tissue abnormality index according to the boundary distortion values of the plurality of edge lines.

7. The method of fusing information from fluorescent images and pathological sections for lymphoma according to claim 6, wherein, The determining of the boundary blur index of each edge line by analyzing the gradient amplitude difference between the boundary point on each edge line and its adjacent pixel points and the length of each edge line in the plurality of edge lines comprises: determining a point blur index of each boundary point on each edge line by analyzing the gradient amplitude difference between the boundary point on each edge line and its adjacent pixel points in the plurality of edge lines; determining, among all the boundary points included in the plurality of edge lines, a boundary point with a boundary blur index greater than a blur threshold value as a blurred boundary point corresponding to the edge line; determining the boundary blur index of each edge line according to the length of each edge line, a total number of the blurred boundary points included in each edge line, and an average value of the point blur indexes corresponding to all the blurred boundary points included in each edge line in the plurality of edge lines. The determining of the point blur index of each boundary point on each edge line by analyzing the gradient amplitude difference between the boundary point on each edge line and its adjacent pixel points in the plurality of edge lines comprises: determining a plurality of target line segments based on all the boundary points included in the plurality of edge lines, wherein a center point of the target line segment is a boundary point included in one of the edge lines, a normal of the edge line where the center point of the target line segment is located is an extension direction of the target line segment, and a length of the target line segment is less than or equal to a length threshold value; calculating an average value of the gradient amplitudes of a plurality of non-center pixel points of each target line segment to obtain a plurality of first gradient mutation values in the plurality of target line segments; analyzing a difference between a maximum non-center gradient amplitude and a gradient amplitude of a center point of each target line segment to obtain a plurality of second gradient mutation values, wherein the maximum non-center gradient amplitude is a maximum gradient amplitude among the gradient amplitudes of the non-center pixel points of the corresponding target line segment; obtaining a plurality of point blur indexes corresponding to the plurality of target line segments according to the plurality of first gradient mutation values and the plurality of second gradient mutation values.

8. The method of fusing information of fluorescent images and pathological sections for lymphoma according to claim 6, wherein, The determining of the boundary abnormality index of each edge line by analyzing the boundary shape abnormality degree of each edge line in the plurality of edge lines comprises: analyzing a chain code value difference between adjacent boundary points in each edge line to obtain a plurality of chain code difference values of each edge line in the plurality of edge lines; calculating an average value of the plurality of chain code difference values of each edge line to obtain the boundary abnormality index of each edge line.

9. The lymphoma-oriented information fusion method of fluorescence images and pathological sections according to claim 1, characterized in that, Adjusting the initial block size based on the slice anomaly index to obtain a target block size, comprising: According to the slice anomaly index and an initial correction coefficient, a target correction coefficient is obtained, wherein the slice anomaly index and the target correction coefficient are in a negative correlation relationship; The product of the target correction coefficient and the initial block size is determined as the target block size.

Images