Tissue status analysis system

By segmenting and feature extraction of slice images, hypergraphs are constructed and multi-scale fusion is carried out, combining hypergraph learning network and self-attention mechanism, the problem of low auxiliary prediction accuracy in traditional technology is solved, and the risk of liver cancer recurrence is achieved More accurate assessment and analysis of the overall status of the organization.

CN118644477BActive Publication Date: 2024-11-12ZHEJIANG LAB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411104179.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-12
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

Traditional tissue status analysis technology has low accuracy in predicting liver cancer recurrence risk risk, and is limited to the relationship between paired cells or tissues, so the overall situation of the tissue cannot be considered.

Method used

The image segmentation module is used to segment the sliced ​​image into small pieces. The image and structural features are acquired through the feature extraction module, hypergraphs are constructed and multi-scale fusion is carried out. Combined with the hypergraph learning network and self-attention mechanism, multi-scale fusion features are generated for the Organizational status analysis.

Benefits of technology

It improves the accuracy of auxiliary prediction of liver cancer recurrence risk, can more comprehensively evaluate tissue status, overcome the limitations of traditional technology, and achieve a more accurate analysis of the overall situation of the tissue.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118644477B_ABST
    Figure CN118644477B_ABST
Patent Text Reader

Abstract

The present application relates to a tissue state analysis system. The tissue state analysis system includes an image segmentation module, which is used to perform image segmentation on the first slice image and the second slice image in a preset area to obtain multiple first small block images and multiple second small block images respectively; a feature extraction module, which is used to extract image features from multiple first small block images respectively to obtain image features; and to extract structural features from multiple second small block images respectively to obtain structural features; a hypergraph construction module, which is used to obtain the first hypergraph and the second hypergraph based on the image features and the structural features respectively; a feature fusion module, which is used to perform multi-scale fusion on the first hypergraph and the second hypergraph to obtain multi-scale fusion features; and a result prediction module, which is used to obtain the tissue state analysis result based on the multi-scale fusion features. The use of this method can improve the accuracy of auxiliary prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of tissue analysis, and in particular to a tissue status analysis system. Background Art

[0002] As a common malignant tumor, liver cancer has a high recurrence rate, which has a serious impact on the survival and life of patients. Therefore, the prediction of liver cancer recurrence is one of the important topics in clinical medicine.

[0003] In traditional technologies, machine learning, deep learning, and graph learning are often used to analyze tissue status based on tissue images. For example, the risk of liver cancer recurrence is predicted by using algorithms such as logistic regression, support vector machine, and random forest, combined with clinical data, biomarkers, or imaging features to assist in predicting the risk of liver cancer recurrence.

[0004] However, the above traditional technologies require operators to manually complete feature engineering and use an interpretable, weakly supervised deep learning framework, which is limited to the relationship between paired cells or tissues and cannot take into account the overall situation of the tissue. Therefore, the auxiliary prediction accuracy of recurrence risk is low.

[0005] This shows that traditional tissue status analysis technology still has the problem of low auxiliary prediction accuracy. Summary of the invention

[0006] Based on this, it is necessary to provide a tissue state analysis system that can improve the accuracy of auxiliary prediction in response to the above technical problems.

[0007] This embodiment provides a tissue state analysis system, the tissue state analysis system comprising:

[0008] An image segmentation module, used to perform image segmentation on the first slice image and the second slice image in a preset area to obtain a plurality of first small block images and a plurality of second small block images respectively;

[0009] A feature extraction module, configured to extract image features from the plurality of first small-block images respectively to obtain image features; and to extract structural features from the plurality of second small-block images respectively to obtain structural features;

[0010] A hypergraph construction module, used to obtain a first hypergraph and a second hypergraph based on the image features and the structural features respectively;

[0011] A feature fusion module, used for performing multi-scale fusion on the first hypergraph and the second hypergraph to obtain a multi-scale fusion feature;

[0012] The result prediction module is used to obtain the tissue state analysis result based on the multi-scale fusion features.

[0013] In one embodiment, the tissue state analysis system further includes an image acquisition module, which is used to:

[0014] Acquire images of the target tissue slice at a first magnification and a second magnification to obtain a first slice image and a second slice image;

[0015] The first slice image and the second slice image are respectively subjected to standardization processing to obtain the first slice image and the second slice image after the standardization processing.

[0016] In one embodiment, the feature extraction module includes an image feature extraction module and a structural feature extraction module, wherein:

[0017] An image feature extraction module, used for inputting the plurality of first small blocks of images into a pre-trained ResNet18 network to obtain image features;

[0018] The structural feature extraction module is used to calculate the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block image respectively; based on multiple second small blocks of images, and the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block image, obtain the structural features.

[0019] In one embodiment, the structural feature extraction module is further used for:

[0020] The cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information of each second small block of image are respectively encoded to obtain encoding information of the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block of image;

[0021] A structural feature is obtained based on the plurality of second small-block images and the encoding information.

[0022] In one of the embodiments, the structural feature extraction module includes a cell nucleus recognition module, a shape factor calculation module and a node degree information acquisition module;

[0023] The cell nucleus recognition module is used to recognize the cell nuclei in each second small block of images; based on the recognition result of the cell nuclei, determine the cell nucleus density and cell nucleus size corresponding to each second small block of images;

[0024] The shape factor calculation module is used to analyze the shape pattern of each cell nucleus in the single second small block image, determine the abnormality index of each cell nucleus based on the shape pattern; and determine the cell nucleus shape factor of the single second small block image based on the abnormality indexes of multiple cell nuclei;

[0025] The node degree information acquisition module is used to acquire degree information of multiple nodes; each node corresponds to a single second small block image.

[0026] In one of the embodiments, the hypergraph construction module is further used to: cluster the image features and the structural features based on a clustering algorithm, respectively, to obtain a first hypergraph and a second hypergraph.

[0027] In one embodiment, the feature fusion module includes a hypergraph convolution layer, a splicing layer, a self-attention module, a hypergraph learning network and a fusion layer, wherein:

[0028] The hypergraph convolution layer is used to transfer information to the nodes in the first hypergraph to obtain the first initial features after the nodes are updated; transfer information to the nodes in the second hypergraph to obtain the second initial features after the nodes are updated;

[0029] The splicing layer is used to splice the nodes in the first initial feature and the second initial feature to obtain a splicing feature;

[0030] The self-attention module is used to update the weights of multiple nodes in the splicing feature based on the dependency relationship between the nodes in the splicing feature to obtain a self-attention adjustment feature;

[0031] The hypergraph learning network is used to perform node update on the self-attention adjustment feature to obtain a node update feature;

[0032] The fusion layer is used to perform node fusion on the node update features to obtain multi-scale fusion features.

[0033] In one embodiment, the hypergraph learning network includes a plurality of process convolution modules, wherein:

[0034] The process convolution module includes a first preset number of downsampling layers and a second preset number of hypergraph convolution blocks, and the process convolution module is used to perform node update on the splicing feature to obtain a node update feature; the hypergraph convolution block includes a clustering algorithm layer, a hypergraph convolution layer and a convolution layer;

[0035] The clustering algorithm layer is used to construct a hyperedge for the self-attention adjustment feature based on a clustering algorithm;

[0036] The hypergraph convolution layer is used to transfer information to nodes in the self-attention adjustment feature;

[0037] The convolution layer is used to convolve the nodes in the self-attention adjustment feature.

[0038] In one embodiment, the feature fusion module includes a first hypergraph learning network and a second hypergraph learning network, wherein:

[0039] The first hypergraph learning network is used to perform node update on the self-attention adjustment feature based on the association matrix corresponding to the first hypergraph to obtain a first node update feature;

[0040] The second hypergraph learning network is used to perform node update on the self-attention adjustment feature based on the association matrix corresponding to the second hypergraph to obtain a second node update feature.

[0041] In one of the embodiments, the result prediction module includes an adaptive average pooling layer and a fully connected layer;

[0042] The adaptive average pooling layer is used to adjust the size of the multi-scale fusion feature to obtain the multi-scale fusion feature after the size adjustment;

[0043] The fully connected layer is used to obtain a prediction result of disease recurrence based on the multi-scale fusion features after size adjustment.

[0044] The above-mentioned tissue state analysis system performs image segmentation on the first slice image and the second slice image in the preset area through the image segmentation module to obtain multiple first small block images and multiple second small block images respectively; the feature extraction module performs image feature extraction on the multiple first small block images respectively to obtain image features; the structural feature extraction on the multiple second small block images respectively to obtain structural features; the hypergraph construction module obtains the first hypergraph and the second hypergraph based on the image features and the structural features respectively; the feature fusion module performs multi-scale fusion on the first hypergraph and the second hypergraph to obtain multi-scale fusion features; the result prediction module obtains the tissue state analysis result based on the multi-scale fusion features, introduces the construction of the hypergraph for the slice image, so as to learn more complex pathological images with high-order modeling capabilities, and realizes the combined analysis and judgment of the spatial relationship between the small block images in the slice image while learning the local features, which can achieve the effect of improving the accuracy of auxiliary prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is an application environment diagram of an organization status analysis system in one embodiment;

[0046] Figure 2 is a structural block diagram of an organization status analysis system in one embodiment;

[0047] Figure 3 A schematic diagram of a flow chart of an organization status analysis system in one embodiment;

[0048] Figure 4A flowchart of an organization status analysis system in one embodiment;

[0049] Figure 5 Schematic diagram of the principle of hyperedge convolution in one embodiment;

[0050] Figure 6 is a structural block diagram of a process convolution module in one embodiment;

[0051] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The tissue state analysis system provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 or the image acquisition device through the network to obtain the first slice image and the second slice image. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 includes a memory, in which an operating system and a computer program are stored. The computer program includes a tissue state analysis system, and the tissue state analysis system includes: an image segmentation module, which is used to perform image segmentation on the first slice image and the second slice image in a preset area, and obtain a plurality of first small pieces of images and a plurality of second small pieces of images respectively; a feature extraction module, which is used to extract image features from the plurality of first small pieces of images respectively to obtain image features; and extract structural features from the plurality of second small pieces of images respectively to obtain structural features; a hypergraph construction module, which is used to obtain a first hypergraph and a second hypergraph based on the image features and the structural features respectively; a feature fusion module, which is used to perform multi-scale fusion on the first hypergraph and the second hypergraph to obtain multi-scale fusion features; and a result prediction module, which is used to obtain a tissue state analysis result based on the multi-scale fusion features. The terminal 102 may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The portable wearable device may be, for example, a head-mounted device, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0054] In one embodiment, Figure 2 As shown, this embodiment provides a tissue state analysis system, which is applied to Figure 1Taking the terminal 102 in the example as an example, the tissue state analysis system includes an image segmentation module 100, a feature extraction module 200, a hypergraph construction module 300, a feature fusion module 400 and a result prediction module 500:

[0055] The image segmentation module 100 is used to perform image segmentation on the first slice image and the second slice image in a preset area to obtain a plurality of first small block images and a plurality of second small block images respectively.

[0056] The first slice image and the second slice image of the preset area may be images obtained by scanning the target tissue slice through an image acquisition device. The first slice image and the second slice image may be images of the tissue slice at different magnifications. The image segmentation module 100 may obtain multiple small block images according to a preset small block image size, or a preset row segmentation and column segmentation rule, or other preset image segmentation rules, which are not limited in this embodiment.

[0057] Different segmentation rules may be used to segment the first small image and the second small image. For example, when the size of the image to be segmented is small, the image may be segmented into a plurality of small images with a smaller number and a larger size of each image. When the size of the image to be segmented is large, the image may be segmented into a plurality of small images with a larger number and a smaller size of each image.

[0058] By segmenting the image, more complex tissue structures can be decomposed into small, manageable images to further improve the accuracy of subsequent feature extraction and tissue analysis.

[0059] The feature extraction module 200 is used to extract image features from the plurality of first small-block images respectively to obtain image features; and to extract structural features from the plurality of second small-block images respectively to obtain structural features.

[0060] Among them, the feature extraction module 200 can use a deep learning network to extract features from multiple first small blocks of images to obtain image features.

[0061] For extracting structural features from multiple second small-block images respectively to obtain structural features, a preset structural feature extraction algorithm may be set according to the characteristics of the tissue structure, so as to perform targeted structural feature extraction. For example, in tissue state analysis, structural features may include cell features, cell nucleus features, cell arrangement tissue pattern, blood vessel features, matrix components, extracellular environment features, macroscopic structural features, etc. Further, structural feature extraction may be obtained by combining an image recognition algorithm and performing statistical calculations on the target structure.

[0062] By processing the first small block of images and the second small block of images by the feature extraction module 200, automated feature extraction can be achieved, thereby avoiding the subjectivity of manual feature engineering, ensuring the comprehensiveness and objectivity of the features themselves, and helping to improve the prediction capability of the system.

[0063] The hypergraph construction module 300 is used to obtain a first hypergraph and a second hypergraph based on image features and structural features respectively.

[0064] Among them, a hypergraph is a graph in a broad sense. In visual representation, it is usually drawn with multiple points, namely vertices, and shapes covering these points, namely hyperedges. Each hyperedge can connect any number of vertices to represent complex many-to-many relationships.

[0065] The hypergraph construction module 300 obtains the first hypergraph and the second hypergraph based on the image features and the structural features respectively. The hypergraph can be constructed based on the image features to obtain the first hypergraph, and the hypergraph can be constructed based on the structural features to obtain the second hypergraph.

[0066] The construction of a hypergraph can be done through distance-based methods such as nearest neighbor search and clustering algorithms.

[0067] Through the processing of the hypergraph construction module 300, more complex relationships than traditional graph models can be represented, including many-to-many relationships and high-order interactions, so as to overcome the limitation of traditional technology that only focuses on the relationship between paired cell nuclei and tissues, thereby improving the accuracy of understanding the overall state of the tissue in the subsequent tissue state analysis process.

[0068] The feature fusion module 400 is used to perform multi-scale fusion on the first hypergraph and the second hypergraph to obtain multi-scale fusion features.

[0069] Among them, the feature fusion module 400 integrates the feature information of different levels and scales of the first hypergraph and the second hypergraph through a preset multi-scale fusion strategy to obtain a multi-scale fusion feature containing multi-level information. Through multi-scale fusion, feature details at different scales can be considered, thereby enhancing the system's understanding of local and global features, and improving the comprehensiveness and accuracy of prediction.

[0070] The result prediction module 500 is used to obtain the tissue state analysis result based on the multi-scale fusion features.

[0071] The result prediction module 500 can be implemented by a fully connected layer, so as to perform tissue state analysis on the multi-scale fusion features to determine the tissue state analysis results. For example, when the tissue state analysis system is applied to auxiliary prediction of liver cancer recurrence risk, auxiliary prediction results of liver cancer recurrence risk can be obtained based on multi-scale fusion features.

[0072] The present embodiment provides a tissue state analysis system, which performs image segmentation on a first slice image and a second slice image in a preset area through an image segmentation module to obtain a plurality of first small block images and a plurality of second small block images respectively; the feature extraction module performs image feature extraction on the plurality of first small block images respectively to obtain image features; the structural feature extraction on the plurality of second small block images respectively to obtain structural features; the hypergraph construction module obtains a first hypergraph and a second hypergraph based on image features and structural features respectively; the feature fusion module performs multi-scale fusion on the first hypergraph and the second hypergraph to obtain multi-scale fusion features; the result prediction module obtains a tissue state analysis result based on the multi-scale fusion features, and introduces the construction of a hypergraph for slice images, so as to learn more complex pathological images with high-order modeling capabilities, and realizes the combined analysis and judgment of the spatial relationship between small block images in the slice image while learning local features, which can achieve the effect of improving the accuracy of auxiliary prediction.

[0073] In one embodiment, the tissue state analysis system further includes an image acquisition module, which is used to:

[0074] Acquire images of the target tissue slice at a first magnification and a second magnification to obtain a first slice image and a second slice image;

[0075] The first slice image and the second slice image are respectively subjected to standardization processing to obtain the first slice image and the second slice image after the standardization processing.

[0076] Among them, the image acquisition device can be an image acquisition device including a microscope, such as a digital slice scanner, to achieve image acquisition of the target tissue slice at a microscopic perspective. The first slice image and the second slice image can be obtained by acquiring images of the target tissue slice at a first magnification and a second magnification, respectively. It can be understood that the low-magnification image provides the approximate layout and structure of the tissue, while the high-magnification image provides detailed information at the cellular level. This multi-scale imaging strategy helps to capture tissue features from macro to micro.

[0077] In a specific embodiment, taking liver cancer recurrence prediction as an example, a target tissue slice may be obtained from a pathological specimen and fixed on a glass slide, and the target tissue slice may be stained with hematoxylin-eosin. A high-resolution slice scan may be performed on the target tissue slice using a digital slice scanner, and images of multiple fields of view may be collected on the target tissue slice to obtain a full slice image of the target tissue slice. The full slice image may be processed at a first magnification and a second magnification to obtain a first slice image and a second slice image.

[0078] In the long run, the imaging conditions of the first slice image and the second slice image acquired at different times may be different, such as lighting, equipment parameters, staining differences, etc. Therefore, the first slice image and the second slice image can be standardized to reduce the differences in the slice images. The standardization step can be to use brightness, contrast adjustment, color correction, size and resolution adjustment, background removal and other methods to achieve standardized processing of the slice image, thereby obtaining the first slice image and the second slice image after standardized processing.

[0079] The present embodiment provides a tissue state analysis system, which, through standardization, can more accurately identify and compare tissue features in different slices or at different magnifications, thereby improving the accuracy and reliability of tissue state analysis, laying the foundation for subsequent structural feature and image feature extraction, and enabling the tissue state analysis system to more carefully and comprehensively evaluate the state of the tissue, thereby achieving the effect of improving the accuracy of auxiliary prediction.

[0080] In one embodiment, the feature extraction module includes an image feature extraction module and a structural feature extraction module, wherein:

[0081] An image feature extraction module, used for inputting a plurality of first small blocks of images into a pre-trained ResNet18 network to obtain image features;

[0082] The structural feature extraction module is used to calculate the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block image respectively; based on multiple second small blocks of images, and the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block of images, the structural features are obtained.

[0083] Among them, ResNet18 is a lightweight deep residual network. ResNet18 can extract common image features through training on large-scale image data sets. The obtained image features can contain key visual information of the image, such as texture, color and shape, which can be used to identify tissue types, lesion areas or cell morphology changes.

[0084] The structural feature extraction module is used to calculate the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information, wherein the cell nucleus density can be obtained by calculating the number of cell nuclei in the second small image and the size of the space occupied by the second small image. The cell nucleus size can be measured by the mapping relationship between the pixels in the image and the actual space size, and the area or diameter of the cell nucleus. The cell nucleus shape factor can be used to calculate whether the shape of the cell nucleus is abnormal, for example, the circularity is calculated by the ratio of the square of the perimeter to the area, as a quantitative indicator of the shape. The node degree can be the number of edges connecting a node to other nodes when constructing a hypergraph, so the node degree information can reflect the interaction strength between the local tissue and the surrounding environment.

[0085] Furthermore, the structural feature extraction module can also perform analysis and calculation based on the features of multiple second small blocks of images. For example, the average value, standard deviation, etc. of the cell nucleus density of multiple second small blocks of images can be calculated as part of the structural feature.

[0086] The present embodiment provides a tissue state analysis system, which combines these two types of features so that the tissue state analysis system can more comprehensively evaluate tissue sections, thereby achieving the effect of improving the accuracy of auxiliary prediction.

[0087] In one embodiment, the structural feature extraction module is further used to:

[0088] The cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information of each second small block of image are respectively encoded to obtain encoding information of the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block of image;

[0089] A structural feature is obtained based on the multiple second small block images and the encoding information.

[0090] The encoding process may be to divide the above structural features into several intervals, thereby converting from numerical values ​​to categories. For example, the continuous cell nucleus size or density value may be divided into several intervals, each interval corresponding to a code, so that it can be converted into discrete categories; for another example, a neural network or an autoencoder may be used to map the cell nucleus shape factor or node degree information to a low-dimensional space, thereby obtaining the corresponding encoding information.

[0091] In a specific embodiment, the cell nucleus density range can be set to [0, 1], and 5 thresholds are set, namely 0, 0.25, 0.50, 0.75 and 1, corresponding to 5 density intervals respectively, and continuous density values ​​are mapped to corresponding density areas. When the cell nucleus density of a single small block image is 0, it is encoded as 10000, when the cell nucleus density of a single small block image is in (0, 0.25], it is encoded as 01000, and when the cell nucleus density is in (0.75, 1], it is encoded as 00001.

[0092] A tissue state analysis system provided in this embodiment encodes the cell nuclear density, cell nuclear size, cell nuclear shape factor and node degree information so that the encoded cell nuclear density, cell nuclear size, cell nuclear shape factor and node degree information can more concisely express the key characteristics of the original data, thereby further improving the accuracy of the tissue state analysis results.

[0093] In one of the embodiments, the structural feature extraction module includes a cell nucleus recognition module, a shape factor calculation module and a node degree information acquisition module;

[0094] The cell nucleus recognition module is used to recognize the cell nucleus in each second small block of image; based on the recognition result of the cell nucleus, determine the cell nucleus density and cell nucleus size corresponding to each second small block of image;

[0095] The shape factor calculation module is used to analyze the shape pattern of each cell nucleus in the single second small block image, determine the abnormal index of each cell nucleus based on the shape pattern; and determine the cell nucleus shape factor of the single second small block image based on the abnormal indexes of multiple cell nuclei;

[0096] The node degree information acquisition module is used to acquire degree information of multiple nodes; each node corresponds to a single second small block image.

[0097] Among them, the cell nucleus recognition module can be obtained by segmentation using a deep learning model, for example, by locating and outlining the cell nucleus through networks such as U-Net and Mask R-CNN, thereby obtaining multiple cell nuclei in a single second small block image.

[0098] Based on the recognition result of the cell nucleus, the cell nucleus density and the cell nucleus size corresponding to each second small block of image are determined. The number of cell nuclei in the second small block of image can be counted. The cell nucleus size can be counted by the number of pixels of a single cell nucleus in the image, and the size of the cell nucleus is determined based on the mapping relationship between the preset number of pixels and the actual size.

[0099] The shape factor calculation module may obtain the shape pattern of the cell in the simplified shape space of principal component analysis through a clustering algorithm. Exemplarily, the shape pattern may be identified through unsupervised K-means cluster analysis, each cluster represents a shape pattern, and each cluster is marked according to the degree of abnormality of the cell nucleus shape.

[0100] Determining the abnormality index of each cell nucleus based on the shape pattern can be to determine the abundance and distribution of cells in each shape pattern. The abundance and distribution can be achieved by counting the number of cells and the position of cells in each cluster. Then, the shape abnormality index is assigned to the cell nucleus of each small image block, and all the abnormality indices in a single second small image block are added and averaged. The obtained value is rounded and quantified, so as to obtain the cell nucleus shape factor of the single second small image block.

[0101] In this embodiment, a node may represent a cell nucleus or a tissue region. The node degree information acquisition module acquires the degree information of each node, that is, the number of edges connected to the node. The node degree information helps to maintain the consistency of the local context.

[0102] The present embodiment provides a tissue state analysis system, which provides basic cell nucleus position and size information through a cell nucleus recognition module, while a shape factor calculation module deeply analyzes the morphological characteristics of the cell nucleus, and a node degree information acquisition module provides a perspective on the interaction between cell nuclei from the perspective of network topology. This can form a comprehensive description of the tissue structure state, and provide rich structural feature input for subsequent hypergraph construction and feature fusion, thereby achieving the effect of improving the accuracy of auxiliary prediction.

[0103] In one of the embodiments, the hypergraph construction module is further used to: cluster the image features and the structural features respectively based on a clustering algorithm to obtain a first hypergraph and a second hypergraph.

[0104] Among them, for the construction of the first hypergraph, that is, the construction of the hypergraph for image features, and for the construction of the second hypergraph, that is, the construction of the hypergraph for structural features, algorithms such as K-Means, spectral clustering, DBSCAN, etc. can be used to group features based on the similarity of feature vectors, and each group of features represents an image pattern or type.

[0105] The present embodiment provides an organizational status analysis system. In the first hypergraph and the second hypergraph obtained by clustering, each hyperedge can represent a similarity, structural pattern or type, which is used to indicate the correlation between vertices, thereby helping the subsequent feature fusion module to more effectively integrate multi-source information, improve the accuracy of analysis and prediction performance, and achieve the effect of improving the accuracy of auxiliary prediction.

[0106] In one embodiment, the feature fusion module includes a hypergraph convolution layer, a concatenation layer, a self-attention module, a hypergraph learning network and a fusion layer, wherein:

[0107] The hypergraph convolution layer is used to transfer information to the nodes in the first hypergraph to obtain the first initial features after the nodes are updated; and transfer information to the nodes in the second hypergraph to obtain the second initial features after the nodes are updated;

[0108] A concatenation layer, used for concatenating nodes in the first initial feature and the second initial feature to obtain a concatenated feature;

[0109] A self-attention module is used to update the weights of multiple nodes in the spliced ​​features based on the dependencies between the nodes in the spliced ​​features to obtain self-attention adjustment features;

[0110] A hypergraph learning network is used to perform node updates on the self-attention adjustment features to obtain node update features;

[0111] The fusion layer is used to perform node fusion on the self-attention adjustment features to obtain multi-scale fusion features.

[0112] The hypergraph convolution layer is used to transfer information between nodes in the hypergraph to obtain the initial features after updating the nodes. The information transfer is in the hypergraph convolution layer, where nodes transfer information through hyperedges, where the updated node features are the weighted sum of the node features before the update, and the weights used in the weighted sum can be determined by the properties of the hyperedges, such as the connection strength.

[0113] In a specific embodiment, Figure 3 As shown in the figure, the hypergraph convolution layer realizes information exchange between nodes through a two-stage information transmission method of node-hyperedge-node. During hypergraph convolution, the association matrix H is passed to the continuous convolution layer, propagated between shared hyperedges, and higher confidence is given to larger weights. Based on the hypergraph structure and related weights, the operation formula of hypergraph convolution is as follows:

[0114] X (l) =σ(D v -1 HWD e -1 H T X (l) θ)

[0115] in, θ is the learning parameter of hypergraph convolution, θ ∈ RF(l)×F(l+1) , X (l) ∈ R n×F(l) Indicates l The node embedding of the layer, σ for sigmoid Activation function, H T The multiplication of realizes the information aggregation from nodes to hyperedges. Multiplication by H can be regarded as the information aggregation from hyperedges to nodes.

[0116] The splicing layer splices the nodes in the first initial feature and the second initial feature to achieve feature integration and obtain a richer vector, namely the splicing feature, so that subsequent processing can consider both image and structural information.

[0117] The self-attention module updates the weights of multiple nodes in the spliced ​​features based on the dependencies between the nodes in the spliced ​​features to obtain the self-attention adjustment features. The self-attention mechanism allows the model to dynamically adjust the weights of the nodes according to the dependencies between the nodes, which means that during the fusion process, the features of some nodes may be considered more important than the features of other nodes. Each element can be associated with other elements, rather than just relying on elements in adjacent positions. By calculating the relative importance between elements, the self-attention network adaptively captures the long-range dependencies between elements, can highlight the features most relevant to the prediction task, and suppress irrelevant or redundant information. It is suitable for building hypergraphs with complex relationships, thereby improving the accuracy and efficiency of predictions.

[0118] In a specific embodiment, the self-attention network formula is as follows:

[0119] ,in, Q = Xθ 1 , K = Xθ 2 , V = Xθ 3

[0120] in, d k Indicates the dimension of the vector in_dim, divided by d k It is to prevent QK T The value is too large, resulting in softmax Overflow during calculation. θ 1 , θ 2Represents the learnable parameter matrix of the fully connected layer with in_dim input channels and Q_dim output channels, θ 3 Represents the learnable parameter matrix of the fully connected layer with in_dim input channels and V_dim output channels. The sizes of Q, K and V are in_dim×Q_dim, in_dim×Q_dim and in_dim×V_dim respectively.

[0121] Therefore, when the input feature size is 56×56×128, in_dim is 128, Q_dim is set to 128, and V_dim is set to 64, then the output node feature size is 56×56×64.

[0122] The hypergraph learning network further processes the self-attention adjustment feature, performs node update on the self-attention adjustment feature, and obtains the node update feature. Exemplarily, the hypergraph learning network may further include multiple hypergraph convolutional layers, which perform node update on the self-attention adjustment feature sequentially or separately.

[0123] The fusion layer is used to perform node fusion on the node update features to obtain multi-scale fusion features. A more comprehensive representation, namely, multi-scale fusion features, can be generated based on the complementary information provided by the multi-scale node update features, which can better reflect the inherent complexity and diversity of the data. Exemplarily, the hypergraph learning network can split the self-attention adjustment features and perform different node update processes respectively. The fusion layer performs node fusion based on the obtained multiple features to obtain multi-scale fusion features.

[0124] This embodiment provides a system for analyzing tissue status. A hypergraph convolution layer transmits information to nodes in a hypergraph. A splicing layer splices nodes in a first initial feature and a second initial feature to obtain a spliced ​​feature. Self-attention, a hypergraph learning network, and a fusion layer capture the complex relationship between features. Multi-scale fusion features that can comprehensively reflect the tissue status can be generated, thereby improving the accuracy of auxiliary predictions.

[0125] In one embodiment, the hypergraph learning network includes a plurality of process convolution modules, wherein:

[0126] The process convolution module includes a first preset number of downsampling layers and a second preset number of hypergraph convolution blocks, and the process convolution module is used to perform node update on the splicing features to obtain node update features; the hypergraph convolution block includes a clustering algorithm layer, a hypergraph convolution layer and a convolution layer;

[0127] The clustering algorithm layer is used to construct hyperedges for self-attention adjustment features based on the clustering algorithm;

[0128] Hypergraph convolutional layer, used to transfer information between nodes in the self-attention adjustment feature;

[0129] Convolutional layer, used to convolve nodes in the self-attention adjusted features.

[0130] Among them, the downsampling layer is used for node selection and feature dimension reduction, which can reduce the computational burden in the deep processing of the network while retaining key features.

[0131] The hypergraph convolution block includes at least a clustering algorithm layer, a hypergraph convolution layer and a convolution layer. It can be understood that in the same hypergraph convolution block, the number of clustering algorithm layers, supergraph convolution layers and convolution layers can be one or more. This embodiment does not limit the setting order of the clustering algorithm layer, the supergraph convolution layer and the convolution layer and the connection relationship between them.

[0132] Exemplarily, the clustering algorithm layer may be to construct hyperedges using algorithms such as Fuzzy C-Means, K-means, and DBSCAN, or to group nodes, capture the intrinsic connections between nodes, and obtain high-order relationships between features.

[0133] The hypergraph convolution layer transmits information to the nodes, which may be achieved by adopting the above-mentioned two-stage information transmission method of node-hyperedge-node to exchange information between nodes, which will not be described in detail in this embodiment.

[0134] The convolution layer can further process node features and capture local patterns, thereby enhancing the nonlinear expression ability of features.

[0135] In a specific embodiment, the hypergraph learning network includes four process convolution modules, wherein the four process convolution modules are respectively composed of 0 sampling layers and 2 hypergraph convolution blocks, 1 sampling layer and 2 hypergraph convolution blocks, 1 sampling layer and 18 hypergraph convolution blocks, and 1 sampling layer and 2 hypergraph convolution blocks. Each hypergraph convolution block may include a convolution layer Conv1, a clustering algorithm layer Fuzzy C-Means, a hypergraph convolution layer, a convolution layer Conv2, a concatenation layer, a convolution layer Conv3, an activation function ReLU, a convolution layer Conv3, a batch normalization layer, and a discard layer.

[0136] The present embodiment provides a tissue state analysis system, which performs multi-level node updates on splicing features through multiple process convolution modules, so that the node update features contain multi-level information from local details to global structures, which can provide comprehensive information representation for subsequent result predictions, thereby achieving the effect of improving the accuracy of auxiliary predictions.

[0137] In one embodiment, the feature fusion module includes a first hypergraph learning network and a second hypergraph learning network, wherein:

[0138] A first hypergraph learning network is used to perform node update on the self-attention adjustment feature based on the association matrix corresponding to the first hypergraph to obtain a first node update feature;

[0139] The second hypergraph learning network is used to perform node update on the self-attention adjustment feature based on the association matrix corresponding to the second hypergraph to obtain a second node update feature.

[0140] It can be understood that in the processing of the hypergraph learning network, the hypergraph convolution layer realizes information aggregation from nodes to hyperedges based on the association matrix of the hypergraph. The first hypergraph learning network and the second hypergraph learning network respectively perform corresponding node updates on the self-attention adjustment features based on the association matrix of the first hypergraph and the association matrix of the second hypergraph, and obtain the first node update feature and the second node update feature respectively. Updating the node features based on the association matrix of the first hypergraph corresponds to using an association matrix that pays more attention to the relationship between image features for hypergraph convolution processing, while updating the node features based on the association matrix of the second hypergraph corresponds to using an association matrix that pays more attention to the relationship between structural features for hypergraph convolution processing.

[0141] Correspondingly, the fusion layer can fuse the first node update feature and the second node update feature to obtain a multi-scale fusion feature, thereby combining the complementary information provided by the two hypergraph structures to produce a more comprehensive representation that can better reflect the inherent complexity and diversity of the data.

[0142] The present embodiment provides an organizational status analysis system, which uses two independent hypergraph learning networks to update features based on different association matrices respectively, and then inputs the two output features into a fusion layer for fusion. This system can capture the local and global characteristics of the data and adapt to the diversity and complexity of the data, thereby achieving the effect of improving the accuracy of auxiliary predictions.

[0143] In one of the embodiments, the result prediction module includes an adaptive average pooling layer and a fully connected layer;

[0144] The adaptive average pooling layer is used to adjust the size of the multi-scale fusion features to obtain the multi-scale fusion features after size adjustment;

[0145] The fully connected layer is used to obtain the prediction results of disease recurrence based on the resized multi-scale fusion features.

[0146] Among them, the adaptive average pooling layer resizes the multi-scale fusion features to obtain the adjusted multi-scale fusion features. Through average pooling, the effect of feature compression can be achieved, the most representative information can be extracted, and the feature dimension and computational complexity can be reduced.

[0147] The fully connected layer obtains the prediction result of disease recurrence based on the multi-scale fusion features. For example, the tissue state analysis system adopted in this embodiment can use the sample slice image and the recurrence situation corresponding to the sample as the training input data and output data respectively, so as to realize the prediction of disease recurrence results.

[0148] During the training process, the weights and biases of the fully connected layers can be updated through back-propagation algorithms and optimizers such as SGD, Adam, etc. to minimize the gap between the predicted results and the actual labels.

[0149] The present embodiment provides a tissue state analysis system, which resizes and compresses multi-scale fusion features through an adaptive average pooling layer, and then uses a fully connected layer to map these features to the prediction results of disease recurrence. This can ensure that the model can handle inputs of different sizes, and improve the efficiency and robustness of the model through feature compression and standardization, thereby achieving the effect of improving the accuracy of auxiliary prediction.

[0150] In order to explain the technical solution of the present application in more detail, the present application also provides a detailed embodiment.

[0151] In one embodiment, a tissue state analysis system is provided. Taking the application of the system in liver cancer recurrence detection as an example, the system includes an image segmentation module, a feature extraction module, a hypergraph construction module, a feature fusion module and a result prediction module, wherein the image segmentation module is used to perform image segmentation on a first slice image and a second slice image in a preset area to obtain a plurality of first small block images and a plurality of second small block images respectively; the feature extraction module is used to perform image feature extraction on the plurality of first small block images respectively to obtain image features; the structural feature extraction on the plurality of second small block images respectively to obtain structural features; the hypergraph construction module is used to obtain a first hypergraph and a second hypergraph based on the image features and the structural features respectively; the feature fusion module is used to perform multi-scale fusion on the first hypergraph and the second hypergraph to obtain multi-scale fusion features; the result prediction module is used to obtain a tissue state analysis result based on the multi-scale fusion features.

[0152] like Figure 4 As shown in the figure, before the tissue status analysis system is used, data acquisition and preprocessing are first performed. Data acquisition and preprocessing include: taking tissue slices from pathological blood specimens, fixing the tissue slices on glass slides, and performing hematoxylin-eosin staining, and then performing high-resolution slice scanning on them with a digital slice scanner. Images of multiple fields of view are collected on each tissue slice to facilitate subsequent screening of representative areas.

[0153] The collected images are standardized to reduce the obstacles caused by image staining differences when the model learns key problems. First, images of the full slice of the pathological image at magnifications of 40 and 20 are obtained respectively, and then they are standardized respectively. The standardization step can be implemented by the Vahadane method, using non-negative matrix decomposition to obtain two dye matrices, and then the image to be standardized and the reference image are aligned, and then a new image is synthesized. The reference image is a well-stained image, and the image to be standardized is the input image. The sizes of the two are the same. It has been found through experiments that when the size is small, the pathological image to be stained may have unstained cell nuclei, which is not conducive to the method to decompose and obtain the dye matrix. When the size is large, the standardization speed is slow. In a specific embodiment, the pathological image can be cut into small blocks of 1000×1000 and then standardized, and then the standardized images are spliced ​​back.

[0154] Obtain a normalized image of the slice at a magnification of 20 and scale it to 112×112, i.e., the second slice image P2. Obtain a normalized image of the slice at a magnification of 40 and scale it to 224×224, i.e., the first slice image P1. Recurrence label S 1 ∈{0, 1}, the recurrence-free survival time is T1.

[0155] After data acquisition and preprocessing are completed, the first slice image and the second slice image are segmented by an image segmentation module to obtain a plurality of first small block images and a plurality of second small block images respectively.

[0156] Small-scale images have low resolution and cover a wider range, and are suitable for extracting overall features such as cell nuclear density, size, and shape specificity. Large-scale images contain more details and are suitable for network learning of local features.

[0157] The feature extraction module is used to extract image features from the plurality of first small-block images respectively to obtain image features; and to extract structural features from the plurality of second small-block images respectively to obtain structural features.

[0158] Extract embeddings of multi-scale images. Figure 5As shown, a plurality of first small-block images obtained based on the first slice image P1 are black-box embedded to obtain image features X1, and the image features can be shallow features learned by ResNet_18 pre-trained by ImageNet. Specifically, the first slice image P1 can be divided into continuous 56×56 small-block images. Each small block is regarded as a node vi1 of the hypergraph G1. The embedding of the node greatly affects the learning effect of the hypergraph, and it is crucial to choose a suitable embedding. The inventor found through experimental comparison that compared with the use of a simple multi-layer perceptron, a randomly initialized ResNet_18 and a randomly initialized DenseNet, the pre-trained ResNet_18 can learn more useful knowledge. The first slice image P1 of 224×224×3 is input into the pre-trained ResNet_18, the output 56×56×64 after the first residual module is taken, and the position embedding with a dimension of 56×56×64 and an initial value of 0 is added. Finally, a node embedding of 56×56×64 is obtained.

[0159] A plurality of second small block images obtained based on the second slice image P2 are white-box embedded to obtain a structural feature X2, which includes cell nucleus density, nuclear size, nuclear shape factor and node degree information. It is understandable that cancer cells have a phenomenon of cell aggregation, so there is a greater possibility that cancer cells exist in areas with high cell nucleus density. Enlargement of the nucleus of cancer cells is also common, and obtaining information on the size of the nucleus is conducive to learning about the recurrence of cancer in patients. The probability of cancer cells appearing in the area of ​​heteromorphic nuclear cells is higher. In addition, the node degree information contains the hypergraph structure information after each update.

[0160] In a specific embodiment, the image P2 can be divided into continuous 112×112 small blocks of images, and each small block is regarded as a node vi2 of the hypergraph G2.

[0161] Obtain nuclear density. Use a cell nucleus segmentation model, such as a multi-step cell nucleus segmentation system based on color space guidance for pathological images, to segment the cell nuclei of small blocks of images and calculate the number of cell nuclei. Quantify and encode the cell nucleus numbers of all small blocks of images and finally process them into a 4-bit binary vector with a vector value of 0 or 1.

[0162] Get the nuclear size. Based on the obtained nucleus segmentation results, calculate the number of pixels occupied by each nucleus to represent the area of ​​each nucleus, and then divide the area of ​​all nuclei in the small image by the pixel area of ​​the image, that is, the product of the pixel size, to obtain the density of the nuclei in the small image. At this time, the density range is [0, 1]. Set 5 thresholds of 0, 0.25, 0.50, 0.75 and 1, corresponding to 5 density intervals respectively, and map the continuous density values ​​to the corresponding density areas. When the nucleus density of a single small image is 0, it is encoded as 10000, when the nucleus density of a single small image is in (0, 0.25], it is encoded as 01000, and when the nucleus density is in (0.75, 1], it is encoded as 00001.

[0163] Obtaining the shape factor of the nucleus. It is understandable that tumor cells are prone to rare or uncommon shapes. Using an unsupervised machine learning clustering method, the shape pattern of the cell is obtained in the simplified shape space of principal component analysis. The coordinates of equally spaced points along the outline of the nucleus can be determined and recorded to define the morphological descriptors; then the number of morphological descriptors is reduced by principal component analysis to better understand the variation between cells; then the shape pattern is identified by unsupervised K-means clustering analysis, each cluster represents a shape pattern, and for each cluster, the shape abnormality index Ad is marked according to the degree of abnormality of the nucleus shape, and the value is [1, 2, 3...], the larger the value, the stronger the abnormality; then, the abundance and distribution of cells in each shape pattern are determined, which is achieved by counting the number of cells and the location of cells in each cluster; finally, the shape abnormality index Ad is assigned to the nucleus of each small image, all the shape abnormality indices of the small image are added and averaged, and the obtained value is rounded, quantized, and encoded as a 4-bit binary vector.

[0164] Get node degree information. Get the degree of each node in the hypergraph before updating the hypergraph each time. And encode it into a 3-bit binary vector.

[0165] Through the above white box embedding calculation of the second small block image of the second slice feature P2, a vector with a dimension of 112×112×16 can be obtained as a structural feature.

[0166] In this embodiment, a hypergraph corresponding to the multi-scale image is constructed by constructing a hypergraph based on the image features and the structural features. The hypergraph construction module is used to obtain a first hypergraph and a second hypergraph based on the image features and the structural features, respectively.

[0167] Specifically, in the process of constructing a hypergraph, the hypergraph is expressed as G=(V, E), where V is a node and E is a hyperedge. V ∈{ v 1 , v2 ,.. v n}, E ∈ V × V , E ∈{ e 1 , e 2 ,.. e k}, the hypergraph includes n nodes, k hyperedges, and the association matrix H ∈ R n×k , the formula is as follows:

[0168]

[0169] in, H ij =1, indicating a hyperedge e j Contains nodes v i Otherwise, it means a hyperedge e j Does not contain nodes v i .

[0170] Diagonal Matrix W ∈ R k×k represents the weight matrix of the hyperedge, where the elements on the diagonal represent the weight of each edge. The degree of the node is , the degree of the hyperedge is .

[0171] The hypergraph construction module may cluster small blocks of images in the image features and structural features through a clustering algorithm, Fuzzy C-means, to construct a first hypergraph G1 and a second hypergraph G2.

[0172] In order to fuse the effective node information of the two hypergraphs, it is necessary to update the hypergraph to obtain more meaningful node embeddings.

[0173] After obtaining the first hypergraph and the second hypergraph, the feature fusion module is used to perform multi-scale fusion on the first hypergraph and the second hypergraph to obtain multi-scale fusion features.

[0174] Among them, the feature fusion module includes a hypergraph convolution layer, a splicing layer, a self-attention module, a hypergraph neural network and a fusion layer. In this embodiment, the self-attention module is implemented using a self-attention network. The hypergraph neural network and the self-attention network are used to mine the multi-scale features of key image blocks, enhance the information interaction of image blocks, effectively fuse cross-scale information, and improve the diagnostic performance of the model.

[0175] The first hypergraph G1 and the second hypergraph G2 are input into the hypergraph convolution layer respectively. The number of output channels of the hypergraph convolution layer is twice the number of input channels. At this time, the distance between data points is more likely to capture the intrinsic structure of the data, which is very helpful for clustering. The hypergraph convolution layer can adopt the message passing method adopted in the above embodiment, which will not be described in detail in this embodiment.

[0176] After the first hypergraph G1 passes through the hypergraph convolution layer, the node embedding is 56×56×128. After passing through the convolution layer Conv0 with a convolution kernel size of 1×1, 128 input channels and 64 output channels and the batch normalization layer, the node feature Em1 size is 56×56×64.

[0177] After the second hypergraph G2 is transformed by the hypergraph convolution layer, its size is 112×112×32, and then it is processed by the downsampling layer to obtain the node feature Em2 of size 56×56×64.

[0178] The concatenation layer concatenates the node features Em1 and Em2 to obtain a node embedding Em_12 of size 56×56×128. Through the size transformation, the fused node embedding is obtained with a size of 56×56×128; then the self-attention network is used to obtain the information that needs attention. In the self-attention network, each element can be associated with other elements, not just dependent on the elements in the adjacent positions. By calculating the relative importance between elements, the self-attention network adaptively captures the long-range dependencies between elements and is suitable for constructing hypergraphs with complex relationships. The formula of the self-attention network is as follows:

[0179] ,in, Q = Xθ 1 , K = Xθ 2 , V = Xθ 3

[0180] in, d k Indicates the dimension of the vector in_dim, divided by d k It is to prevent QK T The value is too large, resulting in softmax Overflow during calculation. θ 1 , θ 2 Represents the learnable parameter matrix of the fully connected layer with in_dim input channels and Q_dim output channels, θ 3 Represents the learnable parameter matrix of the fully connected layer with in_dim input channels and V_dim output channels. The sizes of Q, K and V are in_dim×Q_dim, in_dim×Q_dim and in_dim×V_dim respectively.

[0181] Therefore, when the input feature size is 56×56×128, in_dim is 128, Q_dim is set to 128, and V_dim is set to 64. Then, the output node feature is 56×56×64 after the size transformation. Therefore, the self-attention network can embed Em_12 according to the node after the size transformation to obtain the self-attention adjustment feature.

[0182] The hypergraph neural network consists of two hypergraph learning networks, namely the first hypergraph learning network N1 and the second hypergraph learning network N2. A single hypergraph learning network consists of four process convolution modules, such as Figure 6 As shown, each process convolution module includes d downsampling layers and n hypergraph convolution blocks, where the downsampling layer is implemented by a convolution layer with a stride of 2 and a convolution kernel size of 3×3, which is used to transform the number of nodes when updating the hypergraph.

[0183] Among them, for the four process convolution modules, d is {0, 1, 1, 1}, n is {2, 2, 18, 2}, and the four process convolution modules are respectively composed of 0 sampling layers and 2 supergraph convolution blocks, 1 sampling layer and 2 supergraph convolution blocks, 1 sampling layer and 18 supergraph convolution blocks, and 1 sampling layer and 2 supergraph convolution blocks. Each supergraph convolution block may include a convolution layer Conv1, a clustering algorithm layer Fuzzy C-Means, a supergraph convolution layer, a convolution layer Conv2, a splicing layer, a convolution layer Conv3, an activation function ReLU, a convolution layer Conv3, a batch normalization layer, and a discard layer.

[0184] The number of input channels and output channels of the convolution layer Conv1 are the same. After undergoing the hypergraph convolution layer, the node embedding dimension is twice the original. Then the convolution layer Conv2 restores the node embedding dimension to the original dimension. The number of input and output channels of the convolution layer Conv3 and the convolution layer Conv4 remains unchanged, that is, the dimension of the node embedding will not be changed. The convolution kernel size of the convolution layer is 1×1. Here, 1×1 convolution replaces full connection based on input size considerations. Since the fully connected layer multiplies all elements of the feature map by the weight and then sums them, and the weight vector is preset and unchanged during the network construction process, the fully connected layer cannot adapt to changes in input size. In this embodiment, through the setting of the convolution layer, the output of the 1×1 convolution can be the same size as the input size, so that the output size can change with the change of the input size. The number of hyperedges is fixed to 25.

[0185] Fuzzy C-Means expands the original membership to any value between 0 and 1. A sample can belong to different clusters with different memberships. This method is suitable for constructing hypergraphs, and the nodes of the hypergraph can belong to different hyperedges. FuzzyC-Means assigns each node a membership function belonging to each hyperedge. Nodes are assigned to corresponding hyperedges by the size of the membership value. Nodes may have multiple hyperedges, thereby facilitating complex and diverse inter-object relationships between image patches. The central node of the cluster is regarded as the feature of the class, that is, the weight matrix of the hyperedge.

[0186] The first hypergraph learning network N1 is used to perform node updates on the self-attention adjustment feature based on the association matrix corresponding to the first hypergraph, and obtain the first node update feature Em_1; the second hypergraph learning network N1 is used to perform node updates on the self-attention adjustment feature based on the association matrix corresponding to the second hypergraph, and obtain the second node update feature Em_2. The association matrices of the first hypergraph and the second hypergraph are the association matrices of hypergraphs constructed from image information of two different scales, respectively.

[0187] The self-attention adjustment features are input into the first hypergraph learning network N1 and the second hypergraph learning network N2 respectively to update the representation of the nodes in the hypergraph. The updated node embeddings are input into Fuzzy C-Means to construct the hypergraph, and then the node embeddings are updated through the hypergraph convolution layer. The above tasks are iterated to update the hypergraph. This iterative process ensures the mutual improvement of the small-block image embedding and the hypergraph structure. The structure-aware image representation is enhanced. The introduction of a fixed number of hyperedges as a regularizer further prevents redundant structures in the hypergraph and improves the effectiveness of the hypergraph. Then, the node embeddings are obtained as Em_1 and Em_2, both of size 7×7×512; finally, through the fusion layer, the corresponding elements are added to fuse the two node embedding information to achieve multi-scale information fusion.

[0188] The result prediction module is used to obtain the tissue state analysis result based on the multi-scale fusion features. The result prediction module includes an adaptive average pooling layer and a fully connected layer.

[0189] The size of the first node update feature Em_1 and the second node update feature Em_2 is added together to form a 7×7×512 feature. After passing through an adaptive average pooling layer with an output size of (1, 1), a node feature of 1×1×512 is obtained. Then, through a convolutional layer with an input channel number of 512, an output channel number of 256, and a convolution kernel size of 1×1, a batch normalization layer, and a ReLu activation function, a feature of dimension 1×1×256 is obtained. The squeeze function is used to compress the dimension to 256, and finally the feature is input into a fully connected layer with an input channel number of 256 and an output channel number of 1 to obtain the recurrence risk, that is, the tissue status analysis result.

[0190] For the training of tissue state analysis system, Cox Loss function can be used to estimate the model error. The loss function calculation formula is as follows:

[0191]

[0192] in, R ( t i ) indicates that at time t i A collection of indices of patients who have not previously experienced a relapse. z j and z i They represent the recurrence risk of the j-th and i-th patients output by the model respectively. δ i Represents the indicator function, which is 1 when a patient has a recurrence event, otherwise it is 0. The index for evaluating the recurrence risk prediction task can be a consistency index, which is defined as the proportion of all sample pairs whose predicted recurrence time is correctly sorted in all uncensored samples.

[0193] The present embodiment provides a tissue state analysis system, which combines the multi-scale fusion of pathological images with a hypergraph, fuses the multi-scale pathological image features, and predicts the risk of liver cancer recurrence by learning and fusing them through a hypergraph network. Compared with the traditional graph learning method, the feature fusion in this article realizes cross-scale information interaction, and hypergraph learning can learn high-order complex image features. Acquiring features, i.e., image embedding, is more convenient than the feature engineering acquisition method of machine learning through an automated method. Secondly, the multi-scale hypergraph constructed by the present invention not only takes into account the local features learned by deep learning, but also learns the spatial relationship between small blocks of images. Finally, the feature that the hyperedges of the hypergraph in the present invention can connect any number of nodes provides an advantage for the hypergraph in terms of descriptive ability. The characteristics of more complex pathological images can be learned with high-order modeling capabilities, overcoming the limitation that the graph network method can only establish relationships between pairs of cells or tissues, and realizing accurate recurrence prediction of liver cancer in pathological images.

[0194] Each module in the above-mentioned tissue state analysis system can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0195] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program, and the computer program includes a tissue state analysis system. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The tissue state analysis system is run when the computer program is executed by the processor. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0196] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0197] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the computer program includes a tissue state analysis system, and the tissue state analysis system includes:

[0198] An image segmentation module, used to perform image segmentation on the first slice image and the second slice image in a preset area to obtain a plurality of first small block images and a plurality of second small block images respectively;

[0199] A feature extraction module, configured to extract image features from the plurality of first small-block images respectively to obtain image features; and to extract structural features from the plurality of second small-block images respectively to obtain structural features;

[0200] A hypergraph construction module, used to obtain a first hypergraph and a second hypergraph based on the image features and the structural features respectively;

[0201] A feature fusion module, used for performing multi-scale fusion on the first hypergraph and the second hypergraph to obtain a multi-scale fusion feature;

[0202] The result prediction module is used to obtain the tissue state analysis result based on the multi-scale fusion features.

[0203] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, the computer program comprising a tissue state analysis system, the tissue state analysis system comprising:

[0204] An image segmentation module, used to perform image segmentation on the first slice image and the second slice image in a preset area to obtain a plurality of first small block images and a plurality of second small block images respectively;

[0205] A feature extraction module, configured to extract image features from the plurality of first small-block images respectively to obtain image features; and to extract structural features from the plurality of second small-block images respectively to obtain structural features;

[0206] A hypergraph construction module, used to obtain a first hypergraph and a second hypergraph based on the image features and the structural features respectively;

[0207] A feature fusion module, used for performing multi-scale fusion on the first hypergraph and the second hypergraph to obtain a multi-scale fusion feature;

[0208] The result prediction module is used to obtain the tissue state analysis result based on the multi-scale fusion features.

[0209] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0210] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0211] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0212] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A tissue status analysis system, characterized in that: The tissue state analysis system comprises: An image segmentation module, used to perform image segmentation on the first slice image and the second slice image in a preset area to obtain a plurality of first small block images and a plurality of second small block images respectively; A feature extraction module, configured to extract image features from the plurality of first small-block images respectively to obtain image features; and to extract structural features from the plurality of second small-block images respectively to obtain structural features; A hypergraph construction module, used to obtain a first hypergraph and a second hypergraph based on the image features and the structural features respectively; A feature fusion module is used to perform multi-scale fusion on the first hypergraph and the second hypergraph to obtain a multi-scale fusion feature; the feature fusion module includes a hypergraph convolution layer, a splicing layer, a self-attention module, a hypergraph learning network and a fusion layer, wherein: the hypergraph convolution layer is used to transfer information to the nodes in the first hypergraph to obtain a first initial feature after the node is updated; the information is transferred to the nodes in the second hypergraph to obtain a second initial feature after the node is updated; the splicing layer is used to splice the nodes in the first initial feature and the second initial feature to obtain a splicing feature; the self-attention module is used to update the weights of multiple nodes in the splicing feature based on the dependency relationship between the nodes in the splicing feature to obtain a self-attention adjustment feature; the hypergraph learning network is used to The method comprises the following steps: performing node update on the self-attention adjustment feature to obtain a node update feature; the fusion layer is used to perform node fusion on the node update feature to obtain a multi-scale fusion feature; the hypergraph learning network comprises a plurality of process convolution modules, wherein: the process convolution module comprises a first preset number of downsampling layers and a second preset number of hypergraph convolution blocks, the process convolution module is used to perform node update on the splicing feature to obtain a node update feature; the hypergraph convolution block comprises a clustering algorithm layer, a hypergraph convolution layer and a convolution layer; the clustering algorithm layer is used to perform hyperedge construction on the self-attention adjustment feature based on a clustering algorithm; the hypergraph convolution layer is used to perform information transfer on the nodes in the self-attention adjustment feature; the convolution layer is used to perform convolution on the nodes in the self-attention adjustment feature; The result prediction module is used to obtain the tissue state analysis result based on the multi-scale fusion features.

2. The tissue state analysis system according to claim 1, characterized in that: The tissue state analysis system further comprises an image acquisition module, which is used for: Acquire images of the target tissue slice at a first magnification and a second magnification to obtain a first slice image and a second slice image; The first slice image and the second slice image are respectively subjected to standardization processing to obtain the first slice image and the second slice image after the standardization processing.

3. The tissue state analysis system according to claim 1, characterized in that: The feature extraction module includes an image feature extraction module and a structural feature extraction module, wherein: An image feature extraction module, used for inputting the plurality of first small blocks of images into a pre-trained ResNet18 network to obtain image features; The structural feature extraction module is used to calculate the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block image respectively; based on multiple second small blocks of images, and the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block image, obtain the structural features.

4. The tissue state analysis system according to claim 3, characterized in that: The structural feature extraction module is also used for: The cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information of each second small block of image are respectively encoded to obtain encoding information of the cell nucleus density, cell nucleus size, cell nucleus shape factor and node degree information corresponding to each second small block of image; A structural feature is obtained based on the plurality of second small block images and the encoding information.

5. The tissue state analysis system according to claim 3, characterized in that: The structural feature extraction module includes a cell nucleus recognition module, a shape factor calculation module and a node degree information acquisition module; The cell nucleus recognition module is used to recognize the cell nuclei in each second small block of images; based on the recognition result of the cell nuclei, determine the cell nucleus density and cell nucleus size corresponding to each second small block of images; The shape factor calculation module is used to analyze the shape pattern of each cell nucleus in the single second small block image, and determine the abnormality index of each cell nucleus based on the shape pattern; determining a cell nucleus shape factor of a single second small block of images based on the abnormality indexes of the plurality of cell nuclei; The node degree information acquisition module is used to acquire degree information of multiple nodes; each node corresponds to a single second small block image.

6. The tissue state analysis system according to claim 1, characterized in that: The hypergraph construction module is also used to: cluster the image features and the structural features respectively based on a clustering algorithm to obtain a first hypergraph and a second hypergraph.

7. The tissue state analysis system according to claim 1, characterized in that: The node update feature includes a first node update feature and a second node update feature, and the feature fusion module includes a first supergraph learning network and a second supergraph learning network, wherein: The first hypergraph learning network is used to perform node update on the self-attention adjustment feature based on the association matrix corresponding to the first hypergraph to obtain a first node update feature; The second hypergraph learning network is used to perform node update on the self-attention adjustment feature based on the association matrix corresponding to the second hypergraph to obtain a second node update feature.

8. The tissue state analysis system according to claim 1, characterized in that: The result prediction module includes an adaptive average pooling layer and a fully connected layer; The adaptive average pooling layer is used to adjust the size of the multi-scale fusion feature to obtain the multi-scale fusion feature after the size adjustment; The fully connected layer is used to obtain a prediction result of disease recurrence based on the multi-scale fusion features after size adjustment.