Biological tissue processing method based on semantic consistency network

Abstract

The invention relates to a biological tissue processing method based on a semantic consistency network, and the method comprises the steps: constructing a network model of intra-class and inter-class features of a modeling tissue based on semantic consistency, and the network model comprises N coding blocks, N-1 I2C modules, N compression blocks, and N logic blocks; wherein each coding block, each compression block and each logic block are convolution modules based on a convolutional neural network, and each I2C module comprises an intra-class context module represented by aggregation pixels in a specific class region and an inter-class context module capable of capturing an inter-organization morphological structure; the network model further comprises a consistency module for training the network model; obtaining training sample data; training the network model of the intra-class and inter-class features of the modeling organization based on semantic consistency by using the training sample data; and identifying and processing the biological tissue image data by using the biological tissue identification processing training model. And the representation capability of the feature map is improved through intra-class enhancement and inter-class enhancement.

Description

Technical Field

[0001] The present invention relates to a biological tissue recognition and processing method based on a context semantic consistency network, and in particular to an intra-class and inter-class context and consistency network for supervised and semi-supervised blastocyst image segmentation, which is based on an I2C2Net framework and belongs to the technical field of image pattern recognition based on artificial neural networks and the technical field of artificial intelligence applied to medicine. Background Art

[0002] In recent years, advances in deep learning technology have continuously improved the performance of medical image segmentation. The deep learning network commonly used for medical image segmentation is U-Net[3] and its derivatives. Some studies have attempted to segment blastocyst tissue images, but the segmentation performance still needs to be further improved. In the blastocyst tissue images obtained by photographing, there are significant differences in the intra-class features and adhesion of various tissue categories, such as Figure 1 As shown in a), b) and c), a) shows the original photo image of blastocyst tissue, b) shows the details of the inner cell mass (ICM) area of ​​the original photo image of blastocyst tissue, and c) shows the details of the inner cell mass (ICM) and other types of tissues in the original photo image of blastocyst tissue. Due to the transparency of the blastocyst itself and the fact that blastocyst tissue images are generally collected under low light conditions, the intra-class features of the blastocyst tissue are blurred and the edge features are difficult to distinguish; Figure 1 As shown in Figure (c), the difference in the inter-class structure of different tissue types in the blastocyst tissue image is not obvious, especially in the edge area of ​​the tissue. No inter-class clustering was observed in the results of traditional blastocyst segmentation using U-Net, such as Figure 1 In particular, some regions of the inner cell mass (ICM) were not identified, some regions of the blastocoel were misidentified as trophoblasts, and inter-class structures were not prominent, such as Figure 1 As shown in (f), the errors are mainly concentrated on the boundaries and some difficult-to-process pixels.

[0003] In recent years, some works have attempted to segment blastocyst images, focusing on identifying the inner cell mass (ICM) and trophectoderm (TE). For example, the level set method is used, which defines the initial contour and applies gradient information to segment the inner cell mass and trophoblast; however, in clinical practice, the zona pellucida (ZP) of the blastocyst may rupture, at which time the inner cell mass is outside the zona pellucida, so the recognition effect is not significant. Rad et al. proposed the use of a stacked dilated U-Net model for accurate segmentation of the inner cell mass, and proposed a U-Net variant by combining dilated convolution, which only increased the receptive field of a single pixel, but did not change the attribute relationship between pixels, and the Jaccard index was less than 82%.

[0004] The existing methods are either used for single tissue segmentation or the identification of several tissues. Therefore, methods to improve the performance of blastocyst segmentation still need to be further developed, and the overall identification of single tissues and the structural association of all tissues should be paid attention to. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a biological tissue processing method based on a semantic consistency network, comprising the steps of: A network model for modeling intra-class and inter-class features of tissues based on semantic consistency is constructed, and the network model includes N encoding blocks, N-1 I2C modules, N compression blocks, and N logic blocks; wherein each encoding block, compression block, and logic block is a convolutional module based on a convolutional neural network, and each I2C module includes an intra-class context module representing aggregated pixels in a specific category area and an inter-class context module capable of capturing the morphological structure between tissues; and the N encoding blocks are connected in series in sequence, and the output of the Nth encoding block is input to the corresponding Nth compression block for processing, and the Nth compression block The result is input to the Nth logic block for processing and then input to the N-1th I2C module after upsampling. The N-1th I2C module receives the output of the N-1th coding block and the upsampled output after processing by the Nth logic block, outputs the processed result to the N-1th logic block for processing and then input to the N-2th functional block after upsampling, and so on, until it is processed by the 1st logic block and output as the prediction result; N is an integer greater than 2; Furthermore, the network model for organizing intra-class and inter-class features based on semantic consistency modeling also includes a consistency module for training the network model; Acquire training sample data, where the training sample data is biological tissue image data, a portion of which is labeled data for supervised training, and another portion of which is unlabeled data for semi-supervised training; Using the training sample data, the network model for modeling tissue intra-class and inter-class features based on semantic consistency is trained to obtain a biological tissue recognition processing training model; Use the biological tissue recognition and processing training model to recognize and process biological tissue image data.

[0006] In the above technical solution, N is 5.

[0007] In the above technical solution, each encoding block includes two convolutional network layers, and each convolutional network layer includes two batch normalization layers, two activation layers and one maximum pooling layer.

[0008] In the above technical solution, the coding block is defined as , the encoding block E projects the pixels in each input biological tissue image X into the nonlinear embedding space to obtain the pixel representation matrix :

[0009] in, The size is , C is The number of channels; X is the input biological tissue image; W and H are the width and height of the biological tissue image X respectively.

[0010] In the above technical solution, the compression block is defined as , the compression block F represents the pixel matrix Compress to :

[0011] in, represents the features obtained by aggregating intra-class and inter-class information from different scales of the entire image using the I2C module, The size is , is the number of categories.

[0012] In the above technical solution, the logic block is defined as , the logic block L is used to obtain the supervised logic graph of the biological tissue image X :

[0013] Among them, the logic block L consists of a convolutional layer, a batch normalization layer, and an activation layer.

[0014] In the above technical solution, the probability map is obtained from the output of the logic block The upsampling function is .

[0015] In the above technical solution, the consistency module includes a first convolution block, a second convolution block, a first self-attention block, a third convolution block, a second self-attention block, and a convolution output layer connected in sequence; wherein the first convolution block, the second convolution block, and the third convolution block have the same structure, and all include a convolution layer, a spectral normalization layer, and an activation function layer.

[0016] The present invention also provides a biological tissue processing device based on a semantic consistency network, comprising: A model building module is used to build a network model for modeling intra-class and inter-class features of modeling organizations based on semantic consistency, and the network model includes N encoding blocks, N-1 I2C modules, N compression blocks, and N logic blocks; wherein each encoding block, compression block, and logic block is a convolutional module based on a convolutional neural network, and each I2C module includes an intra-class context module representing aggregated pixels in a specific category area and an inter-class context module capable of capturing the morphological structure between organizations; and the N encoding blocks are connected in series in sequence, and the output of the Nth encoding block is input to the corresponding Nth compression block for processing, and the Nth compression block The result is input to the Nth logic block for processing and then input to the N-1th I2C module after upsampling. The N-1th I2C module receives the output of the N-1th coding block and the upsampled output after processing by the Nth logic block, outputs the processed result to the N-1th logic block for processing and then input to the N-2th functional block after upsampling, and so on, until it is processed by the 1st logic block and output as the prediction result; N is an integer greater than 2; Furthermore, the network model for organizing intra-class and inter-class features based on semantic consistency modeling also includes a consistency module for training the network model; A training sample acquisition module is used to acquire training sample data, wherein the training sample data is biological tissue image data, a portion of which is labeled data for supervised training, and another portion is unlabeled data for semi-supervised training; A model training module, using the training sample data to train the network model of the intra-class and inter-class features of the modeling tissue based on semantic consistency, to obtain a biological tissue recognition processing training model; The data processing module uses the biological tissue recognition and processing training model to recognize and process the biological tissue image data.

[0017] The present invention also provides a biological tissue processing system based on a semantic consistency network, comprising a processor and a storage medium; the storage medium stores program code; the processor is used to call the program code stored in the storage medium to execute the biological tissue processing method based on a semantic consistency network in the above technical solution.

[0018] The present invention has achieved the following technical effects: the present invention proposes a general framework I2C2Net, which utilizes intra-class and inter-class information to continuously improve the performance of biological tissue image segmentation such as blastocysts; a semi-supervised version of I2C2Net based on semantic consistency hypothesis and clustering hypothesis is also proposed, which utilizes labeled and unlabeled data in two different domains without structural changes; the present invention designs a simple and effective IACCM, which enhances aggregation within the same category, and a novel IRCCM, which captures the relationship information between different categories; in addition, the present invention also uses a consistency module to improve the learning and expression capabilities of the basic model without adding additional computational costs; the present invention adopts a top-down feature enhancement path, coupled with a backbone encoder, to improve the representation capability of feature maps through intra-class enhancement and inter-class enhancement in a coarse-to-fine manner. Qualitative and quantitative experimental results demonstrate the effectiveness of our proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The diagram shows the image of blastocyst tissue and its related processing information; a) is the original photo image of blastocyst tissue, b) is the details of the inner cell mass (ICM) region of the original photo image of blastocyst tissue, c) is the details of the inner cell mass (ICM) and the inter-class region of other types of tissues in the original photo image of blastocyst tissue, d) is the true label of different tissues of blastocyst tissue obtained by manual processing, e) is the prediction map obtained by using U-Net, and f) is the difference between e) and d); Figure 2 This is a schematic diagram of the I2C2Net network structure; Figure 3 This is a schematic diagram of the network structure of the intra-class context module (IACCM); Figure 4 Schematic diagram of the network structure of the Inter-Class Context Module (IRCCM); Figure 5 This is a schematic diagram of the network structure of the consistency module (CM); Figure 6 Schematic diagram of the network structure of the semi-supervised version of I2C2Net. DETAILED DESCRIPTION

[0020] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] In order to solve the above problems, the present invention proposes an I2C2Net network architecture to effectively model intra-class and inter-class features of tissues for the segmentation task of blastocyst tissue images. The I2C2Net network architecture consists of an intra-class context module (IACCM) and an inter-class context module (IRCCM); optionally, the I2C2Net network architecture also includes a consistency module (CM) for training.

[0022] The intra-class context module (IACCM) focuses on aggregating pixel representations in a specific category area, strengthening the learning of similar features, and improving the model's ability to identify classification areas related to the real label. By aggregating intra-class features, the intra-class context module (IACCM) can simplify multi-category recognition tasks into multiple binary classification tasks while retaining the ability to learn intra-class features, thereby adaptively enhancing intra-class features and suppressing other types of features, enhancing the degree of aggregation within the class, making the segmentation results more consistent, and improving the accuracy and efficiency of segmentation.

[0023] In addition, the intra-class context module (IACCM) also simplifies a K-class recognition task into K binary classification tasks, reducing the complexity of model learning. However, since the intra-class context module (IACCM) is easier to learn simple samples, a large number of simple samples dominate the gradient, causing the model to face challenges in identifying difficult pixels and weight values. In particular, pixels in edge areas are difficult to identify because of their small proportion and unclear features. In addition, although the decomposition task reduces the complexity of model learning, the intra-class context module (IACCM) does not consider the structural correlation of inter-class samples. Therefore, the present invention proposes an inter-class context module (IRCCM) to solve the problems of the intra-class context module (IACCM) in boundary conditions and difficult pixel processing through a weighted mapping function. The inter-class context module (IRCCM) is designed for the morphological structure reflected in the blastocyst tissue image, and can capture the information changes between various cell tissues during the development of the blastocyst tissue from the inner layer to the outer layer, further enhancing the model's comprehensive understanding and segmentation capabilities of the blastocyst tissue. The inter-class context module (IRCCM) also makes up for the shortcomings of the intra-class context module (IACCM) through the interaction between different categories.

[0024] The combination of the intra-class context module (IACCM) and the inter-class context module (IRCCM) together constitutes a network architecture (referred to as I2C2Net) that can effectively model intra-class and inter-class features of tissues, which is responsible for generating segmentation results of biological tissue images (especially blastocyst tissue images). These two modules are applied at each level of the network and continuously provide supervision information from coarse to fine through a top-down feature enhancement path. In this process, there is full interaction between multi-scale feature information. At the high level of the network, the topological structures of different categories are integrated based on semantic information; at the bottom level of the network, the image edges are refined based on low-level statistical features such as color, edge, and texture. The interaction between intra-class and inter-class enables the model based on the I2C2Net network to learn the potential characteristics of the embryo, thereby improving the accuracy and robustness of blastocyst segmentation.

[0025] In order to improve the accuracy of embryonic tissue recognition without adding additional parameters and reasoning computational burden, the present invention focuses on optimizing the model's ability to handle data distribution differences. These differences, especially category boundaries and distribution characteristics, are crucial for quality assessment of image segmentation. Therefore, the present invention designs an innovative consistency module (CM) for supervised training, which organically integrates the original embryonic image with the segmentation result, and improves the recognition accuracy without increasing the computational burden by improving the model's ability to learn the original data distribution. In the training process of I2C2Net, CM plays a vital role, which promotes the improvement of distribution consistency between model prediction results and true labels. More importantly, in order to meet the challenge of scarce labeled data and better serve clinical needs, the present invention proposes a semi-supervised version of I2C2Net (Semi-supervised learning, SSL) based on semantic consistency assumptions and clustering assumptions across heterogeneous domains, which provides an effective way to solve the above problems. The above method can more efficiently capture the data distribution differences between embryonic tissue categories and improve the accuracy of embryonic segmentation.

[0026] The network (I2C2Net) framework provided by the present invention can effectively model the intra-class and inter-class features of the organization. Figure 2 As shown. The I2C2Net network includes N encoding blocks (such as Figure 2 middle As shown), I2C module (such as Figure 2 IACCM, IACCM), compressed blocks (such as Figure 2 middle As shown), logical blocks (such as Figure 2 middle Each I2C module includes an I2C framework consisting of an intra-class context module IACCM and an inter-class context module IRCCM. The blastocyst image data X is sequentially input into N encoding blocks in a serial manner. , i is 1 to N; among them, the Nth coding block The output of is input to the corresponding Nth compression block Processing, Nth compressed block The result is input to the Nth logic block After processing, it is upsampled and input to the N-1th I2C module, and the N-1th I2C module receives the N-1th encoding block The output of the Nth logic block The processed upsampled output is output to the N-1th logic block after processing After processing, it is upsampled and input to the N-2th functional block, and so on until the 1st logic block After processing, it is output as the prediction result. Among them, the coding block , compressed block , logical block Both are convolution modules.

[0027] During the training and working (testing) process, the I2C2Net framework uses the intra-class and inter-class context information extracted by the intra-class context module (IACCM) and the inter-class context module (IRCCM) to segment biological tissues (such as blastocyst images). The consistency module (CM) is only involved in the training phase, so no additional computational workload is added during the working (testing) phase.

[0028] For the training and working (testing) of I2C2Net, we can consider a set of images S, let ,in represents the original input embryo image, represents the corresponding true label obtained by manual processing. Because each input image They are processed separately, so for simplicity, the subscript m is omitted in the specific processing flow for each embryo image in S.

[0029] Define the encoding block as , where n represents the nth coding block. Each It consists of two convolutional layers, each with two batch normalization (BN) layers, two ReLU activation layers, and a maximum pooling layer. I2C2Net first uses the encoding block E to project the pixels in each input embryonic tissue image X into a nonlinear embedding space to obtain a pixel representation matrix , and then nest the projection N-1 times: (1) in, The size is , C is The number of channels; n is the number of nested projections; X is the input embryonic tissue image; W and H are the width and height of the embryonic tissue image X, respectively; N is the number of encoding blocks E, that is, the total number of nested projections to the nonlinear embedding space, and in the present invention, preferably N=5. The convolution module can be composed of convolution kernels with different kernel sizes, which play the role of feature mapping. The convolution module can also adopt other different network structures.

[0030] The present invention uses compressed blocks Representing pixels as a matrix Compress to , with K channels (K is the number of categories), the process is as follows: (2) in, represents the features obtained by aggregating intra-class and inter-class information from different scales of the entire image using the I2C framework, The size is .

[0031] I2C Framework By the intra-class context model and inter-class context models Composition, intra-class context model and inter-class context models At the same time, the high-level features output by the encoding block with the corresponding serial number and the low-level features output by the logic block with the previous serial number are processed.

[0032] By compressing the block After compressing the channels represented, use the logical block Get the supervised logistic graph of image X , whose size is .

[0033] (3) Among them, the logic block It consists of a convolutional layer, a BN layer, and a ReLU layer. Therefore, before obtaining the aggregate information of intra-class and inter-class modules, we can Determine the probability map in Figure 2 shown.

[0034] In order to match the high-level feature scale, the present invention uses an upsampling function from the logic block The output size is Probability plot of .

[0035] (4) in is the sigmoid function.

[0036] The upsampling function is specifically .

[0037] Then, the probability map For I2C module Enhancement feature : (5) (6) (7) Finally, the prediction output graphs of different scales are represented as: (8) For Intra-Class Context Modules (IACCM): In the I2C module, IACCM It aims to capture the contextual information of the same category in the image, such as Figure 3 As shown in Figure 2, feature enhancement can be achieved between the same scale and different scales. Before scale enhancement within the same category, each layer of the logistic map passes through a supervised module before fusion. These logistic maps are then activated together with neighboring high-level features that focus on the contextual information of the same category.

[0038] In order to match the adjacent high-level features with the probability map of K-type tissues, the present invention sequentially Divided into K groups. is a size of The matrix of represents the number of channels of class k. Accordingly, the number of parameters can be effectively reduced.

[0039] (9) in, Indicates floor operation.

[0040] For convenience, we define is the logical graph under category k. In order to aggregate the context information of each category, the representation area of ​​each category k is calculated as follows: (10) Finally, put all Connect and use Convolution Features, which contain contextual information of the same category in the image. By grouping, the present invention decomposes the multi-category recognition task into multiple two-category recognition tasks, thereby reducing the learning difficulty of the model and allowing the model to focus on learning the features of the same category without being disturbed by other categories. In addition, the number of parameters is reduced to a certain extent.

[0041] For Inter-Class Context Module (IRCCM): Although IACCM effectively solves the problem of feature representation and learning within the same category, it does not consider the inter-class connections. Figure 1 As shown in (f), the prediction errors between different categories account for a large proportion of the image, including some pixels with unclear structural features. In order to solve this problem, the present invention designs IRCCM , which can handle the relationship information between different categories and some difficult pixels that are difficult to identify, such as Figure 4 shown.

[0042] IACCM and IRCCM have the same high-level features and follow the same grouping principles. Intra-class grouping considers the number of tissue categories, but inter-class grouping considers the correlation between tissues, represented by K-1 groups. Taking a typical blastocyst tissue image as an example, generally speaking, from the inside to the outside, they are: inner cell mass, blastocyst cavity, trophoblast, zona pellucida, and background. Therefore, only four tissue relationships are generally considered, namely: 1, 1+2, 1+2+3, and 1+2+3+4, where 1, 2, 3, and 4 represent inner cell mass, blastocyst cavity, trophoblast, and zona pellucida, respectively. Compare Figure 3 and Figure 4 As shown, the feature information of the upsampling logic map processed in IACCM is: 0 (no feature or background feature), 1 (inner cell mass feature), 2 (blastocyst cavity feature), 3 (trophoblast feature), 4 (zona pellucida feature); the feature information of the upsampling logic map processed in IRCCM is: 1 (inner cell mass feature), 1+2 (inner cell mass + blastocyst cavity feature), 1+2+3 (inner cell mass + blastocyst cavity + trophoblast feature), 1+2+3+4 (inner cell mass + blastocyst cavity + trophoblast + zona pellucida feature).

[0043] In order to aggregate contextual information of different categories, the representation region is calculated as follows: (11) in is a weighted mapping function that can be used to enhance the representation of boundaries and difficult pixels. According to formula (11), the structural feature map between tissues from inside to outside (inner cell mass to zona pellucida) can be obtained.

[0044] According to the characteristics of cross entropy loss, when the probability in IRCCM is close to 0.5, the greater the loss, the more difficult it is to identify the sample. For embryonic tissue samples, the boundaries between different tissues are small. In addition, since the blastocyst is transparent and needs to be photographed under low light conditions, its boundary features are not clear. Therefore, the present invention designs a weighted mapping function To redistribute the probability weight of each category, that is, pixels that are not easy to identify (such as boundary areas) usually have a probability value of about 0.5, while pixels that are easy to identify have values ​​close to 1 or 0. In this way, the weight value of sample pixels that are not easy to identify is close to , the weight value of easy-to-identify sample pixels is close to 1. Therefore, the model has ideal learning and reasoning capabilities for edge and difficult pixels. It can be expressed as: (12) Among them, α and β can be selected as activation factors fixed at 1 and 2 respectively. When such values ​​are taken, Equation (12) is formally the same as the upsampling function same.

[0045] Finally, connect in order and use Convolution Features contain contextual information of different categories in the image.

[0046] For the Semantic Consistency Module (CM): Based on the assumption of semantic consistency, the present invention proposes a semantic consistency module (CM) for training, which helps to approximate the data distribution of the results output by I2C2Net to the distribution of true labels, which improves the generalization ability of the model.

[0047] The present invention sends the final results of I2C2Net under different conditions as generated samples to the consistency module (CM) and obtains a score for evaluating the quality. The scoring result can positively supervise the segmentation network and fit the segmentation results from the perspective of distribution, especially in distinguishing the distribution of different categories. At the same time, the segmentation result is a multi-category value map, which contains relatively less information. The real sample (original RGB image) is combined with the segmentation result and input into the discriminator for scoring to improve the semantic understanding ability and segmentation performance, such as Figure 5 shown.

[0048] The consistency module (CM) includes a first convolution block, a second convolution block, a first self-attention block, a third convolution block, a second self-attention block, and a convolution output layer, which are connected in sequence; wherein the first convolution block, the second convolution block, and the third convolution block have the same structure, and all include a convolution layer, a spectral normalization layer, and an activation function layer.

[0049] In actual operation, the manually labeled true label or the segmentation result output by I2C2Net prediction is combined with the original image as the input of the consistency module (CM), where the combination of the true label and the original image is the true result, and the combination of the segmentation result and the original image is the false result. The combined input data is processed by the first convolution block, the second convolution block, and the third convolution block in turn. Each convolution block adds a spectral normalization (SN) operation and a LeakyReLU activation function (such as Figure 5 SN is widely used in the training phase of GAN to ensure stability. The present invention applies SN to the consistency module to satisfy the 1-Lipschitz condition, which limits the degree of function change and makes the model more stable. After three convolution blocks, two self-attention blocks and one convolution block are passed in the middle, and then the final result is obtained through a single-layer convolution.

[0050] The present invention uses a consistency loss function to train the consistency module (CM). The present invention uses a cross entropy loss function to supervise the logic graph. The loss function can be expressed as: (13) in, Represents a unique hot label Zoom to size. Using formula (14) we get: (14) Prediction output graphs at different scales.

[0051] represents the cross entropy loss, It means to sum the values ​​of all positions in the input image X.

[0052] The total cross entropy loss is expressed as: (15) In order to improve the learning and expression ability of the basic model without increasing the computational cost, this paper proposes an adversarial loss function to compensate for the cross entropy loss. The combination of the original embryo image X is input into the consistency module as "fake data", and the combination of the true label Y and the original image X is input into the consistency module as "real data". The consistency module extracts the combined data features from shallow to deep and outputs the true / false result. The process can be expressed by formula 16: (16) Among them, Score represents the discrimination score, which is the output of the consistency module. . Represents a join operation. Represents a consistency module. is a segmentation image, which can be represented by the true label or the predicted map of the same input X.

[0053] In order to make the training stable, we adopt LSGAN loss in our consistency module, and introduce LSGAN loss into the training through alternating updates, which helps the consistency loss : (17) Among them, X represents the original image, is the data distribution of the original image. Y represents the true label, is the prediction result of X, given by get. Represents a segmented network, i.e., I2C2Net in the present invention.

[0054] Finally, the total loss function is expressed as: (18) in Indicates for balance and The hyperparameters of the loss, preferably Using the joint loss function, the model parameters are jointly learned via back-propagation, and the proposed I2C2Net can be trained in a coarse-to-fine manner.

[0055] Through the method of the present invention, it can be found that although the textures of the blastocyst images from the two domains are diverse, it can still be found that they contain the same tissues (e.g., blastocyst cavity, inner cell mass, trophoblast and zona pellucida), that is, these heterogeneous domains share the same semantic context information. Based on this discovery, the present invention proposes a semantic consistency hypothesis: when processing images of another heterogeneous domain, a well-trained network may still maintain partial (not all) semantic consistency. In fact, after being trained on a public dataset, I2C2Net can still distinguish the rough areas of these tissues in the self-collected images to a certain extent (such as Figure 6 On the other hand, the clustering assumption, that is, the decision boundary should be in a low-density area, is a basic assumption in semi-supervised learning. That is, the partial semantic consistency obtained from unlabeled images helps to increase the density of the labeled dataset and thus improve the accuracy. Based on these two assumptions, the design of the entire training strategy is as follows Figure 6 It mainly includes the following five steps: Step S100: I2CNet first labels the data The consistency module and the total loss function of formula (18) are used for training.

[0056] Step S200: Unlabeled blastocyst image From another heterogeneous domain, segmented by the I2CNet pre-trained in the first step, these segmentation maps are then used as input The corresponding pseudo labels According to the semantic consistency hypothesis, the semantic information contained in the pseudo-labels is consistent in physiological structure.

[0057] Step S300: For a labeled dataset ( and the corresponding ), the supervised consistency loss is obtained by (19) in is the I2CNet corresponding to the input The segmentation result. The total loss is: (20) in Calculated by formula (15). Preferably, Equal to 2.0.

[0058] Step S400: For unlabeled images and its pseudo labels , an unsupervised consistency loss is designed: (twenty one) in is the segmentation map. The consistency module tries to exploit semantic consistency and encourages the density of different segmentation categories.

[0059] Step S400: Figure 6 As shown in Figure 1, I2C2Net considers both labeled and unlabeled data. Therefore, the total loss function of network training is finally: (twenty two) in is a parameter, preferably Set to 0.1, Equal to 2.0.

[0060] As for the consistency module, it is: (twenty three) I2C2Net uses I2CNet as the segmentation backbone and achieves semantic consistency in the semi-supervised learning setting.

[0061] The supervised dataset used in the test of the present invention is a public dataset of human blastocysts, which contains 249 blastocyst images and their true labels (GT) provided by the Pacific Center for Reproductive Medicine (PCRM). The training and test sets contain 199 (80%) and 50 images (20%). The dataset collected in the test data includes 84 human blastocyst images, which are from the Reproductive Medicine Center of the First Affiliated Hospital of Anhui Medical University. The blastocyst images were taken on an inverted microscope (IX-71; Olympus, Japan) and a heated micromanipulator (Nikon, eclipse Ti2, Japan). The relevant work has been approved by the Medical Ethics Committee of Anhui Medical University and follows the Declaration of Helsinki.

[0062] The backbone of the I2C2Net of the present invention and the two integrated context modules are randomly initialized by Kaiming initialization. The factor is The “poly” learning rate strategy is used for training. The optimizer is Adam, and the initial learning rate and weight decay are set to The number of training epochs is 2000. Synchronous batch normalization implemented in PyTorch is used during training. For data augmentation, each sample is randomly scaled, randomly rotated, randomly cropped, and flipped left and right during the training phase. The input image size is The proposed method is implemented using PyTorch (version ≥ 1.3) and trained using four NVIDIA Tesla V100 GPUs (each with 32 GB memory). All test programs are executed on a single NVIDIA Tesla V100 GPU.

[0063] In order to evaluate the performance of the network, five common indicators are used: Accuracy, Precision, Recall, Dice coefficient and Jaccard index. These indicators are calculated as follows: Accuracy = (twenty four) Accuracy = , (25) Recall = (26) Dice coefficient = , (27) Jaccard Index = , (28) Where TP represents the number of pixels that correctly identify the tissue region; TN is the number of pixels that are correctly identified as the background (unsegmented region); FP represents the number of pixels that misclassify the background as the blastocoel, inner cell mass, trophoblast, and zona pellucida regions; and FN represents the number of pixels that are misclassified as the background. Accuracy represents the effectiveness of the method in correctly distinguishing between background and blastocyst tissue. Precision represents the proportion of all pixels that are correctly predicted to be blastocyst tissue. Recall represents the effectiveness of the method in detecting blastocyst tissue. The Dice coefficient and Jaccard index are crucial to evaluate the overall segmentation performance and are suitable for dealing with class imbalance; in addition, the impact of misidentified and missed regions is also considered. The range of the Dice coefficient and Jaccard index is 0 to 1, and the closer the index is to 1, the better the segmentation performance.

[0064] The ablation experiment results verify the effectiveness of IACCM, IRCCM and CM proposed in the present invention. Compared with other supervised methods, the I2C2Net of the present invention achieves the best performance in accuracy, precision, recall, Dice coefficient and Jaccard index, which are 93.79%, 91.69%, 92.23%, 91.95% and 85.33% respectively. Moreover, the accuracy, Dice and Jaccard index of I2C2Net on the supervised data set are improved by 0.49%, 0.31% and 0.09% respectively; the semi-supervised version of I2C2Net also achieves the best performance compared with other mainstream semi-supervised methods of the same period; in particular, the model accuracy index of the present invention is improved by at least 0.53%, the precision is improved by at least 0.54%, the Dice coefficient is improved by at least 0.67%, and the Jaccard index is improved by at least 1.14%.

[0065] Through the detailed description of the specific implementation methods and embodiments of the present invention above, ordinary technicians in the field can understand that various changes, modifications, substitutions and variations can be made to these implementation methods and embodiments without departing from the principles and basic concepts of the present invention, and the scope of the present invention is defined by the attached claims and their equivalents. Contents not described in detail in this specification, such as the details of the specific structure and implementation of a typical neural network and its functional layer, are within the scope of what can be achieved by professional and technical personnel in the field based on the known prior art and the technical teachings of the present invention.

Claims

1. A biological tissue processing method based on semantic consistency network, characterized in that Includes steps: A network model for modeling intra-class and inter-class features of tissues based on semantic consistency is constructed, and the network model includes N encoding blocks, N-1 I2C modules, N compression blocks, and N logic blocks; wherein each encoding block, compression block, and logic block is a convolutional module based on a convolutional neural network, and each I2C module includes an intra-class context module representing aggregated pixels in a specific category area and an inter-class context module capable of capturing the morphological structure between tissues; and the N encoding blocks are connected in series in sequence, and the output of the Nth encoding block is input to the corresponding Nth compression block for processing, and the Nth compression block The result is input to the Nth logic block for processing and then input to the N-1th I2C module after upsampling. The N-1th I2C module receives the output of the N-1th coding block and the upsampled output after processing by the Nth logic block, outputs the processed result to the N-1th logic block for processing and then input to the N-2th functional block after upsampling, and so on, until it is processed by the 1st logic block and output as the prediction result; N is an integer greater than 2; Furthermore, the network model for organizing intra-class and inter-class features based on semantic consistency modeling also includes a consistency module for training the network model; Acquire training sample data, where the training sample data is biological tissue image data, a portion of which is labeled data for supervised training, and another portion of which is unlabeled data for semi-supervised training; Using the training sample data, the network model for modeling tissue intra-class and inter-class features based on semantic consistency is trained to obtain a biological tissue recognition processing training model; Use the biological tissue recognition and processing training model to recognize and process biological tissue image data.

2. The biological tissue processing method based on semantic consistency network according to claim 1, characterized in that: The N is an integer of 5 or greater.

3. The biological tissue processing method based on semantic consistency network according to claim 2, characterized in that: Each encoding block consists of two convolutional network layers, and each convolutional network layer consists of two batch normalization layers, two activation layers and a maximum pooling layer.

4. The biological tissue processing method based on semantic consistency network according to claim 3, characterized in that: Define the encoding block as , the encoding block E projects the pixels in each input biological tissue image X into the nonlinear embedding space to obtain the pixel representation matrix : ； in, The size is , C is The number of channels; X is the input biological tissue image; W and H are the width and height of the biological tissue image X respectively.

5. The biological tissue processing method based on semantic consistency network according to claim 4, characterized in that: Define the compression block as , the compression block F represents the pixel matrix Compress to : ； in, represents the features obtained by aggregating intra-class and inter-class information from different scales of the entire image using the I2C module, The size is , K is the number of categories.

6. The biological tissue processing method based on semantic consistency network according to claim 5, characterized in that: Define the logic block as , the logic block L is used to obtain the supervised logic graph of the biological tissue image X : ； Among them, the logic block L consists of a convolutional layer, a batch normalization layer, and an activation layer.

7. The biological tissue processing method based on semantic consistency network according to claim 6, characterized in that: Get the probability map from the output of the logic block The upsampling function is .

8. The biological tissue processing method based on semantic consistency network according to claim 1, characterized in that: The consistency module includes a first convolution block, a second convolution block, a first self-attention block, a third convolution block, a second self-attention block, and a convolution output layer which are connected in sequence; wherein, the first convolution block, the second convolution block, and the third convolution block have the same structure, and all include a convolution layer, a spectral normalization layer, and an activation function layer.

Images