Pathological image classification method and device

Abstract

The invention provides a pathological image classification method and device, and can be applied to the technical field of image processing. The method comprises the following steps: acquiring a tissue slice image for a stomach region of a target object; performing morphological feature extraction on the tumor tissue lesion area to obtain morphological prompt features; performing visual feature extraction on the tissue slice image to obtain a visual prompt feature; based on the self-adaptive weight parameters, the morphological prompt features and the visual prompt features, mask image blocks corresponding to the multiple tissue slice image blocks are processed through a detection model, the recognition result corresponding to the tissue slice image is obtained, the tissue slice image blocks are obtained by cutting the tissue slice image, and the recognition result corresponding to the tissue slice image is obtained. The self-adaptive weight parameters are used for adjusting balance distribution among the morphological prompt features, the visual prompt features and the mask image blocks, and the identification result represents and predicts the expression degree category of the epidermal growth factor receptor 2 gene in the tumor tissue lesion area.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and particularly to a method and apparatus for classifying pathological images. Background Art

[0002] Overexpression or amplification of the Human Epidermal Growth Factor Receptor 2 (HER2) gene is considered an important biomarker in gastric cancer. In many studies, it has been demonstrated that HER2-based targeted therapy technology can significantly improve the survival period of patients with advanced gastric cancer. Therefore, HER2-based targeted therapy technology has shown good effects in the basic treatment research of gastric cancer.

[0003] However, the expression of HER2 in gastric cancer is significantly heterogeneous. Existing models rely on large-scale labeled data, and the accuracy of manual labeling often depends on professional knowledge and experience, with high limitations. In complex tasks, the model's reasoning ability and generalization ability are poor, and the accuracy of HER2 gene expression prediction is low. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a method and apparatus for classifying pathological images.

[0005] According to a first aspect of the present disclosure, there is provided a method for classifying pathological images, including: obtaining a tissue section image of the gastric region of a target object, wherein the tissue section image is marked with a tumor tissue lesion region; extracting morphological features of the tumor tissue lesion region to obtain morphological cue features; extracting visual features of the tissue section image to obtain visual cue features; based on an adaptive weight parameter, the morphological cue features, and the visual cue features, using a detection model to process mask image patches corresponding to respective tissue section image patches to obtain an identification result corresponding to the tissue section image, wherein the tissue section image patches are obtained by cutting the tissue section image, the adaptive weight parameter is used to adjust the balanced distribution among the morphological cue features, the visual cue features, and the mask image patches, and the identification result represents the expression degree category of the epidermal growth factor receptor 2 gene in the predicted tumor tissue lesion region.

[0006] According to an embodiment of the present disclosure, the detection model includes N layers of attention mechanisms; wherein, based on the adaptive weight parameters, morphological cue features, and visual cue features, processing the mask image patches corresponding to multiple tissue slice image patches by using the detection model to obtain the recognition result corresponding to the tissue slice image includes: splicing the morphological cue features and the visual cue features to obtain the spliced cue features; inputting the n-th self-attention result and the spliced cue features into the (n + 1)-th layer of attention mechanism to obtain the (n + 1)-th self-attention result, where, when n is equal to 1, the 1st self-attention result is obtained by performing self-attention mechanism learning on the adaptive weight parameters, the spliced cue features, and the mask image patches corresponding to multiple tissue slice image patches, 1 n N - 1; performing classification processing on the N-th self-attention result to obtain the recognition result corresponding to the tissue slice image.

[0007] According to an embodiment of the present disclosure, the (n + 1)-th layer of attention mechanism includes the (n + 1)-th query weight parameter, the (n + 1)-th key weight parameter, and the (n + 1)-th value weight parameter; wherein, inputting the n-th self-attention result and the spliced cue features into the (n + 1)-th layer of attention mechanism to obtain the (n + 1)-th self-attention result includes: respectively obtaining the (n + 1)-th query vector, the (n + 1)-th key vector, and the (n + 1)-th value vector according to the products of the n-th self-attention result and the (n + 1)-th query weight parameter, the (n + 1)-th key weight parameter, and the (n + 1)-th value weight parameter; splicing the spliced cue features and the (n + 1)-th key vector to obtain the (n + 1)-th key-spliced vector; splicing the spliced cue features and the (n + 1)-th value vector to obtain the (n + 1)-th value-spliced vector; determining the (n + 1)-th self-attention result according to the (n + 1)-th query vector, the (n + 1)-th key-spliced vector, and the (n + 1)-th value-spliced vector.

[0008] According to an embodiment of the present disclosure, the 1st layer of attention mechanism includes the 1st query weight parameter, the 1st key weight parameter, and the 1st value weight parameter; wherein, performing self-attention mechanism learning on the adaptive weight parameters, the spliced cue features, and the mask image patches corresponding to multiple tissue slice image patches to obtain the 1st self-attention result includes: performing encoding processing on the adaptive weight parameters, the spliced cue features, and the mask image patches corresponding to multiple tissue slice image patches to obtain the initial embedding vectors; respectively obtaining the 1st query vector, the 1st key vector, and the 1st value vector according to the products of the initial embedding vectors and the 1st query weight parameter, the 1st key weight parameter, and the 1st value weight parameter; determining the 1st self-attention result according to the 1st query vector, the 1st key vector, and the 1st value vector.

[0009] According to an embodiment of the present disclosure, morphological feature extraction is performed on the lesion area of the tumor tissue, and the morphological prompt features obtained include: determining the area value of the tumor tissue lesion area according to the number of pixel points in the tumor tissue lesion area; determining the perimeter value of the tumor tissue lesion area according to the distance between every two adjacent pixel points on the boundary of the tumor tissue lesion area; determining the circularity of the tumor tissue lesion area according to the area value and the perimeter value; determining the aspect ratio of the tumor tissue lesion area according to the major axis length and the minor axis length of each cell in the tumor tissue lesion area; splicing the area value, the perimeter value, the circularity, and the aspect ratio to obtain the morphological prompt features.

[0010] According to an embodiment of the present disclosure, visual feature extraction is performed on the tissue section image, and the visual prompt features obtained include: determining the preset gray-scale pair condition according to the preset first gray-scale value and the preset second gray-scale value; determining a plurality of initial gray-scale pairs according to the gray-scale values of every two adjacent pixel points in the tissue section image; determining the target gray-scale pair from the plurality of initial gray-scale pairs based on the preset gray-scale pair condition; determining the joint probability according to the number of the target gray-scale pairs and the number of the initial gray-scale pairs; obtaining the visual prompt features according to the joint probability, the pixel hue value, and the pixel brightness value.

[0011] According to an embodiment of the present disclosure, the mask image blocks corresponding to the plurality of tissue section image blocks are obtained based on the following operations: processing the tissue section image by using a threshold segmentation algorithm to obtain an initial mask image; performing morphological processing on the initial mask image to obtain a target mask image; obtaining the mask image blocks corresponding to the plurality of tissue section image blocks according to the target mask image and the plurality of tissue section image blocks.

[0012] According to an embodiment of the present disclosure, the detection model is trained based on the following operations: obtaining training samples, where the training samples include sample tissue section images, sample mask image blocks, and sample labels for the gastric region of the sample target object, and the sample labels represent the expression degree categories of the epidermal growth factor receptor 2 gene on the lesion area of the tumor tissue in the sample tissue section image; performing feature extraction on the sample tissue section image to obtain sample morphological prompt features and sample visual prompt features; processing a plurality of sample mask image blocks by using the detection model based on the adaptive weight parameters, the sample morphological prompt features, and the sample visual prompt features to obtain sample prediction image blocks corresponding to the sample mask image blocks and sample recognition results corresponding to the sample prediction image blocks; training the detection model according to the sample mask image blocks, the sample prediction image blocks, the sample recognition results, and the sample labels to obtain the trained detection model.

[0013] According to an embodiment of the present disclosure, training a detection model based on a sample mask image patch, a sample prediction image patch, a sample recognition result, and a sample label to obtain a trained detection model includes: obtaining a sample recognition result corresponding to a sample tissue section image according to the sample recognition results respectively corresponding to a plurality of sample prediction image patches; calculating a loss value between the sample recognition result corresponding to the sample tissue section image and the sample label by using a cross-entropy loss function to obtain a classification loss value; calculating a loss value between the sample mask image patch and the sample prediction image patch by using a perceptual loss function to obtain a perceptual loss value; obtaining an objective loss value according to the classification loss value and the perceptual loss value; and training the detection model according to the objective loss value to obtain a trained detection model.

[0014] According to a second aspect of the present disclosure, there is provided a pathological image classification device, including: an acquisition module, configured to acquire a tissue section image of a gastric region of a target object, wherein the tissue section image is marked with a tumor tissue lesion region; a first feature extraction module, configured to extract morphological features of the tumor tissue lesion region to obtain morphological cue features; a second feature extraction module, configured to extract visual features of the tissue section image to obtain visual cue features; a detection module, configured to, based on an adaptive weight parameter, the morphological cue features, and the visual cue features, process a mask image patch corresponding to each of a plurality of tissue section image patches by using a detection model to obtain a recognition result corresponding to the tissue section image, wherein the tissue section image patch is obtained by cutting the tissue section image, the adaptive weight parameter is used to adjust a balanced distribution among the morphological cue features, the visual cue features, and the mask image patch, and the recognition result represents a category of the expression level of the epidermal growth factor receptor 2 gene in the predicted tumor tissue lesion region.

[0015] According to the pathological image classification method and device provided by the present disclosure, morphological cue features are obtained by extracting morphological features of a tumor tissue lesion region; visual cue features are obtained by extracting visual features of a tissue section image; and based on an adaptive weight parameter, the morphological cue features, and the visual cue features, a mask image patch corresponding to each of a plurality of tissue section image patches is processed by using a detection model to obtain a recognition result corresponding to the tissue section image. Since significant feature distributions related to HER2 gene expression in the tissue section image are captured based on meticulous and high-dimensional feature extraction by extracting morphological cue features and signal cue features; the detection model receives the mask image patch, and at the same time, takes the adaptive weight parameter, the morphological cue features, and the signal cue features as additional information and transmits them to the detection model, so that the detection model can learn both the features directly presented in the image and the biological features of the image pixels, and assist in targeted adjustment of image features, effectively detecting and classifying the HER2 gene expression, improving the accuracy and efficiency of the detection model classification, and providing an important clinical basis for assisting doctors in targeted treatment of gastric cancer. Description of the Drawings

[0016] Through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become clearer.

[0017] Figure 1 The flowchart of the pathological image classification method according to the embodiment of the present disclosure is shown.

[0018] Figure 2 The flowchart of determining the (n + 1)-th self-attention result according to the embodiment of the present disclosure is shown.

[0019] Figure 3 The schematic diagram of the training process of the detection model according to the embodiment of the present disclosure is shown.

[0020] Figure 4 The structural block diagram of the pathological image classification device according to the embodiment of the present disclosure is shown. Detailed Embodiments

[0021] Hereinafter, embodiments according to the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure. However, it is obvious that one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present disclosure.

[0022] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] In the case of using expressions such as "at least one of A, B, and C", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C).

[0025] In the process of implementing the present disclosure, it is found that the expression of HER2 in gastric cancer has significant heterogeneity, and existing detection methods, such as immunohistochemistry (IHC) and in situ hybridization (ISH) techniques, although widely used in clinical practice, still have certain limitations in accurately evaluating the HER2 expression heterogeneity in gastric cancer patients. Through traditional pathological image analysis methods, clinicians often face the challenge of being unable to accurately classify and evaluate the HER2 expression level of patients, which directly affects treatment decisions and efficacy prediction; in addition, traditional model prompt learning relies on large-scale manually labeled data, and the model has poor reasoning ability and generalization ability in complex tasks, resulting in low prediction accuracy.

[0026] In view of this, according to an embodiment of the present disclosure, a pathological image classification method and apparatus are provided. The method includes: obtaining a tissue section image of the gastric region of a target object; extracting morphological features from the lesion region of the tumor tissue to obtain morphological prompt features; extracting visual features from the tissue section image to obtain visual prompt features; based on an adaptive weight parameter, morphological prompt features, and visual prompt features, using a detection model to process the mask image blocks corresponding to multiple tissue section image blocks to obtain an identification result corresponding to the tissue section image, where the tissue section image blocks are obtained by cutting the tissue section image, and the adaptive weight parameter is used to adjust the balanced distribution among the morphological prompt features, visual prompt features, and mask image blocks, and the identification result represents the expression degree category of the epidermal growth factor receptor 2 gene in the predicted tumor tissue lesion region.

[0027] In the technical solution of the present disclosure, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the processing of relevant data, such as collection, storage, use, processing, transmission, provision, disclosure, and application, complies with relevant laws, regulations, and standards, takes necessary confidentiality measures, does not violate public order and good customs, and provides corresponding operation entrances for users to choose to authorize or refuse.

[0028] It should be noted that the serial numbers of the following operations in the method are only used to represent the operations for description, and should not be regarded as indicating the execution order of the respective operations. Unless explicitly stated, the method does not need to be executed exactly in the order shown.

[0029] Figure 1 The flowchart of the pathological image classification method according to an embodiment of the present disclosure is shown.

[0030] As Figure 1 shown, the pathological image classification method includes operations S110 to S140.

[0031] In operation S110, obtain a tissue section image of the gastric region of the target object.

[0032] According to an embodiment of the present disclosure, the target object is an object with tumor lesion cells in the gastric region.

[0033] According to an embodiment of the present disclosure, a tissue section image (Whole Slide Image, WSI) of the target object is obtained by a high-resolution digital pathology scanner. The tissue section image is obtained by scanning and stitching sections of multiple tissues, with a resolution of several thousand pixels, which can finely display the tissue structure. The tissue section image can be represented as a large-size matrix , where and are the height and width of the tissue section image respectively, is the number of color channels of the tissue section image.

[0034] According to an embodiment of the present disclosure, a pathologist manually annotates or uses an existing automated segmentation tool to assist in marking the tumor tissue lesion area in the tissue section image. The tissue section image also includes information on normal gastric tissue in the gastric region as a control, which is used to better assist in detecting the expression of the Human Epidermal Growth Factor Receptor 2 (HER2) gene in the tumor tissue lesion area.

[0035] In operation S120, extract morphological features from the tumor tissue lesion area to obtain morphological cue features.

[0036] According to an embodiment of the present disclosure, use a statistical algorithm to extract morphological features from the tumor tissue lesion area to obtain morphological cue features.

[0037] According to an embodiment of the present disclosure, the morphological cue features characterize information such as the size, shape, and boundary characteristics of cells or tissues in the tumor tissue lesion area.

[0038] In operation S130, extract visual features from the tissue section image to obtain visual cue features.

[0039] According to an embodiment of the present disclosure, the gray-level co-occurrence matrix algorithm or the wavelet transform algorithm can be used to extract visual features from the tissue section image to obtain visual cue features.

[0040] According to an embodiment of the present disclosure, the visual cue features characterize information such as the gray-level relationship between pixels, pixel color distribution, and intensity in the tissue section image.

[0041] In operation S140, based on the adaptive weight parameter, the morphological cue feature, and the visual cue feature, the detection model is used to process the mask image patches corresponding to the respective multiple tissue section image patches, and the recognition result corresponding to the tissue section image is obtained.

[0042] According to an embodiment of the present disclosure, preprocessing the tissue section image includes: cutting the tissue section image based on a preset pixel size to obtain tissue section image patches, first removing the blurred or distorted tissue section image patches, and then performing quality inspection through a clarity evaluation index (such as gradient intensity or contrast).

[0043] According to an embodiment of the present disclosure, the adaptive weight parameter is determined according to the optimality of the detection model during the training process of the detection model, and the adaptive weight parameter is used to adjust the balanced distribution among the morphological cue feature, the visual cue feature, and the mask image patch.

[0044] According to an embodiment of the present disclosure, a high-dimensional feature vector is formed according to the adaptive weight parameter, the morphological cue feature, and the visual cue feature to capture the significant features related to the HER2 gene expression in the tissue section image.

[0045] According to an embodiment of the present disclosure, each tissue section image patch is masked to obtain a mask image patch, and the mask image patch corresponding to each tissue section image patch is used to distinguish the tumor tissue lesion area and the non-tumor tissue lesion area in this tissue section image patch.

[0046] According to an embodiment of the present disclosure, the data augmentation process for the mask image patch may include: performing geometric transformations such as rotation, flipping, translation, and scaling; performing color adjustments such as brightness, contrast, and saturation; and normalizing processing to map the pixel values of the mask image patch to a unified scale ([0,1]).

[0047] According to an embodiment of the present disclosure, the adaptive weight parameter, the morphological cue feature, the visual cue feature, and multiple enhanced mask image patches are simultaneously input into the detection model, and the recognition result corresponding to the tissue section image is output.

[0048] According to an embodiment of the present disclosure, the recognition result characterizes the expression degree category of the epidermal growth factor receptor 2 gene in the predicted tumor tissue lesion area. For example, the negative category or the positive category.

[0049] According to an embodiment of the present disclosure, by extracting morphological cue features and signal cue features, significant feature distributions related to HER2 gene expression in tissue section images are captured based on detailed and high-dimensional feature extraction. The detection model receives masked image patches and, at the same time, transfers the adaptive weight parameter, morphological cue features, and the signal cue features as additional information to the model, enabling the model to learn not only the features directly presented in the image but also the biological features of the image pixels, and assisting in the targeted adjustment of image features, effectively detecting and classifying HER2 gene expression, improving the accuracy and efficiency of the classification of the detection model, and providing an important clinical basis for assisting doctors in the targeted treatment of gastric cancer.

[0050] According to an embodiment of the present disclosure, the detection model includes N layers of attention mechanisms. Among them, based on the adaptive weight parameter, morphological cue features, and visual cue features, using the detection model to process the masked image patches corresponding to multiple tissue section image patches, the recognition result corresponding to the tissue section image obtained includes: splicing the morphological cue features and visual cue features to obtain spliced cue features; inputting the nth self-attention result and the spliced cue features into the (n + 1)th layer of attention mechanism to obtain the (n + 1)th self-attention result. Among them, when n is equal to 1, the 1st self-attention result is obtained by performing self-attention mechanism learning on the adaptive weight parameter, spliced cue features, and the masked image patches corresponding to multiple tissue section image patches. 1 n N - 1; performing classification processing on the Nth self-attention result to obtain the recognition result corresponding to the tissue section image.

[0051] According to an embodiment of the present disclosure, the detection model can be constructed based on an encoder network (Transformer). The encoder network includes 12 encoder layers, a hidden layer of 768 dimensions, and 12 layers of self-attention heads. 12 layers of attention mechanisms are embedded in the detection model.

[0052] According to an embodiment of the present disclosure, the morphological cue features and visual cue features are spliced to obtain spliced cue features. The spliced cue features can serve as a kind of "cue" to help the detection model pay more attention to the morphological features of tumor lesion cells and guide the detection model to more accurately identify information such as the boundary, pixel texture, and gray scale between the tumor tissue lesion area and the normal tissue area.

[0053] According to an embodiment of the present disclosure, when n is equal to 1, the adaptive weight parameter, spliced cue features, and the masked image patches corresponding to multiple tissue section image patches are input together into the 1st layer of attention mechanism for self-attention mechanism learning to obtain the 1st self-attention result. The size of the masked image patch can be 224x224.

[0054] According to an embodiment of the present disclosure, when n is equal to 2, the first self-attention result and the spliced prompt feature are input into the second-layer attention mechanism together to obtain the second self-attention result; the nth self-attention result and the spliced prompt feature are input into the (n + 1)th layer attention mechanism to obtain the (n + 1)th self-attention result.

[0055] According to an embodiment of the present disclosure, starting from the second-layer attention mechanism, the spliced prompt feature is introduced separately each time to help each layer of the attention mechanism fully and adaptively adjust to learn biological features.

[0056] According to an embodiment of the present disclosure, the self-attention result output by each layer of the attention mechanism represents multi-scale features.

[0057] According to an embodiment of the present disclosure, the Nth layer attention mechanism is connected to the fully connected layer and the activation function. The Nth self-attention result is first input into the fully connected layer to obtain the fully connected feature, and then the fully connected feature is input into the activation function for class probability distribution prediction to obtain the recognition result corresponding to the tissue slice image.

[0058] Figure 2 The flowchart shows the determination of the (n + 1)th self-attention result according to an embodiment of the present disclosure.

[0059] As Figure 2 shown, the process of determining the (n + 1)th self-attention result includes operation S210 to operation S240.

[0060] In operation S210, the (n + 1)th query vector, the (n + 1)th key vector, and the (n + 1)th value vector are obtained respectively according to the product between the nth self-attention result and the (n + 1)th query weight parameter, the (n + 1)th key weight parameter, and the (n + 1)th value weight parameter.

[0061] In operation S220, the spliced prompt feature and the (n + 1)th key vector are spliced to obtain the (n + 1)th key spliced vector.

[0062] In operation S230, the spliced prompt feature and the (n + 1)th value vector are spliced to obtain the (n + 1)th value spliced vector.

[0063] In operation S240, the (n + 1)th self-attention result is determined according to the (n + 1)th query vector, the (n + 1)th key spliced vector, and the (n + 1)th value spliced vector.

[0064] According to an embodiment of the present disclosure, each layer of the attention mechanism has independent weight parameters, such as query weight parameters, key weight parameters, and value weight parameters, and the weight parameters are independently learned during the training process.

[0065] According to an embodiment of the present disclosure, for the (n + 1)-th layer attention mechanism, the (n + 1)-th layer attention mechanism includes the (n + 1)-th query weight parameter, the (n + 1)-th key weight parameter, and the (n + 1)-th value weight parameter.

[0066] According to an embodiment of the present disclosure, the (n + 1)-th query vector is obtained based on the product between the n-th self-attention result and the (n + 1)-th query weight parameter; the (n + 1)-th value vector is obtained based on the product between the n-th self-attention result and the (n + 1)-th value weight parameter; the (n + 1)-th key vector is obtained based on the product between the n-th self-attention result and the (n + 1)-th key weight parameter.

[0067] According to an embodiment of the present disclosure, concatenated prompt features are introduced into each layer of the attention mechanism to enhance the interaction of multi-modal information.

[0068] According to an embodiment of the present disclosure, the concatenated prompt features and the (n + 1)-th key vector are concatenated to obtain the (n + 1)-th key concatenated vector; the concatenated prompt features and the (n + 1)-th value vector are concatenated to obtain the (n + 1)-th value concatenated vector.

[0069] According to an embodiment of the present disclosure, the (n + 1)-th self-attention result is obtained by processing the (n + 1)-th query vector, the (n + 1)-th key concatenated vector, and the (n + 1)-th value concatenated vector using an activation function.

[0070] In one embodiment, the (n + 1)-th self-attention result is obtained As shown in formula (1):

[0071] (1);

[0072] where represents the concatenated prompt features, represents the (n + 1)-th key vector, represents the (n + 1)-th value vector, represents the (n + 1)-th query vector, represents the vector element number parameter, represents the activation function, represents the (n + 1)-th value concatenated vector, represents the (n + 1)-th key concatenated vector.

[0073] According to an embodiment of the present disclosure, multi-modal prompt features such as masked image patches, morphological prompt features, visual prompt features, and adaptive weight parameters are input into the detection model. The "multi-modal prompt features" refer to the combination of different modalities (such as masked image patches and multi-level feature data) to improve the learning ability of the detection model for complex tasks.

[0074] According to an embodiment of the present disclosure, by introducing concatenated prompt features in each layer of the attention mechanism based on the multimodal learning framework, the interaction of multimodal information is enhanced; the concatenated prompt features are passed to the detection model as additional information, so that the detection model can, while learning the features directly presented in the image, also understand the biological features of the image pixels and adjust the learned image features accordingly, improving the classification accuracy of the detection model.

[0075] According to an embodiment of the present disclosure, the first layer of the attention mechanism includes a first query weight parameter, a first key weight parameter, and a first value weight parameter; wherein, performing self-attention mechanism learning on the adaptive weight parameter, the concatenated prompt feature, and the mask image blocks corresponding to the respective multiple tissue section image blocks to obtain the first self-attention result includes: encoding the adaptive weight parameter, the concatenated prompt feature, and the mask image blocks corresponding to the respective multiple tissue section image blocks to obtain an initial embedding vector; respectively obtaining a first query vector, a first key vector, and a first value vector according to the products of the initial embedding vector and the first query weight parameter, the first key weight parameter, and the first value weight parameter; and determining the first self-attention result according to the first query vector, the first key vector, and the first value vector.

[0076] According to an embodiment of the present disclosure, encoding the adaptive weight parameter, the concatenated prompt feature, and the mask image blocks corresponding to the respective multiple tissue section image blocks to obtain an initial embedding vector, which integrates the adjusted morphological information, visual information, and image information.

[0077] According to an embodiment of the present disclosure, the first layer of the attention mechanism includes an adaptively learned first query weight parameter, a first key weight parameter, and a first value weight parameter. The first query weight parameter, the first key weight parameter, and the first value weight parameter are respectively used to project the initial embedding vector into sequential query, key, and value components.

[0078] According to an embodiment of the present disclosure, obtaining a first query vector according to the product of the initial embedding vector and the first query weight parameter; obtaining a first key vector according to the product of the initial embedding vector and the first key weight parameter; and obtaining a first value vector according to the product of the initial embedding vector and the first value weight parameter.

[0079] According to an embodiment of the present disclosure, processing the first query vector, the first key vector, and the first value vector using an activation function to obtain the first self-attention result.

[0080] According to an embodiment of the present disclosure, morphological feature extraction is performed on the lesion area of the tumor tissue, and the morphological cue features obtained include: determining the area value of the lesion area of the tumor tissue according to the number of pixel points in the lesion area of the tumor tissue; determining the perimeter value of the lesion area of the tumor tissue according to the distance between every two adjacent pixel points on the boundary of the lesion area of the tumor tissue; determining the circularity of the lesion area of the tumor tissue according to the area value and the perimeter value; determining the aspect ratio of the lesion area of the tumor tissue according to the major axis length and the minor axis length of each cell in the lesion area of the tumor tissue; splicing the area value, the perimeter value, the circularity, and the aspect ratio to obtain the morphological cue features.

[0081] In one embodiment, the area value A is as shown in formula (2):

[0082] (2);

[0083] Wherein, represents the i-th pixel point.

[0084] In one embodiment, the perimeter value B is as shown in formula (3):

[0085] (3);

[0086] Wherein, M represents the number of pixel points on the boundary of the lesion area of the tumor tissue, represents the abscissa of the m-th pixel point, represents the ordinate of the m-th pixel point, represents the abscissa of the (m + 1)-th pixel point, represents the ordinate of the (m + 1)-th pixel point, and the m-th pixel point is adjacent to the (m + 1)-th pixel point.

[0087] According to an embodiment of the present disclosure, the circularity represents the curvature of the lesion area of the tumor tissue, and the circularity ranges from , and the closer the circularity is to 1, the closer the shape of the lesion area of the tumor tissue is to a circle.

[0088] In one embodiment, the circularity C is as shown in formula (4):

[0089] (4);

[0090] Wherein, A represents the area value of the lesion area of the tumor tissue, and B represents the perimeter value of the lesion area of the tumor tissue.

[0091] According to an embodiment of the present disclosure, the cell is in an elliptical shape, and each cell has a major axis and a minor axis.

[0092] In one embodiment, the aspect ratio AR is as shown in formula (5):

[0093] (5);

[0094] Wherein, represents the major axis length of the h-th cell, represents the minor axis length of the h-th cell.

[0095] According to an embodiment of the present disclosure, according to the ratio of the major axis length to the minor axis length of each cell, the aspect ratio of the cell is obtained; according to the sum of the aspect ratios of multiple cells in the diseased area of the tumor tissue, the aspect ratio of the diseased area of the tumor tissue is obtained.

[0096] According to an embodiment of the present disclosure, the area value, perimeter value, circularity, and aspect ratio are spliced to obtain a one-dimensional morphological cue feature.

[0097] According to an embodiment of the present disclosure, visual feature extraction is performed on the tissue section image to obtain visual cue features, including: determining a preset gray-scale pair condition according to a preset first gray-scale value and a preset second gray-scale value; determining a plurality of initial gray-scale pairs according to the gray-scale values of each two adjacent pixel points in the tissue section image; determining a target gray-scale pair from the plurality of initial gray-scale pairs based on the preset gray-scale pair condition; determining a joint probability according to the number of target gray-scale pairs and the number of initial gray-scale pairs; and obtaining visual cue features according to the joint probability, pixel hue value, and pixel brightness value.

[0098] According to an embodiment of the present disclosure, both the preset first gray-scale value and the preset second gray-scale value are gray-scale value conditions that pixels in the preset tissue section image need to satisfy.

[0099] For example, the preset first gray-scale value is 5, the preset second gray-scale value is 6, and the preset gray-scale pair condition (5, 6) is determined.

[0100] According to an embodiment of the present disclosure, the tissue section image includes a plurality of pixel points, and an initial gray-scale pair corresponding to two adjacent pixel points is determined according to the gray-scale values of each two adjacent pixel points. For example, if the gray-scale values of adjacent pixel point 1 and pixel point 2 are 5 and 6 respectively, the initial gray-scale pair corresponding to pixel point 1 and pixel point 2 is determined as (5, 6).

[0101] According to an embodiment of the present disclosure, a plurality of initial gray-scale pairs are matched with the preset gray-scale pair condition, and the initial gray-scale pairs that meet the preset gray-scale pair condition are determined as target gray-scale pairs.

[0102] For example, taking the above pixel point 1, pixel point 2, and preset gray-scale pair condition (5, 6) as an example, the initial gray-scale pair corresponding to pixel point 1 and pixel point 2 is (5, 6), which meets the preset gray-scale pair condition. Therefore, the initial gray-scale pair corresponding to pixel point 1 and pixel point 2 is determined as the target gray-scale pair.

[0103] According to an embodiment of the present disclosure, a joint probability is obtained based on the ratio of the number of target gray-scale pairs to the number of initial gray-scale pairs, and the joint probability characterizes the frequency of occurrence of two pixel points in a tissue section image under a spatially related condition (such as an adjacent relationship) specified by the target.

[0104] According to an embodiment of the present disclosure, information such as the contrast, energy, entropy, and uniformity of the tissue section image is determined based on the joint probability.

[0105] According to an embodiment of the present disclosure, the contrast reflects the change intensity of the image gray scale; the energy measures the texture uniformity of the image; the entropy represents the randomness of the image; and the uniformity measures the degree of uniformity of the gray-scale distribution.

[0106] In one embodiment, the contrast is as shown in formula (6):

[0107] (6);

[0108] Wherein, represents the joint probability satisfying the preset gray-scale pair condition (k, l), k represents the preset first gray value, and l represents the preset second gray value.

[0109] In one embodiment, the energy is as shown in formula (7):

[0110] (7);

[0111] Wherein, represents the joint probability satisfying the preset gray-scale pair condition (k, l).

[0112] In one embodiment, the entropy is as shown in formula (8):

[0113] (8);

[0114] Wherein, represents the joint probability satisfying the preset gray-scale pair condition (k, l).

[0115] In one embodiment, the uniformity is as shown in formula (9):

[0116] (9);

[0117] Wherein, represents the joint probability satisfying the preset gray-scale pair condition (k, l).

[0118] According to an embodiment of the present disclosure, the pixel hue value includes the average hue and the chromaticity standard deviation.

[0119] In one embodiment, the average hue is as shown in formula (10):

[0120] (10);

[0121] Wherein, represents the hue of the g-th pixel point in the tissue section image, and G represents the number of pixel points in the tissue section image.

[0122] In one embodiment, the chromaticity standard deviation is as shown in formula (11):

[0123] (11);

[0124] Wherein, represents the saturation of the g-th pixel point in the tissue section image, represents the average value of the pixel saturations in the tissue section image.

[0125] In one embodiment, the pixel brightness value is as shown in formula (12):

[0126] (12);

[0127] Wherein, represents the brightness value of the g-th pixel point in the tissue section image.

[0128] According to an embodiment of the present disclosure, the contrast, energy, entropy, uniformity, average hue, chromaticity standard deviation, and pixel brightness value are stitched together to obtain a one-dimensional visual cue feature.

[0129] According to an embodiment of the present disclosure, the mask image blocks corresponding to the multiple tissue section image blocks are obtained based on the following operations: processing the tissue section image using a threshold segmentation algorithm to obtain an initial mask image; performing morphological processing on the initial mask image to obtain a target mask image; and obtaining the mask image blocks corresponding to the multiple tissue section image blocks according to the target mask image and the multiple tissue section image blocks.

[0130] According to an embodiment of the present disclosure, the tissue section image is first subjected to color correction processing and noise reduction processing to obtain a preprocessed tissue section image.

[0131] According to an embodiment of the present disclosure, the color correction processing may include: correcting the color drift of the stain in the tissue section image based on a color deconvolution algorithm to ensure that all tissue section images have a consistent color distribution.

[0132] According to an embodiment of the present disclosure, the noise reduction processing may include: using algorithms such as Gaussian filtering or non-local means filtering to remove the noise in the image while retaining the key tissue details to ensure the clarity of the image.

[0133] According to an embodiment of the present disclosure, the preprocessed tissue section image is processed using a threshold segmentation algorithm to obtain an initial mask image. The initial mask image is used to distinguish the tumor tissue lesion area from the non-tumor tissue lesion area.

[0134] According to an embodiment of the present disclosure, the morphological processing of the initial mask image may include: first performing erosion, dilation, and closing operations on the initial mask image, then removing noise points and connecting broken regions to ensure regional connectivity and integrity, to obtain a target mask image.

[0135] According to an embodiment of the present disclosure, each of multiple tissue section image blocks is multiplied pixel by pixel with the target mask image to obtain a corresponding mask image block for each of the multiple tissue section image blocks. The mask image block can distinguish the tumor tissue lesion area from the non-tumor tissue lesion area on this tissue section image block.

[0136] According to an embodiment of the present disclosure, the mask image block contains rich key information, such as information on the boundaries of diseased tissues, the distribution of cell nuclei, etc., while avoiding as much as possible excessive background or irrelevant regional tissues.

[0137] According to an embodiment of the present disclosure, the detection model is trained based on the following operations: obtaining training samples, where the training samples include sample tissue section images, sample mask image blocks, and sample labels for the gastric region of the sample target object, and the sample labels characterize the expression degree categories of the epidermal growth factor receptor 2 gene in the tumor tissue lesion area of the sample tissue section image; extracting features from the sample tissue section image to obtain sample morphological cue features and sample visual cue features; based on the adaptive weight parameters, sample morphological cue features, and sample visual cue features, using the detection model to process multiple sample mask image blocks to obtain corresponding sample prediction image blocks for the sample mask image blocks and corresponding sample recognition results for the sample prediction image blocks; training the detection model according to the sample mask image blocks, sample prediction image blocks, sample recognition results, and sample labels to obtain the trained detection model.

[0138] According to an embodiment of the present disclosure, the sample target object is a sample object with tumor lesion cells in the gastric region. The sample tissue section image is obtained from a database platform, and the sample mask image block is obtained by processing the sample tissue section image. The sample label characterizes the expression degree category of the epidermal growth factor receptor 2 gene in the tumor tissue lesion area of the sample tissue section image.

[0139] According to an embodiment of the present disclosure, the sample morphological cue features characterize information such as the size, shape, and boundary characteristics of cells or tissues in the tumor tissue lesion area; the sample visual cue features characterize information such as the gray-scale relationship between pixels, pixel color distribution, and intensity in the sample tissue section image.

[0140] According to an embodiment of the present disclosure, the initial adaptive weight parameter is a learnable parameter, which is adaptively adjusted by detecting the training effect of the detection model, and finally an accurate and unchanging adaptive weight parameter is obtained after the training of the detection model ends.

[0141] According to an embodiment of the present disclosure, the initial adaptive weight parameter, the sample morphological cue feature and the sample visual cue feature, and a plurality of sample mask image patches are input into the detection module to obtain a sample prediction image patch corresponding to the sample mask image patch and a sample recognition result corresponding to the sample prediction image patch.

[0142] According to an embodiment of the present disclosure, there may be a local tumor tissue lesion area on each sample mask image patch, and the sample prediction image patch is an image patch reconstructed from the sample mask image patch.

[0143] In one embodiment, the sample prediction image patch is generated as shown in formula (13):

[0144] (13);

[0145] Wherein, represents the e-th sample mask image patch, represents the sample prediction image patch corresponding to the e-th sample mask image patch, represents the sample recognition result corresponding to the e-th sample prediction image patch, represents the sample morphological cue feature and the sample visual cue feature, represents the adaptive weight parameter.

[0146] According to an embodiment of the present disclosure, the sample recognition result corresponding to the sample prediction image patch represents the expression degree category of the epidermal growth factor receptor 2 gene predicted for this sample prediction image patch.

[0147] According to an embodiment of the present disclosure, the loss value between the sample mask image patch and the sample prediction image patch is calculated; the loss value between the sample recognition result and the sample label is calculated, and when the loss value does not meet the preset loss condition, the parameters of the detection model are adjusted; until the loss value of the detection model meets the preset loss condition, the training is stopped.

[0148] According to an embodiment of the present disclosure, training a detection model based on a sample mask image patch, a sample prediction image patch, a sample recognition result, and a sample label to obtain a trained detection model includes: obtaining a sample recognition result corresponding to a sample tissue section image according to the sample recognition results corresponding to the plurality of sample prediction image patches respectively; calculating a loss value between the sample recognition result corresponding to the sample tissue section image and the sample label by using a cross-entropy loss function to obtain a classification loss value; calculating a loss value between the sample mask image patch and the sample prediction image patch by using a perceptual loss function to obtain a perceptual loss value; obtaining an objective loss value according to the classification loss value and the perceptual loss value; and training the detection model according to the objective loss value to obtain a trained detection model.

[0149] According to an embodiment of the present disclosure, in the case where there is no positive recognition result in the sample recognition results corresponding to the plurality of sample prediction image patches respectively, it is obtained that the sample recognition result corresponding to the sample tissue section image is a negative recognition result; otherwise, it is obtained that the sample recognition result corresponding to the sample tissue section image is a positive recognition result.

[0150] In one embodiment, the classification loss value is as shown in formula (14):

[0151] (14);

[0152] wherein, represents the sample label, represents the sample recognition result corresponding to the sample tissue section image, represents the logarithmic function.

[0153] According to an embodiment of the present disclosure, the objective of the cross-entropy loss function is to maximize the classification accuracy, and its parameters are adjusted by the backpropagation algorithm to improve the sensitivity and specificity of HER2 gene detection.

[0154] In one embodiment, the perceptual loss value is as shown in formula (15):

[0155] (15);

[0156] wherein, represents the e-th sample mask image patch, represents the sample prediction image patch corresponding to the e-th sample mask image patch, and E represents the expectation function.

[0157] According to an embodiment of the present disclosure, the perceptual loss function guides the detection model to recover an image through the latent feature space by minimizing the loss, so as to achieve feature distribution perception. The perceptual loss function optimizes the model's learning ability for images and manual features, reconstructs the feature distributions of different classes during the training process, and ensures that the model can effectively distinguish features between different types of tissues. For example, the morphological differences between tumor lesion tissues and normal tissues, as well as the differences between cancer cells with different HER2 expressions, can be more accurately captured through this optimization task.

[0158] In one embodiment, the target loss value L is shown in formula (16):

[0159] (16);

[0160] where, represents a weight factor, such as taking the value of 0.1.

[0161] According to an embodiment of the present disclosure, when the target loss value does not meet the preset loss condition, the parameters of the detection model are adjusted; until the target loss value meets the preset loss condition, the training is stopped.

[0162] According to an embodiment of the present disclosure, through the joint optimization of the perceptual loss and the cross-entropy loss, the trained detection model can not only effectively extract and fuse information from different modalities, but also adaptively adjust through a reinforcement learning strategy during the training process to improve the accuracy of its prediction of HER2 gene expression; in addition, it helps to improve the generalization ability and robustness of the detection model when facing complex gastric cancer pathological image data.

[0163] Figure 3 Shows an example schematic diagram of the training process of the detection model according to an embodiment of the present disclosure.

[0164] As Figure 3 shown, training samples are obtained. The training samples include sample tissue section images and sample labels for the gastric region of the sample target object. The sample tissue section images are subjected to image segmentation and masking processing to obtain sample mask image patches; based on the adaptive weight parameters, sample morphological cue features, and sample visual cue features, the detection model is used to process multiple sample mask image patches to obtain sample prediction image patches corresponding to the sample mask image patches and sample recognition results corresponding to the sample prediction image patches; the cross-entropy loss function is used to calculate the loss value between the sample recognition result corresponding to the sample tissue section image and the sample label to obtain the classification loss value; the perceptual loss function is used to calculate the loss value between the sample mask image patch and the sample prediction image patch to obtain the perceptual loss value; according to the classification loss value and the perceptual loss value, the target loss value is obtained; according to the target loss value, the detection model is trained to obtain the trained detection model.

[0165] Based on the above pathological image classification method, the present disclosure also provides a pathological image classification device. The following will combine Figure 4 to describe this device in detail.

[0166] Figure 4 The structural block diagram of the pathological image classification device according to an embodiment of the present disclosure is shown.

[0167] As Figure 4 shown, the pathological image classification device 400 of this embodiment includes an acquisition module 410, a first feature extraction module 420, a second feature extraction module 430, and a detection module 440.

[0168] The acquisition module 410 is configured to acquire a tissue section image of the gastric region of a target object, wherein the tissue section image is marked with a tumor tissue lesion region. In one embodiment, the acquisition module 410 may be configured to perform the operation S110 described above, which will not be elaborated here.

[0169] The first feature extraction module 420 is configured to extract morphological features from the tumor tissue lesion region to obtain morphological cue features. In one embodiment, the first feature extraction module 420 may be configured to perform the operation S120 described above, which will not be elaborated here.

[0170] The second feature extraction module 430 is configured to extract visual features from the tissue section image to obtain visual cue features. In one embodiment, the second feature extraction module 430 may be configured to perform the operation S130 described above, which will not be elaborated here.

[0171] The detection module 440 is configured to, based on an adaptive weight parameter, morphological cue features, and visual cue features, process mask image patches corresponding to multiple tissue section image patches by using a detection model to obtain an identification result corresponding to the tissue section image, wherein the tissue section image patches are obtained by cutting the tissue section image, the adaptive weight parameter is used to adjust the balanced distribution among the morphological cue features, visual cue features, and mask image patches, and the identification result characterizes the expression degree category of the epidermal growth factor receptor 2 gene in the predicted tumor tissue lesion region. In one embodiment, the detection module 440 may be configured to perform the operation S140 described above, which will not be elaborated here.

[0172] According to an embodiment of the present disclosure, the detection module 440 includes a first detection sub-module, a second detection sub-module, and a third detection sub-module.

[0173] The first detection sub-module is configured to splice the morphological cue features and visual cue features to obtain spliced cue features.

[0174] The second detection sub-module is configured to input the nth self-attention result and the spliced hint feature into the (n + 1)th layer of attention mechanism to obtain the (n + 1)th self-attention result. Wherein, when n is equal to 1, the 1st self-attention result is obtained by performing self-attention mechanism learning on the adaptive weight parameter, the spliced hint feature, and the mask image blocks corresponding to multiple tissue slice image blocks, 1 n N - 1.

[0175] The third detection sub-module is configured to perform classification processing on the Nth self-attention result to obtain the recognition result corresponding to the tissue slice image.

[0176] According to an embodiment of the present disclosure, the second detection sub-module includes a first detection unit, a second detection unit, a third detection unit, and a fourth detection unit.

[0177] The first detection unit is configured to obtain the (n + 1)th query vector, the (n + 1)th key vector, and the (n + 1)th value vector respectively according to the products between the nth self-attention result and the (n + 1)th query weight parameter, the (n + 1)th key weight parameter, and the (n + 1)th value weight parameter.

[0178] The second detection unit is configured to splice the spliced hint feature and the (n + 1)th key vector to obtain the (n + 1)th key spliced vector.

[0179] The third detection unit is configured to splice the spliced hint feature and the (n + 1)th value vector to obtain the (n + 1)th value spliced vector.

[0180] The fourth detection unit is configured to determine the (n + 1)th self-attention result according to the (n + 1)th query vector, the (n + 1)th key spliced vector, and the (n + 1)th value spliced vector.

[0181] According to an embodiment of the present disclosure, the second detection sub-module further includes a fifth detection unit, a sixth detection unit, and a seventh detection unit.

[0182] The fifth detection unit is configured to perform encoding processing on the adaptive weight parameter, the spliced hint feature, and the mask image blocks corresponding to multiple tissue slice image blocks to obtain an initial embedding vector.

[0183] The sixth detection unit is configured to obtain the 1st query vector, the 1st key vector, and the 1st value vector respectively according to the products between the initial embedding vector and the 1st query weight parameter, the 1st key weight parameter, and the 1st value weight parameter.

[0184] The seventh detection unit is configured to determine the 1st self-attention result according to the 1st query vector, the 1st key vector, and the 1st value vector.

[0185] According to an embodiment of the present disclosure, the first feature extraction module 420 includes a first feature extraction sub-module, a second feature extraction sub-module, a third feature extraction sub-module, a fourth feature extraction sub-module, and a fifth feature extraction sub-module.

[0186] The first feature extraction sub-module is configured to determine an area value of the diseased area of the tumor tissue according to the number of pixel points in the diseased area of the tumor tissue.

[0187] The second feature extraction sub-module is configured to determine a perimeter value of the diseased area of the tumor tissue according to the distance between every two adjacent pixel points on the boundary of the diseased area of the tumor tissue.

[0188] The third feature extraction sub-module is configured to determine the circularity of the diseased area of the tumor tissue according to the area value and the perimeter value.

[0189] The fourth feature extraction sub-module is configured to determine the length-width ratio of the diseased area of the tumor tissue according to the major axis length and the minor axis length of each cell in the diseased area of the tumor tissue.

[0190] The fifth feature extraction sub-module is configured to splice the area value, the perimeter value, the circularity, and the length-width ratio to obtain a morphological cue feature.

[0191] According to an embodiment of the present disclosure, the second feature extraction module 430 includes a sixth feature extraction sub-module, a seventh feature extraction sub-module, an eighth feature extraction sub-module, a ninth feature extraction sub-module, and a tenth feature extraction sub-module.

[0192] The sixth feature extraction sub-module is configured to determine a preset gray-scale pair condition according to a preset first gray-scale value and a preset second gray-scale value.

[0193] The seventh feature extraction sub-module is configured to determine a plurality of initial gray-scale pairs according to the gray-scale values of every two adjacent pixel points in the tissue section image.

[0194] The eighth feature extraction sub-module is configured to determine a target gray-scale pair from the plurality of initial gray-scale pairs based on the preset gray-scale pair condition.

[0195] The ninth feature extraction sub-module is configured to determine a joint probability according to the number of target gray-scale pairs and the number of initial gray-scale pairs.

[0196] The tenth feature extraction sub-module is configured to obtain a visual cue feature according to the joint probability, the pixel hue value, and the pixel brightness value.

[0197] According to an embodiment of the present disclosure, the pathological image classification device 400 further includes a sample acquisition module, a sample feature extraction module, a classification module, and a training module.

[0198] A sample acquisition module for acquiring training samples, where the training samples include sample tissue section images, sample mask image patches, and sample labels for the gastric region of the sample target object, and the sample labels represent the expression degree categories of the epidermal growth factor receptor 2 gene in the tumor tissue lesion area in the sample tissue section image.

[0199] A sample feature extraction module for extracting features from the sample tissue section images to obtain sample morphological cue features and sample visual cue features.

[0200] A classification module for processing multiple sample mask image patches using a detection model based on adaptive weight parameters, sample morphological cue features, and sample visual cue features to obtain sample prediction image patches corresponding to the sample mask image patches and sample recognition results corresponding to the sample prediction image patches.

[0201] A training module for training the detection model according to the sample mask image patches, sample prediction image patches, sample recognition results, and sample labels to obtain a trained detection model.

[0202] According to an embodiment of the present disclosure, the training module includes a first training sub-module, a second training sub-module, a third training sub-module, a fourth training sub-module, and a fifth training sub-module.

[0203] The first training sub-module for obtaining the sample recognition result corresponding to the sample tissue section image according to the sample recognition results corresponding to the multiple sample prediction image patches.

[0204] The second training sub-module for calculating the loss value between the sample recognition result corresponding to the sample tissue section image and the sample label using a cross-entropy loss function to obtain a classification loss value.

[0205] The third training sub-module for calculating the loss value between the sample mask image patch and the sample prediction image patch using a perceptual loss function to obtain a perceptual loss value.

[0206] The fourth training sub-module for obtaining a target loss value according to the classification loss value and the perceptual loss value.

[0207] The fifth training sub-module for training the detection model according to the target loss value to obtain a trained detection model.

[0208] According to embodiments of the present disclosure, any number of modules among modules, sub-modules, units, and sub-units can be combined and implemented in one module, or any one of them can be split into multiple modules. Or, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the modules, sub-modules, units, and sub-units can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or can be implemented by any other reasonable means of integrating or packaging circuits, etc., in hardware or firmware, or can be implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Or, at least one of the modules, sub-modules, units, and sub-units can be at least partially implemented as a computer program module, and when the computer program module is run, it can execute the corresponding functions.

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions. Those skilled in the art can understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0210] The embodiments according to the present disclosure have been described above. However, these embodiments are merely for illustrative purposes and not for limiting the scope of the present disclosure. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A method for classifying pathological images, characterized in that The method includes: Obtaining a tissue section image of the gastric region of a target object, wherein the tissue section image is marked with a tumor tissue lesion area; Extracting morphological features of the tumor tissue lesion area to obtain morphological cue features; Extracting visual features of the tissue section image to obtain visual cue features; Based on an adaptive weight parameter, the morphological cue features, and the visual cue features, using a detection model to process mask image patches corresponding to multiple tissue section image patches respectively, to obtain an identification result corresponding to the tissue section image, wherein the tissue section image patches are obtained by cutting the tissue section image, and the adaptive weight parameter is used to adjust the balanced distribution among the morphological cue features, the visual cue features, and the mask image patches, and the identification result represents the predicted expression degree category of the epidermal growth factor receptor 2 gene in the tumor tissue lesion area.

2. The method according to claim 1, wherein The detection model includes N layers of attention mechanisms; Among them, the step of using the detection model to process mask image patches corresponding to multiple tissue section image patches respectively based on the adaptive weight parameter, the morphological cue features, and the visual cue features to obtain an identification result corresponding to the tissue section image includes: Concatenating the morphological cue features and the visual cue features to obtain a concatenated cue feature; Input the n-th self-attention result and the spliced prompt feature into the (n + 1)-th layer of the attention mechanism to obtain the (n + 1)-th self-attention result. Among them, when n is equal to 1, the 1st self-attention result is obtained by performing self-attention mechanism learning on the adaptive weight parameter, the spliced prompt feature, and the mask image blocks corresponding to multiple tissue slice image patches. n N - 1; Performing classification processing on the Nth self-attention result to obtain an identification result corresponding to the tissue section image.

3. The method according to claim 2, wherein The (n + 1)th layer of attention mechanism includes the (n + 1)th query weight parameter, the (n + 1)th key weight parameter, and the (n + 1)th value weight parameter; Among them, the step of inputting the nth self-attention result and the concatenated cue feature into the (n + 1)th layer of attention mechanism to obtain the (n + 1)th self-attention result includes: Respectively obtaining the (n + 1)th query vector, the (n + 1)th key vector, and the (n + 1)th value vector according to the products of the nth self-attention result and the (n + 1)th query weight parameter, the (n + 1)th key weight parameter, and the (n + 1)th value weight parameter; Concatenating the concatenated cue feature and the (n + 1)th key vector to obtain the (n + 1)th key concatenated vector; Concatenating the concatenated cue feature and the (n + 1)th value vector to obtain the (n + 1)th value concatenated vector; Determining the (n + 1)th self-attention result according to the (n + 1)th query vector, the (n + 1)th key concatenated vector, and the (n + 1)th value concatenated vector.

4. The method according to claim 2, wherein The first layer of attention mechanism includes the first query weight parameter, the first key weight parameter, and the first value weight parameter; Among them, the step of performing self-attention mechanism learning on the adaptive weight parameter, the concatenated cue feature, and mask image patches corresponding to multiple tissue section image patches respectively to obtain the first self-attention result includes: Performing encoding processing on the adaptive weight parameter, the concatenated cue feature, and mask image patches corresponding to multiple tissue section image patches respectively to obtain an initial embedding vector; Obtain a first query vector, a first key vector, and a first value vector respectively according to the products between the initial embedding vector and the first query weight parameter, the first key weight parameter, and the first value weight parameter. Determine the first self-attention result according to the first query vector, the first key vector, and the first value vector.

5. The method according to claim 1, wherein The morphological feature extraction of the tumor tissue lesion area to obtain morphological cue features includes: Determine the area value of the tumor tissue lesion area according to the number of pixel points in the tumor tissue lesion area. Determine the perimeter value of the tumor tissue lesion area according to the distance between every two adjacent pixel points on the boundary of the tumor tissue lesion area. Determine the circularity of the tumor tissue lesion area according to the area value and the perimeter value. Determine the length-width ratio of the tumor tissue lesion area according to the major axis length and minor axis length of each cell in the tumor tissue lesion area. Concatenate the area value, the perimeter value, the circularity, and the length-width ratio to obtain the morphological cue features.

6. The method according to claim 1, wherein The visual feature extraction of the tissue section image to obtain visual cue features includes: Determine a preset gray-scale pair condition according to a preset first gray-scale value and a preset second gray-scale value. Determine a plurality of initial gray-scale pairs according to the gray-scale values of every two adjacent pixel points in the tissue section image. Determine a target gray-scale pair from the plurality of initial gray-scale pairs based on the preset gray-scale pair condition. Determine a joint probability according to the number of the initial gray-scale pairs and the number of the target gray-scale pairs. Obtain the visual cue features according to the joint probability, the pixel hue value, and the pixel brightness value.

7. The method according to claim 1, wherein The mask image blocks corresponding to the plurality of tissue section image blocks are obtained based on the following operations: Process the tissue section image using a threshold segmentation algorithm to obtain an initial mask image. Perform morphological processing on the initial mask image to obtain a target mask image. Obtain mask image blocks corresponding to the plurality of tissue section image blocks according to the target mask image and the plurality of tissue section image blocks.

8. The method according to claim 1, wherein The detection model is trained based on the following operations: Obtain training samples, where the training samples include sample tissue section images, sample mask image blocks, and sample labels for the gastric region of the sample target object, and the sample labels represent the expression degree categories of the epidermal growth factor receptor 2 gene on the tumor tissue lesion area in the sample tissue section image. Perform feature extraction on the sample tissue section image to obtain sample morphological cue features and sample visual cue features. Based on the adaptive weight parameter, the sample morphological cue features, and the sample visual cue features, use the detection model to process the plurality of sample mask image blocks to obtain sample prediction image blocks corresponding to the sample mask image blocks and sample recognition results corresponding to the sample prediction image blocks. Train the detection model according to the sample mask image blocks, the sample prediction image blocks, the sample recognition results, and the sample labels to obtain the trained detection model.

9. The method according to claim 8, characterized in that, Training the detection model according to the sample mask image block, the sample prediction image block, the sample recognition result, and the sample label to obtain the trained detection model includes: Obtaining the sample recognition result corresponding to the sample tissue section image according to the sample recognition results respectively corresponding to a plurality of the sample prediction image blocks; Calculating the loss value between the sample recognition result corresponding to the sample tissue section image and the sample label by using a cross-entropy loss function to obtain a classification loss value; Calculating the loss value between the sample mask image block and the sample prediction image block by using a perceptual loss function to obtain a perceptual loss value; Obtaining a target loss value according to the classification loss value and the perceptual loss value; Training the detection model according to the target loss value to obtain the trained detection model.

10. A pathological image classification device, characterized in that, The device includes: An acquisition module, configured to acquire a tissue section image of the gastric region of a target object, wherein the tissue section image is marked with a tumor tissue lesion region; A first feature extraction module, configured to perform morphological feature extraction on the tumor tissue lesion region to obtain morphological cue features; A second feature extraction module, configured to perform visual feature extraction on the tissue section image to obtain visual cue features; A detection module, configured to, based on an adaptive weight parameter, the morphological cue features, and the visual cue features, process mask image blocks respectively corresponding to a plurality of tissue section image blocks by using a detection model to obtain the recognition result corresponding to the tissue section image, wherein the tissue section image blocks are obtained by cutting the tissue section image, the adaptive weight parameter is used to adjust the balanced distribution among the morphological cue features, the visual cue features, and the mask image blocks, and the recognition result represents the predicted expression level category of the epidermal growth factor receptor 2 gene in the tumor tissue lesion region.

Images