Automatic visual question-answering method and system for pathology full-field graph, and medium

Abstract

The invention provides an automatic visual question and answer method and system for a pathology full-field graph and a medium. The method comprises the steps that a full-slice image is acquired, the full-slice image is preprocessed, non-information areas are removed, and a foreground area is obtained; segmenting the foreground region into a plurality of low-pixel image units based on a partitioning strategy, extracting visual features of the plurality of low-pixel image units, and performing dimension reduction processing on the plurality of visual features to obtain a visual feature sequence; cue words input by a user are obtained, the cue words input by the user are preprocessed to obtain an input text, and the input text is converted into a text feature sequence; the visual feature sequence and the text feature sequence are interacted based on a cross attention mechanism, probability distribution information of answers is analyzed and predicted, and a question and answer result is generated; by analyzing the full-slice image and the prompt words of the user, image-text interaction is realized, and then corresponding answers are output according to the questions of the user, so that the automatic diagnosis and treatment level of a pathological department is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence generation and medical imaging technology, and more specifically, to an automated visual question-answering method, system, and medium for pathology full-field images. Background Art

[0002] Multimodal large language models (MLLMs), such as GPT-4V, LLaVa, and Qwen-VL, have demonstrated significant superiority across a wide range of vision-language tasks. They also demonstrate exceptional performance in the medical field, which requires specialized knowledge and exhibits significant visual variability. Despite their impressive performance in the medical field, these MLLMs struggle to directly perform visual understanding and image-text interaction on the large size of pathology whole-field images (WSIs).

[0003] In visual question answering (VQA) of natural images, the input image size is approximately 256×256, equivalent to a vocabulary of 16×16 labeled visual descriptions. In the field of pathology, full-field image-level prediction is a challenging problem, often addressed using the multiple instance learning (MIL) framework. Patches are cropped from the entire image and then reduced in dimensionality to make computation feasible. These patches are embedded as individual instances and subsequently aggregated in a weakly supervised manner to produce the final result.

[0004] Vision-to-language models unify multiple vision tasks such as object segmentation, scene understanding, and optical character recognition by carefully constructing appropriate prompts and obtaining textual descriptions. In the field of pathology, previous MIL models specialized for specific tasks can only solve certain problems, such as tumor grading, although they perform well.

[0005] In recent years, pathology MLLMs (such as MI-Zero, PLIP, and CONCH) have also emerged. Although the model has observed a large number of pathology images, the pre-training based on image patches makes it difficult to achieve ideal results in case-level tasks that require understanding the entire full-field image (such as cancer subtype classification). Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide an automated visual question-answering method, system, and medium for full-field pathology images. By analyzing full-slice images and user prompts, image-text interaction is achieved, and then corresponding answers are output based on the user's questions, thereby improving the automated diagnosis and treatment level of the pathology department.

[0007] The present application also provides an automated visual question-answering method for pathology full-field images, including:

[0008] Obtain a full-slice image, preprocess the full-slice image, remove non-information areas, and obtain a foreground area;

[0009] Based on the partitioning strategy, the foreground area is divided into multiple low-pixel image units, and the visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence;

[0010] Obtaining the prompt word input by the user, preprocessing the prompt word input by the user to obtain the input text, and converting the input text into a text feature sequence;

[0011] Based on the cross-attention mechanism, the visual feature sequence and the text feature sequence interact to obtain interactive information;

[0012] The probability distribution information of the predicted answer is analyzed based on the interactive information, and the question-answering results are generated based on the probability distribution information.

[0013] Optionally, in the automated visual question-answering method for pathology full-field images described in the embodiments of the present application, a full-slice image is obtained, and the full-slice image is preprocessed to remove non-information areas to obtain a foreground area, specifically including:

[0014] Acquire ultra-high-resolution full-field pathology images to obtain full-slice images, and adjust the resolution of the full-slice images to obtain low-resolution full-field image thumbnails;

[0015] Convert the full-field image thumbnail from RGB color space to HSV color space to obtain an HSV image;

[0016] Set the saturation threshold, divide the HSV image into regions, generate several sub-regions, and analyze the saturation of each sub-region;

[0017] Compare the saturation of each sub-region with the set saturation to obtain the analysis results;

[0018] Based on the analysis results, the background area is screened out to obtain the non-information area, and the non-information area is eliminated to obtain the foreground area.

[0019] Optionally, in the automated visual question answering method for pathology full-field images described in an embodiment of the present application, the foreground area is divided into multiple low-pixel image units based on a partitioning strategy, and visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence, specifically including:

[0020] Setting sliding windows based on adaptive grid partitioning strategy;

[0021] Crop multiple non-overlapping image blocks from the foreground area based on a sliding window;

[0022] Generate multiple low-pixel image units based on multiple image blocks of different stacks to obtain multiple image slices;

[0023] The ViT network is used to reduce the dimension of the image slices to obtain the dimension reduction result, which is then converted into a visual feature sequence.

[0024] Optionally, in the automated visual question answering method for pathology full-field images described in the embodiment of the present application, obtaining a prompt word input by a user, preprocessing the prompt word input by the user to obtain input text, and converting the input text into a text feature sequence specifically includes:

[0025] Obtain the prompt word input by the user, generate input text, clean the input text, remove noise and delete irrelevant characters, such as extra spaces, line breaks and HTML tags;

[0026] Obtaining the text format of the input text, standardizing the text format to obtain the standard input text, and analyzing the length of the standard input text;

[0027] If the length of the standard input text exceeds the length threshold, it will be truncated;

[0028] If the length of the standard input text does not exceed the length threshold, then add markers to the beginning and end of the standard input text;

[0029] Based on the word segmenter, the standard input text is converted into a token sequence, and the token sequence is mapped into a continuous text feature sequence.

[0030] Optionally, in the automated visual question answering method for pathology full-field images described in the embodiment of the present application, the visual feature sequence and the text feature sequence are interacted based on the cross-attention mechanism to obtain interactive information, specifically including:

[0031] Obtain visual features based on image units, analyze the self-attention relationship of visual features, and establish visual feature sequences;

[0032] Convert visual feature sequences into key vectors and value vectors;

[0033] Obtain a text feature sequence and convert the text feature sequence into a query vector;

[0034] The key vector, value vector and query vector are interacted based on the cross attention mechanism to obtain interactive information.

[0035] Optionally, in the automated visual question-answering method for pathology full-field images described in the embodiment of the present application, probability distribution information of the predicted answer is analyzed based on the interactive information, and question-answering results are generated based on the probability distribution information, specifically including:

[0036] Interact based on visual features and text features to obtain interactive information;

[0037] Build a prediction model, output the predicted answer based on the prediction model, and analyze the probability distribution information of the predicted answer;

[0038] Calculate the cross entropy loss function between the probability distribution of the prediction model output and the standard answer based on the probability distribution information of the predicted answer;

[0039] Back propagation is performed based on the cross entropy loss function to optimize the parameters of the prediction model until the prediction model converges;

[0040] The full slice image and the prompt word are input into the prediction model, and the prediction result of the answer is output to obtain the question-answering result.

[0041] In a second aspect, an embodiment of the present application provides an automated visual question-answering system for pathology full-field images, the system comprising: a memory and a processor, the memory comprising a program for an automated visual question-answering method for pathology full-field images, the program for the automated visual question-answering method for pathology full-field images, when executed by the processor, implementing the following steps:

[0042] Obtain a full-slice image, preprocess the full-slice image, remove non-information areas, and obtain a foreground area;

[0043] Based on the partitioning strategy, the foreground area is divided into multiple low-pixel image units, and the visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence;

[0044] Obtaining the prompt word input by the user, preprocessing the prompt word input by the user to obtain the input text, and converting the input text into a text feature sequence;

[0045] Based on the cross-attention mechanism, the visual feature sequence and the text feature sequence interact to obtain interactive information;

[0046] The probability distribution information of the predicted answer is analyzed based on the interactive information, and the question-answering results are generated based on the probability distribution information.

[0047] Optionally, in the automated visual question-answering system for pathology full-field images described in the embodiment of the present application, a full-slice image is obtained, and the full-slice image is preprocessed to remove non-information areas to obtain a foreground area, specifically including:

[0048] Acquire ultra-high-resolution full-field pathology images to obtain full-slice images, and adjust the resolution of the full-slice images to obtain low-resolution full-field image thumbnails;

[0049] Convert the full-field image thumbnail from RGB color space to HSV color space to obtain an HSV image;

[0050] Set the saturation threshold, divide the HSV image into regions, generate several sub-regions, and analyze the saturation of each sub-region;

[0051] Compare the saturation of each sub-region with the set saturation to obtain the analysis results;

[0052] Based on the analysis results, the background area is screened out to obtain the non-information area, and the non-information area is eliminated to obtain the foreground area.

[0053] Optionally, in the automated visual question-answering system for pathology full-field images described in an embodiment of the present application, the foreground area is divided into multiple low-pixel image units based on a partitioning strategy, and visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence, specifically including:

[0054] Setting sliding windows based on adaptive grid partitioning strategy;

[0055] Crop multiple non-overlapping image blocks from the foreground area based on a sliding window;

[0056] Generate multiple low-pixel image units based on multiple image blocks of different stacks to obtain multiple image slices;

[0057] The ViT network is used to reduce the dimension of the image slices to obtain the dimension reduction result, which is then converted into a visual feature sequence.

[0058] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes an automated visual question-answering method program for pathology full-field images. When the automated visual question-answering method program for pathology full-field images is executed by a processor, the steps of the automated visual question-answering method for pathology full-field images as described in any one of the above items are implemented.

[0059] As can be seen from the above, the embodiments of the present application provide an automated visual question-answering method, system, and medium for full-field pathology images, which obtains a full-slice image, preprocesses the full-slice image, eliminates non-information areas, and obtains a foreground area; based on a partitioning strategy, the foreground area is divided into multiple low-pixel image units, and visual features of the multiple low-pixel image units are extracted, and the multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence; the prompt words input by the user are obtained, the prompt words input by the user are preprocessed to obtain input text, and the input text is converted into a text feature sequence; based on a cross-attention mechanism, the visual feature sequence and the text feature sequence are interacted to obtain interaction information; based on the interaction information, the probability distribution information of the predicted answer is analyzed and predicted, and the question-answering result is generated based on the probability distribution information; by analyzing the full-slice image and the user's prompt words, image-text interaction is achieved, and then the corresponding answer is output according to the user's question, thereby improving the level of automated diagnosis and treatment in the pathology department. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 A flowchart of the automated visual question-answering method for pathology full-field images provided in an embodiment of the present application;

[0062] Figure 2 A network architecture diagram of the automated visual question-answering method for full-field pathology images provided in an embodiment of the present application;

[0063] Figure 3 The left side of the automated visual question-answering method for pathology full-field images provided in the embodiment of the present application is a thumbnail of the pathology full-field image, and the right side is the question, answer and output result of the model. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0065] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0066] Please refer to Figure 1-Figure 3 As shown, the present application discloses an automated visual question-answering method, system, and medium for pathology full-field images. The automated visual question-answering method for pathology full-field images is used in a terminal device and includes the following steps:

[0067] S101, obtaining a full-slice image, preprocessing the full-slice image, removing non-information areas, and obtaining a foreground area;

[0068] S102, dividing the foreground area into multiple low-pixel image units based on a partitioning strategy, extracting visual features of the multiple low-pixel image units, and performing dimensionality reduction processing on the multiple visual features to obtain a visual feature sequence;

[0069] S103, obtaining a prompt word input by the user, preprocessing the prompt word input by the user to obtain an input text, and converting the input text into a text feature sequence;

[0070] S104, interacting the visual feature sequence with the text feature sequence based on the cross attention mechanism to obtain interaction information;

[0071] S105, analyzing and predicting probability distribution information of the answer based on the interactive information, and generating a question-answering result based on the probability distribution information.

[0072] It should be noted that a Transformer encoder is used, and a multi-head attention mechanism is adopted to model the cross-region association of visual feature sequences, and the topological constraints of the tissue structure are retained through spatial relative position encoding; the user's prompt words are preprocessed and converted into text feature sequences; a hybrid decoding architecture with multimodal perception capabilities is used to achieve dynamic alignment of visual and text features through an interactive attention mechanism, in which pathological features participate in spatial attention calculations as value-key matrices, and medical text semantic vectors serve as query matrices to guide feature fusion. Finally, a text answer with visual understanding capabilities and responding to user prompt words is generated autoregressively.

[0073] According to an embodiment of the present invention, a full-slice image is acquired, and preprocessed to remove non-information areas to obtain a foreground area, specifically including:

[0074] Acquire ultra-high-resolution full-field pathology images to obtain full-slice images, and adjust the resolution of the full-slice images to obtain low-resolution full-field image thumbnails;

[0075] Convert the full-field image thumbnail from RGB color space to HSV color space to obtain an HSV image;

[0076] Set the saturation threshold, divide the HSV image into regions, generate several sub-regions, and analyze the saturation of each sub-region;

[0077] Compare the saturation of each sub-region with the set saturation to obtain the analysis results;

[0078] Based on the analysis results, the background area is screened out to obtain the non-information area, and the non-information area is eliminated to obtain the foreground area.

[0079] According to an embodiment of the present invention, the foreground area is divided into multiple low-pixel image units based on a partitioning strategy, and visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence, which specifically includes:

[0080] Setting sliding windows based on adaptive grid partitioning strategy;

[0081] Crop multiple non-overlapping image blocks from the foreground area based on a sliding window;

[0082] Generate multiple low-pixel image units based on multiple image blocks of different stacks to obtain multiple image slices;

[0083] The ViT network is used to reduce the dimension of the image slices to obtain the dimension reduction result, which is then converted into a visual feature sequence.

[0084] It should be noted that the resulting visual feature sequence is input into the Transformer encoder. The Transformer encoder incorporates a self-attention mechanism that captures interactions within the sequence, specifically the spatial information of the full-field pathology image. Each attention layer is followed by residual connections and normalization to prevent gradient explosion and stabilize training, ultimately yielding high-quality visual features. Using a grid-based block strategy and a sampling step of 256 pixels, the input image is parsed into regularized image units of 256×256 pixels. Feature compression is achieved through the visual Transformer architecture, and linear mapping is further used to reduce the dimensionality of each image unit to a 512-dimensional feature vector, constructing a spatially aware visual feature matrix.

[0085] According to an embodiment of the present invention, obtaining a prompt word input by a user, preprocessing the prompt word input by the user to obtain input text, and converting the input text into a text feature sequence specifically includes:

[0086] Get the prompt word entered by the user, generate the input text, clean the input text, remove noise and delete irrelevant characters, including extra spaces, line breaks and HTML tags;

[0087] Obtaining the text format of the input text, standardizing the text format to obtain the standard input text, and analyzing the length of the standard input text;

[0088] If the length of the standard input text exceeds the length threshold, it will be truncated;

[0089] If the length of the standard input text does not exceed the length threshold, then add markers to the beginning and end of the standard input text;

[0090] Based on the word segmenter, the standard input text is converted into a token sequence, and the token sequence is mapped into a continuous text feature sequence.

[0091] It should be noted that the input text is cleaned, including format checking and length control. The cleaned text is processed using a word segmenter to convert it into a token sequence. A clinical knowledge-guided text encoding module, that is, the BioClinicalBERT pre-trained model, is used to perform medical entity recognition and relationship extraction on user input, and natural language is converted into a 768-dimensional medical semantic vector through domain-adapted word vector mapping.

[0092] Furthermore, the length of the input visual feature sequence is controlled. If it is too long, it is truncated; if it is too short, it is padded with zeros to make its length fixed. Relative position encoding is added to the input visual series so that the model can learn the spatial position information of the visual input. An N-layer stacked Transformer architecture is constructed. Each layer realizes dynamic feature weighting through a multi-head attention mechanism, and adaptive focusing of key pathological areas is achieved through learnable attention weights.

[0093] According to an embodiment of the present invention, the visual feature sequence and the text feature sequence are interacted based on the cross attention mechanism to obtain interactive information, which specifically includes:

[0094] Obtain visual features based on image units, analyze the self-attention relationship of visual features, and establish visual feature sequences;

[0095] Convert visual feature sequences into key vectors and value vectors;

[0096] Obtain a text feature sequence and convert the text feature sequence into a query vector;

[0097] The key vector, value vector and query vector are interacted based on the cross attention mechanism to obtain interactive information.

[0098] It should be noted that the text sequence features are used as query vectors and interact in the cross-attention module of the Transformer decoder, so that the model has the ability to understand the full-field pathology image and the user's input instructions, and finally output the corresponding answer.

[0099] According to an embodiment of the present invention, analyzing the probability distribution information of the predicted answer based on the interactive information and generating the question-answer result based on the probability distribution information specifically includes:

[0100] Interact based on visual features and text features to obtain interactive information;

[0101] Build a prediction model, output the predicted answer based on the prediction model, and analyze the probability distribution information of the predicted answer;

[0102] Calculate the cross entropy loss function between the probability distribution of the prediction model output and the standard answer based on the probability distribution information of the predicted answer;

[0103] Back propagation is performed based on the cross entropy loss function to optimize the parameters of the prediction model until the prediction model converges;

[0104] The full slice image and the prompt word are input into the prediction model, and the prediction result of the answer is output to obtain the question-answering result.

[0105] In a second aspect, an embodiment of the present application provides an automated visual question-answering system for pathology full-field images, the system comprising: a memory and a processor, the memory including a program for an automated visual question-answering method for pathology full-field images, and the program for the automated visual question-answering method for pathology full-field images, when executed by the processor, implementing the following steps:

[0106] Obtain a full-slice image, preprocess the full-slice image, remove non-information areas, and obtain a foreground area;

[0107] Based on the partitioning strategy, the foreground area is divided into multiple low-pixel image units, and the visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence;

[0108] Obtaining the prompt word input by the user, preprocessing the prompt word input by the user to obtain the input text, and converting the input text into a text feature sequence;

[0109] Based on the cross-attention mechanism, the visual feature sequence and the text feature sequence interact to obtain interactive information;

[0110] The probability distribution information of the predicted answer is analyzed based on the interactive information, and the question-answering results are generated based on the probability distribution information.

[0111] It should be noted that the system consists of three core modules: the preprocessing module based on the visual Transformer effectively alleviates the resource pressure of ultra-high-resolution medical images on the GPU video memory through block feature extraction and spatial compression mechanism; secondly, the feature encoder based on the Transformer adopts a multi-head attention mechanism to model the spatial correlation between pathological tissue morphological features; finally, the Transformer decoder module based on cross-modal interaction realizes the deep fusion of visual features and semantic information through interactive attention, and adopts a causal mask mechanism to achieve serialized text output that conforms to medical logic.

[0112] Furthermore, a Transformer encoder is used, and a multi-head attention mechanism is adopted to model the cross-region association of visual feature sequences, retaining the topological constraints of the tissue structure through spatial relative position encoding; the user's prompt words are preprocessed and converted into text feature sequences; a hybrid decoding architecture with multimodal perception capabilities is used to achieve dynamic alignment of visual and text features through an interactive attention mechanism, in which pathological features participate in spatial attention calculations as value-key matrices, and medical text semantic vectors serve as query matrices to guide feature fusion. Finally, a text answer with visual understanding capabilities and responding to user prompt words is generated autoregressively.

[0113] According to an embodiment of the present invention, a full-slice image is acquired, and preprocessed to remove non-information areas to obtain a foreground area, specifically including:

[0114] Acquire ultra-high-resolution full-field pathology images to obtain full-slice images, and adjust the resolution of the full-slice images to obtain low-resolution full-field image thumbnails;

[0115] Convert the full-field image thumbnail from RGB color space to HSV color space to obtain an HSV image;

[0116] Set the saturation threshold, divide the HSV image into regions, generate several sub-regions, and analyze the saturation of each sub-region;

[0117] Compare the saturation of each sub-region with the set saturation to obtain the analysis results;

[0118] Based on the analysis results, the background area is screened out to obtain the non-information area, and the non-information area is eliminated to obtain the foreground area.

[0119] According to an embodiment of the present invention, the foreground area is divided into multiple low-pixel image units based on a partitioning strategy, and visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence, which specifically includes:

[0120] Setting sliding windows based on adaptive grid partitioning strategy;

[0121] Crop multiple non-overlapping image blocks from the foreground area based on a sliding window;

[0122] Generate multiple low-pixel image units based on multiple image blocks of different stacks to obtain multiple image slices;

[0123] The ViT network is used to reduce the dimension of the image slices to obtain the dimension reduction result, which is then converted into a visual feature sequence.

[0124] It should be noted that the resulting visual feature sequence is input into the Transformer encoder. The Transformer encoder incorporates a self-attention mechanism that captures interactions within the sequence, specifically the spatial information of the full-field pathology image. Each attention layer is followed by residual connections and normalization to prevent gradient explosion and stabilize training, ultimately yielding high-quality visual features. Using a grid-based block strategy and a sampling step of 256 pixels, the input image is parsed into regularized image units of 256×256 pixels. Feature compression is achieved through the visual Transformer architecture, and linear mapping is further used to reduce the dimensionality of each image unit to a 512-dimensional feature vector, constructing a spatially aware visual feature matrix.

[0125] According to an embodiment of the present invention, obtaining a prompt word input by a user, preprocessing the prompt word input by the user to obtain input text, and converting the input text into a text feature sequence specifically includes:

[0126] Get the prompt word entered by the user, generate the input text, clean the input text, remove noise and delete irrelevant characters, including extra spaces, line breaks and HTML tags;

[0127] Obtaining the text format of the input text, standardizing the text format to obtain the standard input text, and analyzing the length of the standard input text;

[0128] If the length of the standard input text exceeds the length threshold, it will be truncated;

[0129] If the length of the standard input text does not exceed the length threshold, then add markers to the beginning and end of the standard input text;

[0130] Based on the word segmenter, the standard input text is converted into a token sequence, and the token sequence is mapped into a continuous text feature sequence.

[0131] It should be noted that the input text is cleaned, including format checking and length control. The cleaned text is processed using a word segmenter to convert it into a token sequence. A clinical knowledge-guided text encoding module, that is, the BioClinicalBERT pre-trained model, is used to perform medical entity recognition and relationship extraction on user input, and natural language is converted into a 768-dimensional medical semantic vector through domain-adapted word vector mapping.

[0132] Furthermore, the length of the input visual feature sequence is controlled. If it is too long, it is truncated; if it is too short, it is padded with zeros to make its length fixed. Relative position encoding is added to the input visual series so that the model can learn the spatial position information of the visual input. An N-layer stacked Transformer architecture is constructed. Each layer realizes dynamic feature weighting through a multi-head attention mechanism, and adaptive focusing of key pathological areas is achieved through learnable attention weights.

[0133] According to an embodiment of the present invention, the visual feature sequence and the text feature sequence are interacted based on the cross attention mechanism to obtain interactive information, which specifically includes:

[0134] Obtain visual features based on image units, analyze the self-attention relationship of visual features, and establish visual feature sequences;

[0135] Convert visual feature sequences into key vectors and value vectors;

[0136] Obtain a text feature sequence and convert the text feature sequence into a query vector;

[0137] The key vector, value vector and query vector are interacted based on the cross attention mechanism to obtain interactive information.

[0138] It should be noted that the text sequence features are used as query vectors and interact in the cross-attention module of the Transformer decoder, so that the model has the ability to understand the full-field pathology image and the user's input instructions, and finally output the corresponding answer.

[0139] According to an embodiment of the present invention, analyzing the probability distribution information of the predicted answer based on the interactive information and generating the question-answer result based on the probability distribution information specifically includes:

[0140] Interact based on visual features and text features to obtain interactive information;

[0141] Build a prediction model, output the predicted answer based on the prediction model, and analyze the probability distribution information of the predicted answer;

[0142] Calculate the cross entropy loss function between the probability distribution of the prediction model output and the standard answer based on the probability distribution information of the predicted answer;

[0143] Back propagation is performed based on the cross entropy loss function to optimize the parameters of the prediction model until the prediction model converges;

[0144] The full slice image and the prompt word are input into the prediction model, and the prediction result of the answer is output to obtain the question-answering result.

[0145] The third aspect of the present invention provides a computer-readable storage medium, which includes an automated visual question-answering method program for pathology full-field images. When the automated visual question-answering method program for pathology full-field images is executed by a processor, it implements the steps of the automated visual question-answering method for pathology full-field images as described in any one of the above items.

[0146] The present invention discloses an automated visual question-answering method, system, and medium for full-field pathology images. The method comprises the following steps: obtaining a full-slice image, preprocessing the full-slice image, eliminating non-information areas, and obtaining a foreground area; dividing the foreground area into multiple low-pixel image units based on a partitioning strategy, extracting visual features of the multiple low-pixel image units, and performing dimensionality reduction processing on the multiple visual features to obtain a visual feature sequence; obtaining a prompt word input by a user, preprocessing the prompt word input by the user to obtain input text, and converting the input text into a text feature sequence; interacting the visual feature sequence with the text feature sequence based on a cross-attention mechanism to obtain interaction information; analyzing and predicting probability distribution information of an answer based on the interaction information, and generating a question-answering result based on the probability distribution information; and realizing image-text interaction by analyzing the full-slice image and the user's prompt word, thereby outputting corresponding answers according to the user's questions, thereby improving the level of automated diagnosis and treatment in the pathology department.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0148] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0149] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0150] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0151] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the existing technology, can be embodied in the form of a software product. The software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. An automated visual question answering method for full-field pathology images, characterized by: include: Obtain a full-slice image, preprocess the full-slice image, remove non-information areas, and obtain a foreground area; Based on the partitioning strategy, the foreground area is divided into multiple low-pixel image units, and the visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence; Obtaining the prompt word input by the user, preprocessing the prompt word input by the user to obtain the input text, and converting the input text into a text feature sequence; Based on the cross-attention mechanism, the visual feature sequence and the text feature sequence interact to obtain interactive information; The probability distribution information of the predicted answer is analyzed based on the interactive information, and the question-answering results are generated based on the probability distribution information.

2. The automated visual question answering method for pathology full-field images according to claim 1, characterized in that: Obtain a full-slice image and preprocess it to remove non-information areas and obtain the foreground area, including: Acquire ultra-high-resolution full-field pathology images to obtain full-slice images, and adjust the resolution of the full-slice images to obtain low-resolution full-field image thumbnails; Convert the full-field image thumbnail from RGB color space to HSV color space to obtain an HSV image; Set the saturation threshold, divide the HSV image into regions, generate several sub-regions, and analyze the saturation of each sub-region; Compare the saturation of each sub-region with the set saturation to obtain the analysis results; Based on the analysis results, the background area is screened out to obtain the non-information area, and the non-information area is eliminated to obtain the foreground area.

3. The automated visual question answering method for pathology full-field images according to claim 2, characterized in that: Based on the partitioning strategy, the foreground area is divided into multiple low-pixel image units, and the visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence, which specifically includes: Setting sliding windows based on adaptive grid partitioning strategy; Crop multiple non-overlapping image blocks from the foreground area based on a sliding window; Generate multiple low-pixel image units based on multiple image blocks of different stacks to obtain multiple image slices; The ViT network is used to reduce the dimension of the image slices to obtain the dimension reduction result, which is then converted into a visual feature sequence.

4. The automated visual question answering method for pathology full-field images according to claim 3, characterized in that: Obtain the prompt word input by the user, preprocess the prompt word input by the user to obtain the input text, and convert the input text into a text feature sequence, specifically including: Obtain the prompt word input by the user, generate input text, clean the input text, remove noise and delete irrelevant characters, such as extra spaces, line breaks and HTML tags; Obtaining the text format of the input text, standardizing the text format to obtain the standard input text, and analyzing the length of the standard input text; If the length of the standard input text exceeds the length threshold, it will be truncated; If the length of the standard input text does not exceed the length threshold, then add markers to the beginning and end of the standard input text; Based on the word segmenter, the standard input text is converted into a token sequence, and the token sequence is mapped into a continuous text feature sequence.

5. The automated visual question answering method for pathology full-field images according to claim 4, characterized in that: Based on the cross-attention mechanism, the visual feature sequence and the text feature sequence interact to obtain interactive information, including: Obtain visual features based on image units, analyze the self-attention relationship of visual features, and establish visual feature sequences; Convert visual feature sequences into key vectors and value vectors; Obtain a text feature sequence and convert the text feature sequence into a query vector; The key vector, value vector and query vector are interacted based on the cross attention mechanism to obtain interactive information.

6. The automated visual question answering method for pathology full-field images according to claim 5, characterized in that: Analyze the probability distribution of the predicted answer based on the interactive information, and generate the question-answering results based on the probability distribution information, specifically including: Interact based on visual features and text features to obtain interactive information; Build a prediction model, output the predicted answer based on the prediction model, and analyze the probability distribution information of the predicted answer; Calculate the cross entropy loss function between the probability distribution of the prediction model output and the standard answer based on the probability distribution information of the predicted answer; Back propagation is performed based on the cross entropy loss function to optimize the parameters of the prediction model until the prediction model converges; The full slice image and the prompt word are input into the prediction model, and the prediction result of the answer is output to obtain the question-answering result.

7. An automated visual question answering system for pathology full-field images, characterized by: The system includes: a memory and a processor, wherein the memory includes a program of an automated visual question-answering method for pathology full-field images, and when the program of the automated visual question-answering method for pathology full-field images is executed by the processor, the following steps are implemented: Obtain a full-slice image, preprocess the full-slice image, remove non-information areas, and obtain a foreground area; Based on the partitioning strategy, the foreground area is divided into multiple low-pixel image units, and the visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence; Obtaining the prompt word input by the user, preprocessing the prompt word input by the user to obtain the input text, and converting the input text into a text feature sequence; Based on the cross-attention mechanism, the visual feature sequence and the text feature sequence interact to obtain interactive information; The probability distribution information of the predicted answer is analyzed based on the interactive information, and the question-answering results are generated based on the probability distribution information.

8. The automated visual question answering system for pathology full-field images according to claim 7, characterized in that: Obtain a full-slice image and preprocess it to remove non-information areas and obtain the foreground area, including: Acquire ultra-high-resolution full-field pathology images to obtain full-slice images, and adjust the resolution of the full-slice images to obtain low-resolution full-field image thumbnails; Convert the full-field image thumbnail from RGB color space to HSV color space to obtain an HSV image; Set the saturation threshold, divide the HSV image into regions, generate several sub-regions, and analyze the saturation of each sub-region; Compare the saturation of each sub-region with the set saturation to obtain the analysis results; Based on the analysis results, the background area is screened out to obtain the non-information area, and the non-information area is eliminated to obtain the foreground area.

9. The automated visual question answering system for pathology full-field images according to claim 8, characterized in that: Based on the partitioning strategy, the foreground area is divided into multiple low-pixel image units, and the visual features of the multiple low-pixel image units are extracted. The multiple visual features are subjected to dimensionality reduction processing to obtain a visual feature sequence, which specifically includes: Setting sliding windows based on adaptive grid partitioning strategy; Crop multiple non-overlapping image blocks from the foreground area based on a sliding window; Generate multiple low-pixel image units based on multiple image blocks of different stacks to obtain multiple image slices; The ViT network is used to reduce the dimension of the image slices to obtain the dimension reduction result, which is then converted into a visual feature sequence.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes an automated visual question-answering method program for pathology full-field images. When the automated visual question-answering method program for pathology full-field images is executed by a processor, the steps of the automated visual question-answering method for pathology full-field images as described in any one of claims 1 to 6 are implemented.

Images