Structured pathological report automatic generation method and system based on multi-dimensional features
By constructing a multidimensional feature fusion network model, multidimensional information from pathological images and reports is integrated to generate structured pathological reports. This solves the problems of universality and information integration in existing pathological report generation, and improves the accuracy and readability of the reports.
Patent Information
- Application Number
- CN202510839408.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-23
- Publication Date
- 2025-10-21
AI Technical Summary
Existing methods for generating pathology reports lack systematic analysis and structured expression of key diagnostic elements, making it difficult to integrate image information with molecular detection results. Furthermore, existing models are mostly trained for single diseases or single tasks, resulting in poor versatility and scalability, and failing to meet the multidimensional information fusion needs of modern precision medicine.
A multi-dimensional feature fusion network model is constructed, and attention pooling, cross-attention and self-attention mechanisms are introduced. Through multi-level feature mapping processing, structured pathology reports are generated and shared feature information between different diagnostic tasks is integrated.
It improves the accuracy of diagnostic information in pathology reports, generates reports in a structured format with controllable and highly consistent content, can be directly embedded into hospital information systems, enhances report readability and multidisciplinary collaboration efficiency, and has cross-disease and cross-modal scalability capabilities.
Smart Images
Figure CN120823943A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a method and system for automatically generating structured pathology reports based on multidimensional features. Background Art
[0002] Structured pathology reports improve the consistency and readability of reports through fixed templates and standard terminology, and have been initially applied in multiple oncology fields. However, with the development of precision medicine, clinical requirements for pathology reports have gradually increased. In addition to traditional morphological information, immunohistochemistry and molecular test results must also be integrated. The content of reports is becoming increasingly complex, and the time and expertise required of physicians have increased significantly. While visual-language models can be trained on large-scale image and text data to automatically generate radiology reports, most methods only generate brief descriptions of local image regions and lack systematic analysis and structured expression of key diagnostic elements (such as typing, grading, and depth of invasion). Existing models are mostly trained for single diseases or single tasks, with poor versatility and scalability. They are difficult to adapt to diverse clinical needs and difficult to simultaneously integrate image information and molecular test results. They are not suitable for the requirements of modern precision medicine for multidimensional information fusion. Summary of the Invention
[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a method and system for automatically generating structured pathology reports based on multidimensional features, which can effectively integrate shared feature information between different diagnostic tasks and improve the accuracy of reported diagnostic information.
[0004] The first technical solution adopted by the present invention is: a method for automatically generating a structured pathology report based on multidimensional features, comprising the following steps:
[0005] Obtain pathological image datasets and corresponding pathological reports and perform data preprocessing to obtain pathological image blocks and structured pathological reports;
[0006] Introducing the attention pooling module, cross attention module and self-attention mechanism to build a multi-dimensional feature fusion network model;
[0007] Based on the multi-dimensional feature fusion network model, feature set mapping processing is performed on the pathological image blocks and the structured pathology report to obtain the structured pathology report.
[0008] Furthermore, the step of acquiring a pathology image dataset and a corresponding pathology report and performing data preprocessing to obtain an initial pathology image feature set and a structured pathology report specifically includes:
[0009] Acquire a pathological image data set and scan it through a scanner to obtain a digital pathological image;
[0010] Performing scaling processing on the digitized pathological image to obtain a low-resolution thumbnail pathological image;
[0011] Performing grayscale conversion and binarization on the low-resolution thumbnail pathology image to obtain a binarized pathology mask image;
[0012] The tissue mask is optimized on the binarized pathological mask image through morphological operations to obtain the tissue region of the pathological image;
[0013] Cutting the tissue area of the pathological image by fixing the window size to obtain the pathological image block;
[0014] The pathology report corresponding to the pathology image dataset is obtained and structured to obtain a structured pathology report.
[0015] Furthermore, the multidimensional feature fusion network model includes an image block encoder, a slice-level feature extractor, an image block-level feature aggregator, a mapping network, and a language decoder. The image block encoder, the slice-level feature extractor, the image block-level feature aggregator, the mapping network, and the language decoder are connected in sequence, wherein:
[0016] The image block encoder includes a multi-head self-attention mechanism, a feedforward network, and a layer normalization operation layer;
[0017] The slice-level feature extractor consists of a two-layer transformer encoder;
[0018] The image block level feature aggregator includes a cross attention layer;
[0019] The mapping network includes an image block feature mapper and a slice-level feature mapper;
[0020] The language decoder is constructed in an autoregressive manner and includes several Transformer decoders.
[0021] Furthermore, the loss function of the multi-dimensional feature fusion network model is expressed as follows:
[0022]
[0023] In the above formula, θ represents all learnable parameters of the model, f WSI Represents the feature set of pathological images, Indicates the longest character length that can be processed, j indicates the character sequence number, p θ represents the probability distribution modeled by the model with parameter θ, c j Indicates the jth character processed.
[0024] Furthermore, the step of performing feature set mapping processing on the pathology image blocks and the structured pathology report based on the multi-dimensional feature fusion network model to obtain the structured pathology report specifically includes:
[0025] Based on the multi-dimensional feature fusion network model, feature extraction processing is performed on the pathological image blocks to obtain the feature set of the pathological image;
[0026] Combined with the feature set of the pathology image, the feature set mapping process is performed on the structured pathology report to obtain a structured pathology report.
[0027] Furthermore, the step of performing feature extraction processing on the pathological image block based on the multi-dimensional feature fusion network model to obtain a feature set of the pathological image specifically includes:
[0028] Input the pathological image block into the multi-dimensional feature fusion network model;
[0029] Perform feature extraction on the pathological image block to obtain an initial pathological image feature set;
[0030] Randomly initialize the token set. Based on the cross-attention module, the randomly initialized token set is used as the query and the initial pathological image feature set as the key and value. The weighted feature aggregation is calculated to extract the key information related to the task and obtain a streamlined feature set.
[0031] Based on the task-related key information, the category token is initialized and the category token is interacted with the initial pathological image feature set through the self-attention mechanism to obtain parameter-related features;
[0032] The simplified feature set is concatenated with the parameter-related features to obtain the feature set of the pathological image.
[0033] Furthermore, the step of combining the feature set of the pathology image and performing feature set mapping processing on the structured pathology report to obtain the structured pathology report specifically includes:
[0034] Based on the multi-perceptron, the feature set of the pathological image is mapped to the language space to obtain the image prefix representation;
[0035] Performing pathology modeling on the structured pathology report to obtain a sequence of continuous tokens and performing expansion processing to obtain an expanded sequence of continuous tokens;
[0036] Text is generated by autoregression, and the image prefix representation is combined with the expanded continuous token sequence to predict the current token to obtain a structured pathology report.
[0037] The second technical solution adopted by the present invention is: a system for automatically generating structured pathology reports based on multidimensional features, comprising:
[0038] The first module is used to obtain the pathology image data set and the corresponding pathology report and perform data preprocessing to obtain the pathology image block and structured pathology report;
[0039] The second module is used to introduce the attention pooling module, the cross attention module and the self-attention mechanism to build a multi-dimensional feature fusion network model;
[0040] The third module is used to perform feature set mapping processing on the pathological image blocks and the structured pathology report based on the multi-dimensional feature fusion network model to obtain the structured pathology report.
[0041] The beneficial effects of the method and system of the present invention are as follows: the present invention obtains pathological image data sets and corresponding pathological reports and performs data preprocessing to obtain pathological image blocks and structured pathological reports, further introduces attention pooling modules, cross-attention modules and self-attention mechanisms, constructs a multi-dimensional feature fusion network model, designs a multi-level attention pooling module and feature mapping network, first extracts key image block-level and task-level features, and then uniformly maps them to the language space to generate tasks, which greatly reduces the computational complexity and improves the training efficiency. Finally, based on the multi-dimensional feature fusion network model, the pathological image blocks and structured pathological reports are subjected to feature set mapping processing to obtain a structured pathology report. By constructing a unified visual language model architecture, the joint prediction of multiple pathological parameters and automatic report generation are realized, effectively integrating the shared feature information between different diagnostic tasks and improving the accuracy of the reported diagnostic information. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a flowchart of the steps of a method for automatically generating a structured pathology report based on multidimensional features of the present invention;
[0043] Figure 2 This is a structural block diagram of a system for automatically generating structured pathology reports based on multidimensional features according to the present invention;
[0044] Figure 3 is a schematic diagram of a multi-dimensional feature fusion network model provided by a specific embodiment of the present invention;
[0045] Figure 4 It is a schematic diagram of a structured pathology report provided by a specific embodiment of the present invention. DETAILED DESCRIPTION
[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.
[0047] Reference Figure 1 The present invention provides a method for automatically generating a structured pathology report based on multidimensional features, the method comprising the following steps:
[0048] S100, obtaining a pathology image dataset and a corresponding pathology report and performing data preprocessing to obtain a pathology image block and a structured pathology report;
[0049] Specifically, a pathology image dataset is acquired and scanned by a scanner to obtain a digitized pathology image; the digitized pathology image is scaled to obtain a low-resolution thumbnail pathology image; the low-resolution thumbnail pathology image is grayscale converted and binarized to obtain a binarized pathology mask image; the binarized pathology mask image is optimized by morphological operations to obtain a tissue mask to obtain a tissue area of the pathology image; the tissue area of the pathology image is cut by a fixed window size to obtain a pathology image block; the pathology report corresponding to the pathology image dataset is acquired and structured to obtain a structured pathology report.
[0050] In this example, a colorectal cancer pathology dataset was obtained from three domestic tertiary-level A hospitals, including a total of 10,481 WSI slice images and corresponding reports, covering two types of samples: postoperative resection specimens and biopsy samples.
[0051] All images were stained with H&E, with a resolution of 0.25 μm / pixel (40x magnification), and the image format was SVS or iSyntax. They were scanned with a Philips Ultra-Fast scanner or an Aperio AT2 scanner to obtain digital pathological images. The original 40x magnified pathological images were scaled to obtain low-resolution thumbnails for locating tissue areas. The thumbnails were converted into grayscale images, and the Otsu threshold method was used for binarization to separate the tissue area from the background. The tissue mask was further optimized through morphological operations to obtain the final tissue area. In the tissue area, sliding cutting was performed with a fixed window size to extract image blocks. The image blocks were input into the feature extractor to obtain 512-dimensional features and construct the initial feature set. M is the number of image blocks. The original report content regarding tumor classification, grade, invasion depth, vascular invasion, and nerve tract invasion was extracted and structured into a structured report. This served as a label for subsequent model training to supervise the model training.
[0052] S200, introduce the attention pooling module, cross attention module and self-attention mechanism to build a multi-dimensional feature fusion network model;
[0053] Specifically, if Figure 3As shown, the multi-dimensional feature fusion network model includes an image block encoder, a slice-level feature extractor, an image block-level feature aggregator, a mapping network and a language decoder. The image block encoder, the slice-level feature extractor, the image block-level feature aggregator, the mapping network and the language decoder are connected in sequence, wherein:
[0054] The image block encoder includes a multi-head self-attention mechanism, a feedforward network, and a layer normalization operation layer;
[0055] In this embodiment, the image block encoder is used to extract representational features for each image block. The encoder consists of 12 layers of transformer encoders, each layer of which includes a multi-head self-attention mechanism, a feedforward network, and a layer normalization operation in sequence. The multi-head self-attention mechanism includes 8 attention heads, each of which independently learns information interactions of different dimensions, and then splices the outputs of each attention head and performs linear mapping to restore the original dimension. The feedforward network consists of two layers of fully connected layers, and the GeLU activation function is introduced in the middle to enhance the nonlinear expression ability. Each layer of transformer introduces residual connections and layer normalization operations to improve gradient fluidity and model training stability. Finally, each image block is encoded as a 512-dimensional feature vector. Since the entire pathological image is split into multiple image blocks, an image block-level feature group will be obtained, which will be used as the input of the slice-level feature extractor and the image block-level feature aggregator.
[0056] The slice-level feature extractor consists of a two-layer transformer encoder;
[0057] In this embodiment, the slice-level feature extractor is used to globally model image block-level features and extract specific slice-level features for different pathological parameters (such as tumor type and degree of differentiation). This module uses a subnetwork consisting of a two-layer transformer encoder, with the same structure as the image block encoder, to capture global dependencies between image blocks. The output is multiple sets of slice-level task-related features, which serve as input to the subsequent mapping network.
[0058] The image block level feature aggregator includes a cross attention layer;
[0059] In this embodiment, the block-level feature aggregator extracts key task-related content from block-level features. Its architecture consists of a cross-attention layer, where the query is mapped from a randomly initialized token sequence, and the key and value are mapped from the block-level features. This mechanism uses the cross-attention operation to compress image information and extract key features, ultimately outputting a set of aggregated block-level features.
[0060] The mapping network includes an image block feature mapper and a slice-level feature mapper;
[0061] In this embodiment, the mapping network includes two parts: a patch feature mapper and a slice-level feature mapper, which are used to map the aggregated features of the image patch and the slice-level features to the language representation space, respectively. Both mappers are multi-layer perceptron structures with nonlinear feature transformation capabilities.
[0062] The language decoder is constructed in an autoregressive manner and includes several Transformer decoders.
[0063] In this embodiment, the mapped slice-level features and image block-level features are finally spliced together as the language prefix representation input to the language decoder. This module is used to generate a structured pathology diagnosis report based on the above-mentioned mapped features. The language decoder is constructed in an autoregressive manner. The overall structure is a 12-layer Transformer decoder, each layer contains 8 multi-head self-attention heads, and the hidden dimension is 768. The decoder uses the mapped image representation as the prefix input, combines the generated token sequence, and gradually predicts the next token, thereby realizing the serialized output of the report. Each structured report consists of a series of consecutive tokens, and the upper limit of the token length is 256.
[0064] The loss function of the model training in the embodiment of the present invention is: The model training goal is to maximize the conditional likelihood of the report token sequence: θ represents all learnable parameters of the model, and the optimization objective is to minimize the negative log-likelihood. This objective ensures that the generated report is fluent and accurately reflects the diagnostic information contained in the image.
[0065] S300 , based on a multi-dimensional feature fusion network model, performing feature set mapping processing on the pathology image block and the structured pathology report to obtain a structured pathology report.
[0066] S310, performing feature extraction processing on the pathological image block based on a multi-dimensional feature fusion network model to obtain a feature set of the pathological image;
[0067] Specifically, the pathological image block is input into the multidimensional feature fusion network model; the pathological image block is subjected to feature extraction processing to obtain the initial pathological image feature set; the token set is randomly initialized, and based on the cross-attention module, the randomly initialized token set is used as the query and the initial pathological image feature set is used as the key and value to calculate the weighted feature aggregation, extract the task-related key information, and obtain the streamlined feature set; based on the task-related key information, the category token is initialized, and through the self-attention mechanism, the category token is interacted with the initial pathological image feature set to obtain the parameter-related features; the streamlined feature set is spliced with the parameter-related features to obtain the feature set of the pathological image.
[0068] In this embodiment, each pathological image has thousands or tens of thousands of image blocks. If all features are input into the language decoder at the same time to generate a report, not only will the model be unable to determine the effective information, but it will also greatly increase the demand for computing resources. Therefore, the attention pooling module is used to extract effective information from the initial feature set. First, a token set is randomly initialized. Where N = 128. Then, through the cross attention module, the randomly initialized token set is used as the query and the initial feature set As keys and values, calculate weighted feature aggregation and extract key task-related information: Get a reduced feature set In order to construct a parameter-related feature for each pathological parameter, a category token is initialized for each parameter, and the category token is interacted with the initial feature set through the self-attention mechanism: Get parameter-related features f WSI_t Finally, the simplified feature set and parameter-related features are concatenated to obtain the feature set that ultimately represents the entire pathological image:
[0069] S320 , combining the feature set of the pathology image and performing feature set mapping processing on the structured pathology report to obtain a structured pathology report.
[0070] Specifically, based on the multi-perceptron, the feature set of the pathological image is mapped to the language space to obtain the image prefix representation; the structured pathology report is pathologically modeled to obtain a sequence of continuous tokens and expanded to obtain the extended sequence of continuous tokens; text is generated through autoregression, and the image prefix representation and the expanded sequence of continuous tokens are combined to predict the current token to obtain a structured pathology report.
[0071] In this embodiment, the feature set f of the pathological image is transformed into WSI Mapped to the language space, the image prefix representation h is obtained prefix . And the image prefix representation is used as the prefix of the language decoder. The structured report is modeled as a sequence of consecutive tokens: If the actual reported length is less than 256, the token will be extended to the maximum length In order to generate a report with semantic integrity and structural consistency, the embodiment of the present invention adopts an autoregressive method to generate text, that is, in each step of token generation, the language decoder generates the text based on the prefix image representation h. prefix and the generated history tokens {c1, c2, ..., c t-1}, predict the current token:
[0072] P(y t |y <t , h prefix )
[0073] By iteratively predicting the next token, the entire report sequence can be generated step by step, such as Figure 4 shown.
[0074] In summary, the embodiment of the present invention first obtains cases that have been diagnosed with colorectal cancer, and requires that the cases have complete pathological images and pathological reports. Excluding cases with image quality problems (blurred, broken, etc.) or missing clinical information, the original pathology reports are sorted out and adjusted to the form of structured text, the content of which includes tissue typing, grading, infiltration depth, vascular invasion and nerve bundle invasion, to obtain a structured pathology report; all pathology images are identified by tissue detection algorithm. The tissue area is divided into 1024×1024 image blocks at a magnification of 10 times. Each image block is extracted with a 512-dimensional visual feature vector through a pre-trained image encoder to obtain the initial feature set corresponding to each pathological image. In order to adapt to different task requirements, the attention pooling module is used to extract the most relevant information from the initial feature set based on the cross-attention mechanism to obtain streamlined image block-level features; the initial feature set is aggregated into a multi-pathological parameter diagnostic vector for different pathological parameters through the self-attention mechanism to obtain pathological image-level features constructed for each pathological parameter. The image block-level features and pathology image-level features are concatenated to obtain a final feature set. This is used to map the image features to the text space and generate a report based on the mapped features. The final feature set is input into a multi-layer perceptron and mapped to the language space to obtain the image pre-order input. The structured report is modeled as a token sequence. The image pre-order input is fed into a language decoder and generated word by word through autoregression until a termination symbol is generated. The model outputs a structured pathology report. The model training objective is to maximize the probability of correctly predicting the next token based on the generated token sequence, that is, to maximize the log-likelihood of the report sequence given the image features. Through training with this loss function, a structured pathology report generation model with good parameters is ultimately obtained.
[0075] Therefore, the embodiments of the present invention differ from the prior art in the following technical points:
[0076] 1) Existing intelligent pathology diagnosis systems often adopt a "single-task, single-model" approach, requiring independent model training for each diagnostic parameter (such as tumor typing, grading, staging, and molecular phenotype). This leads to issues such as information redundancy, resource waste, and lack of inter-task dependencies. This invention builds a unified visual language model architecture to enable joint prediction and automatic report generation for multiple pathology parameters, effectively integrating shared feature information across different diagnostic tasks, thereby improving the model's overall performance and application value.
[0077] 2) Traditional AI models primarily generate free-text descriptions, lacking controllability and structure, hindering clinical application and standardized management. The pathology reports generated by this invention are structured, with controllable content and clear fields. These reports can be directly embedded into hospital information systems or electronic medical records, improving report consistency and readability, and enhancing the efficiency of multidisciplinary collaboration and clinical decision support.
[0078] 3) Traditional models are often customized for specific tasks or diseases, resulting in limited versatility. This invention has been extensively validated on multicenter colorectal cancer data and successfully extended to prediction of biopsy samples and MMR molecular phenotypes, demonstrating excellent cross-morphological and cross-modal scalability. Its underlying architecture is highly flexible and can be transferred to other cancer types and pathology tasks, promising broad prospects for clinical application.
[0079] 4) To address the computational burden caused by oversized pathological images, the present invention designs a multi-level attention pooling module and feature mapping network, which first extracts key image block-level and task-level features and then uniformly maps them to the language space for task generation, greatly reducing the computational complexity and improving the training efficiency, as shown in Tables 1 and 2.
[0080] Table 1 Prediction performance of the model on different datasets
[0081]
[0082]
[0083] Table 2 Text evaluation index results of model-generated text
[0084]
[0085]
[0086] Reference Figure 2 , a structured pathology report automatic generation system based on multidimensional features, including:
[0087] The first module 201 is used to obtain a pathology image dataset and a corresponding pathology report and perform data preprocessing to obtain a pathology image block and a structured pathology report;
[0088] The second module 202 is used to introduce an attention pooling module, a cross attention module, and a self-attention mechanism to construct a multi-dimensional feature fusion network model;
[0089] The third module 203 is used to perform feature set mapping processing on the pathology image block and the structured pathology report based on the multi-dimensional feature fusion network model to obtain the structured pathology report.
[0090] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0091] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for automatically generating structured pathology reports based on multidimensional features, characterized in that: The following steps are involved: Obtain pathological image datasets and corresponding pathological reports and perform data preprocessing to obtain pathological image blocks and structured pathological reports; Introducing the attention pooling module, cross attention module and self-attention mechanism to build a multi-dimensional feature fusion network model; Based on the multi-dimensional feature fusion network model, feature set mapping processing is performed on the pathological image blocks and the structured pathology report to obtain the structured pathology report.
2. The method for automatically generating a structured pathology report based on multidimensional features according to claim 1, characterized in that: The step of obtaining a pathology image dataset and a corresponding pathology report and performing data preprocessing to obtain an initial pathology image feature set and a structured pathology report specifically includes: Acquire a pathological image data set and scan it through a scanner to obtain a digital pathological image; Performing scaling processing on the digitized pathological image to obtain a low-resolution thumbnail pathological image; Performing grayscale conversion and binarization on the low-resolution thumbnail pathology image to obtain a binarized pathology mask image; The tissue mask is optimized on the binarized pathological mask image through morphological operations to obtain the tissue region of the pathological image; Cutting the tissue area of the pathological image by fixing the window size to obtain the pathological image block; The pathology report corresponding to the pathology image dataset is obtained and structured to obtain a structured pathology report.
3. The method for automatically generating a structured pathology report based on multidimensional features according to claim 2, characterized in that: The multi-dimensional feature fusion network model includes an image block encoder, a slice-level feature extractor, an image block-level feature aggregator, a mapping network, and a language decoder. The image block encoder, the slice-level feature extractor, the image block-level feature aggregator, the mapping network, and the language decoder are connected in sequence, wherein: The image block encoder includes a multi-head self-attention mechanism, a feedforward network, and a layer normalization operation layer; The slice-level feature extractor consists of a two-layer transformer encoder; The image block level feature aggregator includes a cross attention layer; The mapping network includes an image block feature mapper and a slice-level feature mapper; The language decoder is constructed in an autoregressive manner and includes several Transformer decoders.
4. The method for automatically generating a structured pathology report based on multidimensional features according to claim 3, characterized in that: The loss function of the multi-dimensional feature fusion network model is expressed as follows: In the above formula, θ represents all learnable parameters of the model, f WSI represents the feature set of the pathological image, l represents the longest character length that can be processed, j represents the character sequence number, and p θ represents the probability distribution modeled by the model with parameter θ, c j Indicates the jth character processed.
5. The method for automatically generating a structured pathology report based on multidimensional features according to claim 4, characterized in that: The step of performing feature set mapping processing on the pathology image block and the structured pathology report based on the multi-dimensional feature fusion network model to obtain the structured pathology report specifically includes: Based on the multi-dimensional feature fusion network model, feature extraction processing is performed on the pathological image blocks to obtain the feature set of the pathological image; Combined with the feature set of the pathology image, the feature set mapping process is performed on the structured pathology report to obtain a structured pathology report.
6. The method for automatically generating a structured pathology report based on multidimensional features according to claim 5, characterized in that: The step of performing feature extraction processing on the pathological image block based on the multi-dimensional feature fusion network model to obtain a feature set of the pathological image specifically includes: Input the pathological image block into the multi-dimensional feature fusion network model; Perform feature extraction on the pathological image block to obtain an initial pathological image feature set; Randomly initialize the token set. Based on the cross-attention module, the randomly initialized token set is used as the query and the initial pathological image feature set as the key and value. The weighted feature aggregation is calculated to extract the key information related to the task and obtain a streamlined feature set. Based on the task-related key information, the category token is initialized and the category token is interacted with the initial pathological image feature set through the self-attention mechanism to obtain parameter-related features; The simplified feature set is concatenated with the parameter-related features to obtain the feature set of the pathological image.
7. The method for automatically generating a structured pathology report based on multidimensional features according to claim 6, characterized in that: The step of combining the feature set of the pathology image and performing feature set mapping processing on the structured pathology report to obtain the structured pathology report specifically includes: Based on the multi-perceptron, the feature set of the pathological image is mapped to the language space to obtain the image prefix representation; Performing pathology modeling on the structured pathology report to obtain a sequence of continuous tokens and performing expansion processing to obtain an expanded sequence of continuous tokens; Text is generated by autoregression, and the image prefix representation is combined with the expanded continuous token sequence to predict the current token to obtain a structured pathology report.
8. A structured pathology report automatic generation system based on multidimensional features, characterized in that: Includes the following modules: The first module is used to obtain the pathology image data set and the corresponding pathology report and perform data preprocessing to obtain the pathology image block and structured pathology report; The second module is used to introduce the attention pooling module, the cross attention module and the self-attention mechanism to build a multi-dimensional feature fusion network model; The third module is used to perform feature set mapping processing on the pathological image blocks and the structured pathology report based on the multi-dimensional feature fusion network model to obtain the structured pathology report.