Multimodal human-computer interaction application system based on the pathologist's perspective

Abstract

This application relates to the field of human-computer interaction, specifically disclosing a multimodal human-computer interaction application system based on a pathologist's perspective. It first generates a heatmap of candidate lesions through rapid scanning of whole-slice images, simulating a doctor's low-magnification global browsing, guiding the doctor to interactively identify key areas based on their professional judgment. This significantly improves the processing efficiency of ultra-large images. Subsequently, the system performs high-resolution depth analysis only on the lesions confirmed by the doctor, and fuses the extracted microscopic visual features with patient text information in a multimodal manner to generate a preliminary report with interpretable evidence, solving the problem of multimodal functions deviating from core clinical tasks. Finally, doctors can review and finalize the report through intuitive interaction, ensuring the doctor's dominant position and final decision-making power in the diagnostic process, thus overcoming the bottlenecks of traditional AI systems' "black box" operation and low clinical acceptance.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction, and more specifically, to a multimodal human-computer interaction application system based on the perspective of a pathologist. Background Technology

[0002] Pathological diagnosis, considered the "gold standard" of clinical medicine, is crucial for patient treatment planning due to its accuracy and efficiency. However, the field currently faces severe challenges: on the one hand, there is a severe shortage of pathologists, resulting in an enormous workload; on the other hand, whole-slide digital images (WSI) are enormous (reaching billions of pixels), making manual interpretation time-consuming and laborious. To alleviate this contradiction, AI-assisted diagnostic technology has emerged, but existing technologies still have some limitations when applied in clinical practice.

[0003] While existing AI-assisted pathology diagnostic systems have made progress in image processing speed and recognition of certain specific tasks, their design philosophy is often centered on technical implementation and fails to fully align with the actual workflow and thought processes of pathologists. In clinical practice, pathologists follow the habit of "global scanning first, then local focusing" when reading slides. That is, they first quickly browse the entire slide under low magnification to identify suspicious areas, and then switch to high magnification to carefully observe the cellular morphology, structure, and other detailed features of the lesions. However, many AI models employ strategies of brute-force computation or gridded analysis of the entire high-resolution image, which not only results in a large amount of redundant computing resources but also contradicts the intuitive workflow of doctors. Furthermore, in terms of multimodal data fusion, existing technologies often stop at the technical level of visual and textual feature stitching, failing to deeply integrate and collaboratively analyze multidimensional tasks that doctors are truly concerned with during the diagnostic process, such as tumor grading, staging, and detection of key pathological events (such as vascular tumor thrombi and nerve invasion). More importantly, the current human-computer interaction methods are generally rigid, mostly providing one-way "black box" results. Doctors cannot guide, intervene in, or verify the AI's analysis process using familiar medical terminology or routine image reading operations (such as selection, zooming, and annotation). This results in a lack of flexibility and interpretability in the diagnostic process, making it difficult to gain the full trust and acceptance of clinicians. Fundamentally, these shortcomings stem from the fact that existing technological solutions fail to take the "pathologist's perspective" as the core starting point of system design. These systems lack simulation of doctors' image reading habits, a deep understanding of diagnostic logic, and respect for operational intuition, creating a seemingly insurmountable gap between technological functionality and clinical needs.

[0004] Therefore, how to construct a working mode that can deeply simulate the combination of doctors' global insight and local precision observation, integrate multimodal information, and provide an interactive method that conforms to doctors' intuition has become a key technical bottleneck in promoting the real clinical application of digital pathology AI technology and improving diagnostic quality and efficiency. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. According to this application, a multimodal human-computer interaction application system based on a pathologist's perspective includes:

[0006] The whole-slice image scanning module is used to perform a rapid global feature scan on the acquired whole-slice digital image data to obtain a pyramid structure image, a candidate lesion heat map, and a candidate lesion list;

[0007] The lesion coordinate confirmation module is used to interactively lock suspicious areas based on doctor interaction data in pyramid structure images, candidate lesion heat maps and candidate lesion lists to obtain the coordinates of confirmed lesions.

[0008] The detailed feature deep analysis module is used to extract high-resolution patches from the pyramid structure image based on the confirmed lesion coordinates, and to perform detailed feature deep analysis on the high-resolution patches to obtain microscopic visual feature vectors.

[0009] The multimodal data processing module is used to perform multimodal data fusion and generate preliminary diagnostic reports from microscopic visual feature vectors and patient information text to obtain preliminary report data, interpretability evidence, and lesion localization data.

[0010] The report document interaction log generation module is used to interactively verify and finalize the preliminary report data, interpretability evidence, and lesion location data based on doctor feedback data, in order to obtain the final report document and interaction log.

[0011] Compared to existing technologies, this application provides a multimodal human-computer interaction application system based on the pathologist's perspective. It constructs a closed-loop system that deeply simulates the pathologist's workflow of "global scanning - local focusing - comprehensive diagnosis." To address the disconnect between existing technologies and clinical practice, the system first generates a candidate lesion heatmap through rapid scanning of whole-slice images, simulating a doctor's low-magnification global browsing and guiding the doctor to interactively identify key areas based on their professional judgment. This significantly improves the processing efficiency of ultra-large images. Subsequently, the system performs high-resolution deep analysis only on lesions confirmed by the doctor and performs multimodal fusion of the extracted microscopic visual features with patient text information to generate a preliminary report with interpretable evidence, solving the problem of multimodal functions deviating from core clinical tasks. Finally, doctors can review and finalize the report through intuitive interaction, ensuring the doctor's dominant position and final decision-making power in the diagnostic process, thus overcoming the bottlenecks of traditional AI systems' "black box" operation and low clinical acceptance. Attached Figure Description

[0012] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is a block diagram of a multimodal human-computer interaction application system based on the perspective of a pathologist, according to an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of data flow in a multimodal human-computer interaction application system based on the perspective of a pathologist, according to an embodiment of this application.

[0015] Figure 3 This is a block diagram of the whole-slice image scanning module in a multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application.

[0016] Figure 4 This is a block diagram of the detailed feature deep analysis module in a multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application.

[0017] Figure 5 This is a block diagram of the multimodal data processing module in a multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0020] This application is made in response to the problems existing in the aforementioned prior art. Figure 1 This is a block diagram of a multimodal human-computer interaction application system based on the perspective of a pathologist, according to an embodiment of this application. Figure 2 This is a schematic diagram of the data flow in a multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application. Specifically, as shown... Figure 1 and Figure 2 As shown, the multimodal human-computer interaction application system 100 based on the pathologist's perspective according to an embodiment of this application includes: a whole-slice image scanning module 110, used to perform global feature rapid scanning on the acquired whole-slice digital image data to obtain a pyramid structure image, a candidate lesion heatmap, and a candidate lesion list; a lesion coordinate confirmation module 120, used to interactively lock suspicious areas based on doctor interaction data in the pyramid structure image, candidate lesion heatmap, and candidate lesion list to obtain the coordinates of confirmed lesions; and a detailed feature deep analysis module 130, used to perform deep analysis based on the coordinates of confirmed lesions... High-resolution patches are extracted from the pyramid structure image, and detailed feature depth analysis is performed on the high-resolution patches to obtain microscopic visual feature vectors; multimodal data processing module 140 is used to perform multimodal data fusion of microscopic visual feature vectors and patient information text to generate a preliminary diagnostic report to obtain preliminary report data, interpretability evidence, and lesion localization data; report file interaction log generation module 150 is used to interactively verify and finalize the preliminary report data, interpretability evidence, and lesion localization data based on doctor feedback data to obtain the final report file and interaction log.

[0021] Specifically, the whole-slice image scanning module 110 is used to perform a rapid global feature scan on the acquired whole-slice digital image data to obtain a pyramid structure image, a candidate lesion heatmap, and a candidate lesion list. It is understandable that in pathological diagnostic practice, facing whole-slice digital images with billions of pixels, directly performing high-resolution detailed analysis would not only consume enormous computational resources, leading to low processing efficiency, but also contradict the pathologist's habit of viewing images globally first and then locally. Doctors typically first perform a rapid scan under low magnification to quickly locate suspicious lesion areas from a macroscopic perspective, and then switch to high magnification for detailed observation. Existing technologies often neglect this core workflow, resulting in a disconnect between technology and clinical application. Therefore, setting up a full-slice image scanning module can accurately simulate the doctor's rapid global scanning behavior under low magnification. By performing dimensionality reduction processing and rapid feature extraction on the image, a navigation map containing the location and probability of potential lesions, namely a candidate lesion heatmap and list, can be efficiently generated. This allows subsequent high-precision, high-energy-consuming analysis to focus on a few high-value areas, thereby greatly improving the overall efficiency of the diagnostic process while ensuring that no key information is missed, and keeping the starting point of human-computer interaction consistent with the doctor's operational intuition.

[0022] In a feasible technical solution Figure 3 This is a block diagram of the whole-slice image scanning module in a multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application. Figure 3As shown, the full-slice image scanning module 110 includes: an image downsampling unit 111, used to downsample the full-slice digital image step by step to obtain the pyramid structure image; a low-resolution layer extraction unit 112, used to extract a preset low-resolution layer from the pyramid structure image as a scanning layer; and a candidate lesion information generation unit 113, used to perform parallel feature extraction and heat map generation on the scanning layer to obtain the candidate lesion heat map and candidate lesion list.

[0023] Specifically, the execution process of the whole-slice image scanning module 110 is as follows: First, it is necessary to acquire a digital image of the whole slide. Also known as a digital pathological slide (WSI), it is an ultra-high resolution digital image generated by using a dedicated whole-slice scanning device to perform high-precision, full-field scanning of a physical tissue specimen on a glass slide (e.g., tissue that has been formalin-fixed, paraffin-embedded, sectioned, and stained with hematoxylin and eosin) under a high-power objective lens, typically 20x or 40x. This image completely reproduces all the histological information on the physical slide, and its data volume is extremely large. For example, a typical colorectal cancer tissue slide, after being scanned under a 40x objective lens, may generate a digital image data with an original resolution of up to 120,000 x 90,000 pixels and a file size of several gigabytes (GB).

[0024] Next, after receiving the high-resolution full-slice digital image data, the image downsampling unit 111 initiates a step-by-step downsampling process to generate a pyramid structure image. This process uses digital image processing algorithms to progressively reduce the image resolution. The selected algorithms aim to reduce the image size while minimizing information distortion and preserving key organizational and textural features. Commonly used algorithms include bilinear interpolation, bicubic interpolation, or the more complex Lanczos interpolation algorithm. Bicubic interpolation is a preferred option in this embodiment because it achieves a good balance between image quality and computational efficiency. This algorithm calculates the gray value of the new pixel by examining the gray values ​​of the 16 neighboring pixels around the sampling point and performing a weighted average, thereby generating a smoother, less jagged reduced image. In detail: a fixed downsampling factor is set, which is set to 2, corresponding to the conventional relationship between microscope objective magnifications such as 40x, 20x, and 10x. The following will use the above 120000x90000 pixel original image corresponding to 40x magnification as an example for explanation. First, the original image is defined as Level 0 of the pyramid. Next, the image downsampling unit 111 applies a bicubic interpolation algorithm to the Level 0 image, using a downsampling factor of 2, processing simultaneously in both the horizontal and vertical directions. When calculating the value of each new pixel in the target image, the algorithm first maps its coordinates back to the coordinate space of the original high-resolution image. Then, using this mapped point as the center, a 4x4 pixel matrix is ​​selected, consisting of 16 neighboring pixels, as the basis for calculation. The core of bicubic interpolation lies in its weighted summation of the color or grayscale values ​​of these 16 pixels, with the weights determined based on a cubic polynomial function. This function calculates the contribution of each neighboring pixel based on its precise distance to the center mapped point; pixels closer to the center have higher weights. Thus, the algorithm not only considers the pixel value itself but also indirectly utilizes the gradient information of local pixel values, ensuring the interpolation result remains continuous in both numerical value and first derivative. This ensures that image edges remain smooth and subtle texture details are better preserved during resolution reduction, resulting in a new image with higher visual quality and fewer artifacts at a resolution of 60,000 x 45,000 pixels. This new image is visually equivalent to the image observed under a 20x objective lens and is defined as Level 1 of the pyramid. The unit then continues the same downsampling operation on the Level 1 image, obtaining an image with a resolution of 30,000 x 22,500 pixels, which is equivalent to Level 2 under a 10x objective lens. This process is iteratively executed, successively generating lower resolution image levels such as Level 3 (5x, 15,000 x 11,250 pixels), Level 4 (2.5x, 7,500 x 5,625 pixels), and so on.The downsampling process continues until the image resolution is reduced to a preset minimum level, determined by the lowest resolution required to clearly display the complete outline of a tissue section on a standard monitor, such as 1.25x or lower. At this point, the image primarily displays the overall outline and position of the tissue section on the slide. In this way, the original single high-resolution image is transformed into a collection of images containing multiple different resolution levels. This structured dataset is known as a pyramid-structured image. This data structure greatly optimizes the slide viewing experience. When a specific region of the image needs to be viewed at a particular magnification (e.g., 10x), the data for that region can be decoded directly from the second layer of the pyramid, without needing to scale in real-time from the massive amount of data in the first layer. This enables view switching and smooth zooming and panning operations.

[0025] Next, the low-resolution layer extraction unit 112 extracts a preset low-resolution layer from the pyramid structure image generated in the previous step as the scanning layer. The operation of this unit is extraction, and which layer to extract is determined by a preset value. This preset low-resolution layer level is based on a comprehensive consideration of clinical pathology knowledge and algorithm performance optimization. In this embodiment, for the colorectal cancer screening task, the preset extraction level is determined to be layer 3, which corresponds to a magnification of 5 times. The unit's operation is to directly locate and read all image data of this layer from the pyramid structure image data file according to the index (i.e., layer number 3), thereby obtaining an image with a resolution of 15000x11250 pixels. This extracted image is the scanning layer. Compared to the original 40x image, its data volume is significantly reduced, making it suitable for rapid parallel processing; at the same time, compared to layers with lower magnification, it still retains morphological features sufficient to distinguish normal glands from cancerous gland clusters.

[0026] After receiving the scanned layer, the candidate lesion information generation unit 113 initiates a parallel processing flow to generate a candidate lesion heatmap and a candidate lesion list. In a feasible technical solution, the candidate lesion information generation unit 113 includes: a tile segmentation subunit 1131, used to segment the scanned layer into tiles to obtain a tile network composed of scanned tiles; a scanned tile category probability generation subunit 1132, used to perform feature extraction and prediction based on a lightweight visual model on each scanned tile in the tile network to obtain a category probability vector for each scanned tile; a candidate lesion heatmap generation subunit 1133, used to generate the candidate lesion heatmap based on the category probability vectors of each scanned tile in the tile network; and a candidate lesion list generation subunit 1134, used to generate the candidate lesion list based on a comparison between the category probability vectors of each scanned tile in the tile network and a preset threshold.

[0027] The process is first executed by the tile-segmentation subunit 1131. This subunit receives the input of the scan layer determined in the previous stage, such as an image with a resolution of 15000x11250 pixels, corresponding to a 5x magnification. Since subsequent deep learning models process fixed-size inputs, and to achieve large-scale parallel computation, directly processing the entire scan layer is not feasible. Therefore, the tile-segmentation subunit 1131 discretizes this continuous image. It employs a sliding window strategy with overlap to segment the entire scan layer into a structured network composed of numerous scan tiles. In a specific embodiment, the tile size is set to 256x256 pixels, a size that strikes a good balance between capturing local histological features and controlling computational load. To prevent the segmentation of important features at tile boundaries, such as an anomalous glandular structure spanning two tiles, an overlap pixel value is set during segmentation, determined experimentally, for example, 32 pixels. That is, each tile shares a 32-pixel region with its adjacent tiles in both the horizontal and vertical directions. After this processing, the 15000x11250 pixel scan layer was decomposed into a tile network containing thousands of 256x256 pixel scan tiles. This network not only contains the image data of all tiles, but also accurately records the coordinate position of each tile in the original scan layer.

[0028] Subsequently, the scan patch category probability generation subunit 1132 processes each scan patch in the patch network. This subunit integrates a pre-trained lightweight visual model. To meet the need to process tens of thousands of patches in a very short time, one technical solution in this application employs the efficient MobileNetV3 convolutional neural network architecture, whose core advantages lie in the design of depthwise separable convolution and inverted residual blocks. Specifically, depthwise separable convolution decomposes standard convolution into two steps: first, depthwise convolution, which uses a set of independent filters to extract spatial features from each channel of the input data; then, pointwise convolution, which linearly combines the features from each channel of the depthwise convolution output using a 1x1 convolution kernel. This decomposition method significantly reduces the number of model parameters and floating-point operations while maintaining the receptive field and feature extraction capability. The inverted residual block is an innovative bottleneck structure. It first expands the input feature map's channels using a 1x1 pointwise convolution, then performs feature transformation using a lightweight depthwise convolution in a higher-dimensional space. Finally, it compresses the number of channels back to the original dimension using another 1x1 pointwise convolution and adds it to the original input feature map through a short-circuit connection. This combination of short-circuit connections ensures the model learns rich feature representations while effectively preventing the gradient vanishing problem in deep network training. The model's weights and biases were obtained through supervised learning on a large-scale, pathology-annotated patch dataset. This dataset contains a massive number of 256x256 pixel patches extracted from whole-slice digital images, each patch assigned a specific category label, such as normal colorectal glands, high-grade intraepithelial neoplasia, adenocarcinoma, and mucinous adenocarcinoma. During training, a loss function such as cross-entropy is used to measure the gap between the model's predictions and the true labels. The Adam optimizer is used to iteratively adjust the model's weights and biases via backpropagation until the model achieves the preset classification accuracy, sensitivity, and specificity metrics on independent validation sets. In actual execution, each scanned patch in the tile network is input into this trained lightweight vision model. As image data flows through the network backbone, which is composed of multiple stacked inverted residual blocks, its spatial dimension gradually decreases while its feature dimension gradually increases, thus being abstracted into a set of high-dimensional feature maps. At the end of the network, a global average pooling layer transforms the two-dimensional feature map into a one-dimensional feature vector. This vector is then fed into a fully connected layer and a Softmax activation function. The Softmax function converts the output of the fully connected layer into a probability distribution, ultimately outputting a class probability vector. The dimension of this vector is exactly the same as the number of classes set during training, and the sum of all elements in the vector is 1.For example, for an input patch, the output class probability vector could be [normal colorectal gland: 0.02, high-grade intraepithelial neoplasia: 0.04, adenocarcinoma: 0.91, mucinous adenocarcinoma: 0.03]. This vector indicates that the model classifies the patch as a normal colorectal gland with 2% confidence, high-grade intraepithelial neoplasia with 4% confidence, adenocarcinoma with 91% confidence, and mucinous adenocarcinoma with 3% confidence. This process is executed in parallel on multiple computational cores, ensuring high-throughput processing of the entire patch network. The final output is a set of class probability vectors that correspond one-to-one with the patches in the patch network.

[0029] Based on these probability vectors, the candidate lesion heatmap generation subunit 1133 begins operation. To accurately transform this probability information into a continuous and intuitive visual image, especially when dealing with overlapping areas introduced during tile segmentation, this subunit first constructs a floating-point probability map in memory with the exact same size as the original scan layer (15000x11250 pixels), and initializes all pixel values ​​of this map to zero. Subsequently, the subunit initiates a systematic traversal process, sequentially processing each scan patch in the tile network and its corresponding category probability vector. For each patch, it first reads the patch's precise starting coordinates on the scan layer from the tile network's metadata and extracts the target lesion category, such as the probability value of adenocarcinoma, from its probability vector. Next, it plots this probability value onto the probability map. Specifically, for each pixel location covered by the 256x256 pixel patch, the subunit reads the existing value in the probability map for that location, compares it with the adenocarcinoma probability value of the current patch, and then writes the larger value back to that pixel location. By performing this maximum projection operation, when a pixel is covered by multiple overlapping patches, its final value in the probability map will be the highest lesion probability given by all the patches covering it. This mechanism ensures that even weak but consistent positive signals are effectively preserved, and the signal intensity of high-probability regions is not diluted by neighboring low-probability overlapping regions. After traversing all patches and fully integrating their probability information into the probability map, the map becomes a data matrix that accurately reflects the lesion probability of each pixel. The final step is visualization rendering. The sub-unit applies a predefined color mapping function that non-linearly or linearly correlates continuous probability values ​​between 0.0 and 1.0 with a specific color gradient. For example, an optimized color gradient can map 0.0 to a deep blue, 1.0 to a bright red, and smoothly transition intermediate probability values ​​through a series of colors such as cyan, green, and yellow to enhance the human eye's ability to distinguish key probability intervals. This color mapping function is applied to each pixel in the probability map, converting its floating-point probability value into a specific RGB color value. Ultimately, the rendered collection of pixels forms a richly colored and information-rich heatmap of candidate lesions. On this map, bright red areas indicate regions with the highest probability of cancer, while blue areas represent tissue background that the model judges to be normal, thus providing pathologists with an intuitive and comprehensive view of the global lesion distribution.

[0030] Meanwhile, the candidate lesion list generation subunit 1134 transforms the set of category probability vectors into a highly structured candidate lesion list. Initial screening is performed based on a preset threshold. This threshold is scientifically determined by plotting and analyzing receiver operating characteristic (ROC) curves on an independent test dataset containing a large number of positive and negative samples. In this embodiment, the threshold can be set to 0.80. The subunit iterates through the category probability vectors of all patches and marks patches with adenocarcinoma category probability values ​​strictly exceeding 0.80 as positive patches in its internal logic, thereby generating a binary positive patch distribution map covering the entire scan area. Based on this, the subunit applies a connected component analysis algorithm to all marked positive patches. This algorithm uses 8-neighborhood connections—that is, a patch is considered adjacent to patches above, below, left, right, and four diagonally opposite it—as a criterion, systematically aggregating spatially connected or overlapping positive patches into independent, clinically significant candidate lesion regions. After clustering, the system calculates and encapsulates a series of key descriptive attributes for each aggregated candidate lesion region. These include: determining the smallest bounding box that completely encloses all patches within the region and accurately mapping its coordinates back to the coordinate system of the original 40x image through magnification conversion for subsequent high-resolution detail observation; calculating the geometric centroid coordinates of the region, also mapped back to the 40x coordinate system, providing a precise center point for rapid navigation; counting the total number of positive patches constituting the region as a quantitative indicator of lesion area size; and calculating the maximum adenocarcinoma probability of all patches within the region, using this value as the final confidence score for the candidate lesion, as it represents the most definitive malignant signal within the region. Finally, all these information-rich candidate lesion regions are organized into a structured data list and strictly sorted from highest to lowest confidence score. This final, priority-sorted list constitutes the candidate lesion list, providing pathologists with a clear, quantifiable, and focused checklist for examination.

[0031] Specifically, the lesion coordinate confirmation module 120 is used to interactively lock suspicious areas based on doctor interaction data in the pyramid structure image, candidate lesion heatmap, and candidate lesion list to obtain the coordinates of confirmed lesions. Correspondingly, the candidate lesion heatmap and list provide valuable preliminary screening results for pathological diagnosis, but these results are essentially probabilistic predictions based on algorithmic models, which may contain false positives or fail to fully capture the nuances of specific cases. In clinical practice, the final diagnostic authority rests with the pathologist. Therefore, simple automated output cannot meet the clinical requirements for rigor and reliability. Based on this, to establish a crucial interactive bridge between automated analysis and the doctor's professional judgment, this application introduces processing based on doctor interaction data to transform machine-generated candidate information into target information that is ultimately decided and confirmed by the doctor. This ensures that all subsequent high-precision, resource-intensive in-depth analyses accurately focus on the areas of true interest to the doctor, thereby seamlessly integrating the doctor's experience and wisdom into the diagnostic process and solving the fundamental problems of black-box operation and lack of doctor-led intervention in existing technologies.

[0032] In one feasible technical solution, the lesion coordinate confirmation module 120 includes: an image overlay unit 121, used to overlay a heat map of candidate lesions onto a thumbnail of a pyramid structure image in the left image area of ​​the interactive interface; and a candidate lesion list display unit 122, used to display a list of candidate lesions in the right interactive area of ​​the interactive interface.

[0033] Specifically, the execution process of the lesion coordinate confirmation module 120 is as follows: This module's execution unfolds within a dedicated interactive interface. This interface is divided into two main areas to simulate a pathologist's workbench. First, the image overlay unit 121 is responsible for constructing and managing the image area on the left side of the interface. This unit receives a pyramid structure image and a candidate lesion heatmap as input. First, a suitable layer for overview is selected from the pyramid structure image as the base image; for example, the same layer used when generating the heatmap is selected, i.e., layer 3, 5x, 15000x11250 pixels. Then, the candidate lesion heatmap is overlaid on this base image in a semi-transparent manner. The transparency value is adjustable, for example, preset to 50%, and can be dynamically adjusted as needed by the doctor to ensure that the underlying tissue structure is clearly visible while observing the heatmap indications. This overlay display method allows the doctor to intuitively correlate the high-risk areas (red areas) indicated by the model with the actual tissue morphology. The image area also supports smooth interactive operation. Doctors can use the mouse wheel to zoom in and out steplessly (achieved in the background by decoding different levels of the pyramid structure image in real time) and pan by dragging, completely replicating the operating experience of using a physical microscope.

[0034] Meanwhile, the candidate lesion list display unit 122 is responsible for constructing and managing the interactive area on the right side of the interface. This unit receives the candidate lesion list as input. It displays each item in the list, i.e., each candidate lesion region, in a structured form. Each row represents a candidate lesion and clearly lists its key information, such as number, confidence score (e.g., 92%), and possible preliminary classification (e.g., adenocarcinoma). The list is sorted from high to low confidence score, allowing doctors to prioritize the most suspicious areas. Once the interface is constructed, the entire module enters a state awaiting doctor interaction. Doctors can provide interactive data in various ways that align with their operational intuition to pinpoint areas they deem necessary for further analysis. In one feasible technical solution, the doctor's interactive data includes: a bounding box area selected by the doctor on the pyramid structure image or candidate lesion heatmap, an item clicked by the doctor in the candidate lesion list, and text commands entered by the doctor. Specifically, one interaction method is for the doctor to click on an item in the candidate lesion list on the right side with the mouse. For example, the doctor clicks on candidate lesion 1, which has the highest confidence score in the list. The click event is captured by the module and immediately triggers a linked response: in the image area on the left, the view automatically pans and zooms, placing the area corresponding to candidate lesion 1 in the center of the view and marking it with a bright rectangle. This image-text linkage greatly improves positioning efficiency. Another interaction method allows doctors to operate directly in the image area on the left, without relying on a list. Doctors can use the mouse to directly select areas of any shape or size on the image based on their judgment of the overlaid image. For example, a doctor might find an area on a heatmap that is not significantly highlighted in red but has a suspicious shape, so they manually select that area. The coordinate information of this selection operation (coordinates of the top-left vertex and width and height) is captured by the module in real time. In addition, more advanced text command interaction is supported. Doctors can enter commands conforming to medical terminology in the text input box in the interaction area, such as analyzing the vascular tumor thrombus here. This interaction method can be used in combination with the first two methods. Regardless of the interaction method used by the doctor, their operation (click, selection, text input) will be parsed by the module as doctor interaction data. The module converts this interaction data into precise coordinate information. If a doctor clicks on a list item, the coordinates of the bounding box of the lesion associated with that item are extracted. If the doctor manually selects a lesion, the coordinates of the selected area are directly adopted. These coordinates, obtained directly or indirectly from the doctor's interaction, are defined as the coordinates of the confirmed lesion. For example, if the doctor clicks on candidate lesion 1, its corresponding bounding box coordinates in a coordinate system 40 times that of the original image, such as x:10240, y:20480, width:512, height:512, are locked.

[0035] Specifically, the detailed feature depth analysis module 130 is used to extract high-resolution patches from the pyramid structure image based on the confirmed lesion coordinates, and perform detailed feature depth analysis on the high-resolution patches to obtain a microscopic visual feature vector. It should be understood that the preceding steps successfully simulate the process of a pathologist quickly scanning and identifying suspicious areas under low magnification, but this is only the first step in discovering the problem. To accurately determine the nature of the lesion, such as distinguishing between benign and malignant, and determining the tumor grade, it is necessary to rely on meticulous observation of microscopic details such as cell morphology, nuclear features, and tissue structure under high magnification. This high-resolution information is the cornerstone of diagnosis, but it exists in the form of unstructured pixels. Therefore, this module is designed to simulate the depth analysis and quantitative evaluation process after a pathologist switches to high magnification. That is, the unstructured, high-resolution image region confirmed by the doctor is transformed into a structured, quantifiable numerical vector that comprehensively describes its microscopic pathological characteristics through a series of deep learning models and image analysis algorithms, providing accurate and reliable visual evidence for subsequent multimodal data fusion and automated report generation.

[0036] In a feasible technical solution Figure 4 This is a block diagram of the detailed feature deep analysis module in a multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application. Figure 4 As shown, the detailed feature deep analysis module 130 includes: a text instruction judgment and parsing unit 131, used to determine whether there is a text instruction in the doctor's interaction data, and in response to the existence, to perform instruction intent parsing on the text instruction to obtain a context analysis instruction; an original analysis result generation unit 132, used to perform context-aware model reasoning and original feature extraction on the high-resolution image patch based on the context analysis instruction to obtain an original analysis result, the original analysis result including a segmentation mask and detection results; and a feature quantization and encapsulation unit 133, used to perform feature quantization and structured vector encapsulation on the original analysis result to obtain the microscopic visual feature vector.

[0037] Specifically, the execution process of the detailed feature deep parsing module 130 is as follows: First, it receives the confirmed lesion coordinates output by the lesion coordinate confirmation module 120, as well as the pyramid structure image generated by the full-slice image scanning module 110. For example, if a doctor confirms a lesion, its confirmed lesion coordinates in the 40x original image coordinate system are (x:10240, y:20480, width:512, height:512). It accesses the highest resolution level in the pyramid structure image, namely Level 0, which completely preserves the original pixel information at 40x magnification. Based on the received coordinates (10240, 20480), the process locates the starting point in the massive data of Level 0 and reads all pixel data within a rectangular area with a width and height of 512 pixels. This extraction process is efficiently completed by a decoder that supports image file formats with random region access (such as SVS or block TIFF), avoiding the loading of the entire billions of pixels of image data.

[0038] After obtaining high-resolution image tiles, the text instruction interpretation and parsing unit 131 begins operation. For example, if a doctor selects a region with coordinates (x:10240, y:20480, width:512, height:512) and simultaneously enters the instruction "Analyze the glandular structure and mitotic figures in this region" in the text box of the interactive interface, this unit first performs a judgment operation to determine whether a text instruction exists in the interactive data. This is a simple check, namely, whether the text instruction field is empty. In response to determining the existence of a text instruction (i.e., the field is not empty), the unit initiates the instruction intent parsing function. This function is handled by an embedded, specially trained natural language understanding model. The specific architecture of this model can be a Transformer-based pre-trained language model, such as a BERT model fine-tuned for the biomedical text domain. Structurally, this model mainly consists of an input embedding layer, multiple stacked Transformer encoders, and a final classification output layer. The input embedding layer is responsible for segmenting text instructions, such as analyzing the kernel splitting image of this region, and generating an initial vector representation for each word that integrates semantic, positional, and fragment information. These vectors are then fed into the core Transformer encoder. This encoder, through its internal multi-head self-attention mechanism, can capture the complex dependencies between words within the instruction in parallel, thus gaining a deep understanding of the instruction's context. Each encoder layer also contains a feedforward neural network for non-linearly transforming the features extracted by the attention mechanism, further deepening the representation. After layer-by-layer processing by the encoder, the final output vector is fed into the top classification layer. This classification layer uses a fully connected network to map high-dimensional features onto a predefined intent category space, thus completing the parsing from natural language to structured instructions. The model's weights and biases are obtained through supervised learning on a large dataset of medical instruction-intent annotation pairs. This dataset consists of pairs of samples, each pair containing a natural language instruction that a doctor might input and its corresponding structured intent representation. For example, the task "Analyze the mitotic figure of this region" is {"task": "detection","target": "mitotic_figure"}, and the task "Accurately delineate the glandular structure" is {"task":"segmentation","target":"gland"}. During training, the model minimizes the classification loss (such as cross-entropy loss) between the model's predicted intent and the true labeled intent, and uses optimizers such as Adam to iteratively update the model parameters until the model can accurately extract the core task and target entity from natural language. During actual execution, the obtained text instruction to analyze the mitotic figure of this region is used as input and fed into this trained model.The model, through its multi-layered self-attention mechanism, understands the contextual relationships between words in a sentence, ultimately identifying the corresponding core task "detection" and the corresponding target entity "mitotic figure". The model outputs a structured JSON object: {"tasks":[{"type": "segmentation","target":"gland"},{"type":"detection","target":"mitotic_figure"}]}. If the judgment result indicates the absence of a text instruction—that is, the doctor only performed a selection or click operation without providing any text information—the unit will generate a preset, general contextual analysis instruction. This default instruction is designed to perform a comprehensive set of basic pathological analyses, for example: {"task":"general_analysis"}.

[0039] Next, the raw analysis result generation unit 132 performs targeted model inference and raw feature extraction on the high-resolution image patch based on the received context analysis instructions. This unit maintains a model library containing multiple pre-trained deep learning models for different pathological tasks. It is a conditional judgment and model scheduling process based on context analysis instructions. If the context analysis instruction received by the unit is a clear composite task instruction: {"tasks":[{"type":"segmentation","target":"gland"},{"type":"detection","target":"mitotic_figure"}]}, and a 512x512 pixel high-resolution image patch, for the first task in the instruction, {"type":"segmentation","target":"gland"}, the unit loads a U-Net model for semantic segmentation from the model library, designed to accurately delineate glandular structures. This model features a symmetrical encoder-decoder structure. The encoder progressively extracts deep features and reduces the feature map size through a series of convolution and pooling operations, while the decoder progressively restores the image resolution through upsampling and convolution operations. Skip connections fuse shallow and deep features from the corresponding encoder layers, achieving precise target boundary localization. The model's weights and biases were trained on a large dataset of patch images with meticulously drawn gland outlines by pathologists, optimizing metrics such as the Dice loss function. During execution, a high-resolution 512x512 pixel patch is input into the U-Net model, and the output is a 512x512 pixel binary segmentation mask of the same size as the original patch, where pixels identified as glandular regions have a value of 1, and background regions have a value of 0. Meanwhile, for the second task in the instruction, {"type":"detection","target":"mitotic_figure"}, this unit loads a highly efficient single-stage object detection model specifically designed for mitotic figure detection, such as YOLOv8. This model primarily consists of three parts: a backbone network for extracting multi-level features from the input patch; a neck network (e.g., a PANet structure) for fusing features from different levels to enhance the detection capability for objects of different sizes; and a head network for predicting the fused features to output bounding boxes and class probabilities. Similarly, the model's weights and biases are obtained through supervised training on a large-scale dataset meticulously annotated by experienced pathologists. In actual execution, this 512x512 pixel high-resolution patch is fed into the loaded YOLOv8 model for one forward inference iteration.The model outputs a list of detection results, where each item represents a detected potential mitotic figure and includes its bounding box coordinates, confidence score, and class label. For example, the output might be [[123,234,15,15,0.95,"mitotic figure"],[301,410,18,18,0.88,"mitotic figure"]]. When receiving the default instruction {"task":"general_analysis"}, this unit will call multiple models in parallel or sequentially. For example, it will simultaneously perform the gland segmentation and mitotic figure detection tasks mentioned above. Finally, this unit systematically aggregates the outputs generated by these two parallel tasks—the gland segmentation mask generated by the U-Net model and the list of mitotic figure detection results generated by the YOLOv8 model—to form a unified, structured raw analysis result. This result was generated in a single execution based on specific compound instructions input by the doctor. It explicitly includes a pixel-level segmentation mask of the same size as the patch, generated by the segmentation model, and a list of detection results, including bounding box coordinates and confidence scores, generated by the detection model.

[0040] Finally, the feature quantization and encapsulation unit 133 performs in-depth processing on this complex original analysis result, transforming it from qualitative image-level information into quantitative, structured numerical features. In a feasible technical solution, the feature quantization and encapsulation unit 133 includes: a result parameter calculation subunit 1331, used to calculate the morphological and textural parameters of each segmentation mask in the original analysis result to obtain an object index list; a detection result filtering and analysis subunit 1332, used to perform detection box filtering and statistical analysis on each detection result in the original analysis result to obtain a kernel splitting image count; and a feature vectorization subunit 1333, used to perform feature vectorization on the object index list and the kernel splitting image count to obtain the microscopic visual feature vector.

[0041] Specifically, the execution of this unit begins by receiving the raw analysis results, which include a 512x512 pixel gland segmentation mask generated by the U-Net model and a list of detection results generated by the YOLOv8 model, containing two potential mitotic figures [[123,234,15,15,0.95,"mitotic figure"],[301,410,18,18,0.88,"mitotic figure"]]. This raw analysis result is distributed to the result parameter calculation subunit 1331 and the detection result filtering analysis subunit 1332 for parallel processing. The result parameter calculation subunit 1331 is responsible for processing the gland segmentation mask in the raw analysis result. This subunit applies a connected component analysis algorithm to scan the 512x512 binary image, identify each independent connected region consisting of pixels with a pixel value of 1, and label each region as an independent cell nucleus instance. For example, five independent glandular structures were identified in this patch. Next, for each of the five gland instances, the subunit calculates a series of morphological and textural parameters. The morphological parameters are calculated directly based on the pixel contours of each cell nucleus instance, including: area (total number of pixels), perimeter, equivalent diameter, roundness (describing how close its shape is to a circle), and fractal dimension describing the irregularity of the gland cavity. Specifically, the area is calculated by counting the total number of pixels constituting the gland instance. The perimeter is determined by tracing the outermost pixel contours of the instance using an algorithm and summing the lengths of its boundaries. Based on the calculated area, the equivalent diameter is derived, representing the diameter of an ideal circle with the same area, thus providing a standardized dimensional measure for irregular glands. Roundness quantifies the compactness of the gland shape through a ratio combining area and perimeter; the closer the value is to 1, the closer the gland's contour is to a perfect circle. Finally, fractal dimension is used to measure the complexity and irregularity of glandular boundaries. By calculating the degree of detail filling of its contours at different scales, a higher fractal dimension value indicates that the glandular boundaries are more tortuous and complex, which has important implications in certain pathological diagnoses. After completing the calculations for all five glandular structures, this subunit obtains a list containing five entries, each of which is a numerical vector containing all the aforementioned morphological parameters. This list is the object index list. Simultaneously, the detection result filtering analysis subunit 1332 processes the list of detection results from the original analysis results. Its input is [[123,234,15,15,0.95,"mitotic image"],[301,410,18,18,0.88,"mitotic image"]]. This subunit first performs confidence threshold filtering. It uses a preset confidence threshold, such as 0.90. This threshold is set based on plotting precision-recall curves on independent validation datasets and selecting a balance point that ensures high precision (reducing false positives).In this example, the confidence scores of the two detection results are 0.95 and 0.88. The former passes the threshold, while the latter fails and is therefore filtered out. Next, Non-Maximum Suppression (NMS) is performed on the detection boxes that passed the threshold filtering to eliminate duplicate detections of the same target. The NMS algorithm requires an Intersection over Union (IoU) threshold, such as 0.5. In this example, since only one detection box remains, the NMS step does not change the result. After filtering and suppression, this subunit performs statistical analysis on the remaining valid detection boxes, i.e., counting. Therefore, the final output of this subunit is a single numerical value, the cleavage count, which is 1. Finally, the feature vectorization subunit 1333 combines the outputs of the first two subunits and performs the final encapsulation task. It receives a list of object indicators containing five glandular structure feature vectors and a cleavage count of 1 as input. Since the length of the object indicator list is variable, it needs to be converted into a fixed-length feature representation. To do this, it performs statistical aggregation on each dimension of the feature in the list. For example, for the parameter of area of ​​all five glands, it calculates the mean, standard deviation, median, minimum, and maximum of these five area values. Similarly, it performs the same statistical aggregation operation on all other calculated morphological parameters such as roundness and fractal dimension. In this way, regardless of the number of glands within a patch, their overall morphological features are condensed into a fixed set of statistical values. This subunit then concatenates all these statistical values ​​aggregated from the gland parameters with the mitotic count directly output by the detection result filtering and analysis subunit 1332 in a predefined order. For example, the concatenation order could be: [mean gland area, standard deviation of gland area, ..., mean gland roundness, standard deviation of gland roundness, ..., mitotic count]. This ultimately forms a long, purely numerical vector with a fixed dimension, which is the microscopic visual feature vector.

[0042] Specifically, the multimodal data processing module 140 is used to perform multimodal data fusion and preliminary diagnostic report generation on microscopic visual feature vectors and patient information text to obtain preliminary report data, interpretability evidence, and lesion localization data. It is understandable that although the preceding modules have successfully extracted high-resolution visual information into structured microscopic visual feature vectors, the final pathological diagnosis does not solely rely on microscopic observation. In clinical practice, pathologists need to comprehensively consider visual findings along with the patient's clinical background information (such as age, gender, medical history, tumor markers, etc.) to arrive at a comprehensive and accurate diagnostic conclusion. Therefore, in order to simulate the core thought process of a pathologist making comprehensive diagnostic decisions, this application requires multimodal data fusion and preliminary diagnostic report generation to deeply integrate the separated visual evidence with textual clinical information. Based on this fused, more complete information, multi-dimensional reasoning consistent with clinical diagnostic logic is performed, ultimately generating a preliminary pathology report that not only contains a diagnostic conclusion but also includes interpretability evidence.

[0043] In a feasible technical solution Figure 5 This is a block diagram of the multimodal data processing module in a multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application. Figure 5 As shown, the multimodal data processing module 140 includes: a visual-text fusion processing unit 141, used to perform multimodal feature alignment and deep fusion on microscopic visual feature vectors and patient information text to obtain a visual-text fusion feature vector; a diagnosis result generation unit 142, used to input the visual-text fusion feature vector into a multi-task decision head to obtain a diagnosis result, the multi-task decision head including a classification head, a grading head, and a key event detection head; and a preliminary report data generation unit 143, used to perform template-driven report filling on the diagnosis result to obtain the preliminary report data.

[0044] In a feasible technical solution, the multimodal data processing module 140 further includes: a lesion localization data designation unit 144, used to designate the coordinates of confirmed lesions as the lesion localization data; a key visual evidence extraction unit 145, used to extract key visual evidence from the original analysis results based on the lesion localization data; an extraction unit 146, used to extract key quantitative indicators from the microscopic visual feature vector based on the lesion localization data; and an evidence integration unit 147, used to integrate the key visual evidence and key quantitative indicators to obtain the interpretability basis.

[0045] Specifically, the execution process of the multimodal data processing module 140 is as follows: The entire workflow begins with the visual-text fusion processing unit 141. The patient information text may be a structured or unstructured text extracted from the electronic medical record system, for example: Patient, male, 65 years old, chief complaint: rectal bleeding for one month. Colonoscopy revealed a rectal mass, biopsy performed. No special medical history. Since visual features are numerical vectors, while patient information is natural language text, there are huge differences between the two in data structure and semantic space, making direct computation impossible. Therefore, the primary task of this unit is to use a pre-trained text encoder to map the patient information text into the same semantic space as the visual features. This text encoder can be a language model based on the Transformer architecture, such as the BioBERT model specifically fine-tuned for biomedical text. The network structure of this model consists of multiple stacked Transformer encoder layers, each containing a multi-head self-attention mechanism and a feedforward neural network. Its weights and biases are pre-trained on a massive corpus of professional linguistic data, including medical literature, clinical notes, and pathology reports, through self-supervised learning tasks such as masked language models. When the aforementioned patient information text is input into the model, the model encodes this natural language text into a high-dimensional, fixed-length text feature vector through its deep network structure. After obtaining the numerical visual feature vector and text feature vector respectively, this unit performs multimodal feature alignment and deep fusion. To address the issues of dimensionality mismatch and modality differences, two independent fully connected layers are first used to linearly project the visual feature vector and text feature vector into a common latent space of the same dimension. Next, a cross-modal attention mechanism is employed to achieve deep fusion. In this mechanism, the projected features of one modality serve as the query, and the projected features of the other modality serve as the key and value. For example, the projected visual features are used as the query to calculate their correlation with each semantic component in the text features, and the text features are weighted and summed based on these correlations to generate a visual representation rich in relevant clinical context. The reverse process is also performed. This bidirectional attention interaction allows the features of the two modalities to align and enhance each other at a deep level. Finally, the feature vectors of the two modalities, enhanced by the attention mechanism, are concatenated and then fed into a multilayer perceptron (MLP) consisting of several fully connected layers and nonlinear activation functions such as ReLU for final nonlinear transformation and information compression. The final output is a visual-text fusion feature vector, which serves as a highly condensed information carrier, containing both key features of microscopic pathological morphology and macroscopic clinical background information of the patient.

[0046] Next, the diagnosis result generation unit 142 receives this visual-text fusion feature vector and inputs it into the multi-task decision head to simultaneously output diagnostic conclusions across multiple dimensions. The multi-task decision head is a collection of multiple parallel, structurally simple but functionally specialized sub-networks. Specifically, the multi-task decision head includes the following four parts: Classification Head: This is a sub-network used to determine the nature of the lesion. Its structure is a fully connected layer followed by a Softmax activation function. The fully connected layer linearly transforms the visual-text fusion feature vector into a vector with a dimension equal to the preset number of disease categories. For example, if it is necessary to distinguish between adenocarcinoma, neuroendocrine tumor, and lymphoma, the output dimension is 3. The Softmax activation function then converts this 3-dimensional vector into a probability distribution, with the sum of all terms being 1. In this example, its output might be {"Adenocarcinoma":0.98,"Neuroendocrine tumor":0.01,"Lymphoma":0.01}. Grading Head: This is a sub-network used to determine the malignancy of the tumor. Its network structure is exactly the same as the classification head, consisting of a fully connected layer and a Softmax activation function. The fully connected layer maps the visual-text fusion feature vector to a vector with a dimension equal to the preset number of levels, such as high, medium, and low, resulting in an output dimension of 3. The Softmax function also converts this into a probability distribution. In this example, its output might be {"high differentiation": 0.1,"medium differentiation": 0.8,"low differentiation": 0.1}. Key Event Detection Head: This is a sub-network used to detect the presence of specific high-risk pathological phenomena. Its structure is a fully connected layer followed by a Sigmoid activation function. Unlike Softmax, the Sigmoid function independently calculates a probability between 0 and 1 for each output node, suitable for scenarios where multiple events may coexist. This fully connected layer maps the visual-text fusion feature vector to a vector with a dimension equal to the number of key events to be detected, such as vascular invasion and neural invasion, resulting in an output dimension of 2. In this example, its output might be {"vascular invasion": 0.85,"neurial invasion": 0.10}. It is worth noting that the weights and biases of these decision heads are learned collectively during the end-to-end training process. After completing forward inference, this unit summarizes the outputs of all decision heads. It selects the category with the highest probability for each head as the final judgment. Finally, this unit integrates these judgments into a unified, structured diagnostic result. This diagnostic result is a data object containing multiple key-value pairs, whose content explicitly includes various aspects of the diagnosis, such as: {"Diagnosis Type":"Adenocarcinoma","Tumor Grade":"Moderately Differentiated","Key Event":["Vascular Invasion"]}.

[0047] Next, the preliminary report data generation unit 143 transforms the diagnostic results into preliminary report data. The execution of this unit begins with receiving a structured diagnostic result. This input is a data object, such as: {"Diagnosis Type":"Adenocarcinoma","Tumor Grade":"Moderately Differentiated","Key Event":["Vascular Invasion"]}. This unit is a template-driven report population unit, internally containing a library of preset report templates and a set of sophisticated population rules. These templates and rules are pre-defined and solidified based on authoritative pathology reporting guidelines and the expertise of senior pathologists. First, the unit selects the most matching report template from its template library based on the diagnosis type field in the received diagnostic result. For example, if adenocarcinoma is detected, combined with the case's metadata, such as the specimen origin being a rectal biopsy, it will precisely select the colorectal biopsy adenocarcinoma pathology report template. This template is a text frame with multiple predefined placeholders, and its structure might be as follows: "Pathological Diagnosis: (Rectum, Biopsy) [Diagnosis Type], [Tumor Grade]. Microscopic Findings: Submitted as colorectal mucosal tissue. Microscopically observed atypical glandular infiltrative growth, [Microscopic Description_Structure]. Cell nuclei are [Microscopic Description_Cell], mitotic figures are readily visible. [Key Event Description]. No definite nerve invasion was observed." After selecting the template, this unit initiates the filling procedure. The filling process is divided into direct mapping filling and rule-based conditional content generation. Direct mapping filling processes placeholders whose values ​​can be directly obtained from the input data. For example, it directly fills the [Diagnosis Type] placeholder with the value adenocarcinoma corresponding to the key diagnosis type in the input object. Similarly, moderately differentiated is filled into the [Tumor Grade] placeholder. Rule-based conditional content generation handles more complex descriptive text. The rule engine of this unit selects appropriate standard descriptive terms based on the input diagnosis results. For example, for [Microscopic Description_Structure] and [Microscopic Description_Cell], the appropriate standard descriptive terms are selected. For each placeholder, the rule engine queries a pre-defined mapping table. This table defines standard descriptions corresponding to different grades. If a tumor is detected as moderately differentiated, the rule engine searches for the corresponding description. For example, for [Microscopic Description_Structure], it selects "Some glandular structures remain, some are cribriform or irregularly branched," and for [Microscopic Description_Cells], it selects "Pseudostratified, moderate atypia, increased nuclear-cytoplasmic ratio." For [Critical Event Description], it iterates through the list of critical events. If it finds vascular invasion in the list, it generates a standard description: "Microscopic view shows intravascular carcinoma." Thrombus. If the list is empty, a result of "No clear vascular invasion observed" will be generated. After all placeholders have been filled, the unit integrates the filled text fragments to generate a complete and coherent report, for example: Pathological diagnosis: (rectum, biopsy) adenocarcinoma, moderately differentiated. Microscopic findings: Submitted tissue was colorectal mucosa. Microscopically, atypical glandular infiltrative growth was observed; some glandular structures remained intact, while others showed cribriform or irregular branching. Cell nuclei showed pseudostratification, moderate atypia, increased nuclear-cytoplasmic ratio, and readily visible mitotic figures. Intravascular tumor thrombi were observed microscopically.No clear neurological damage was observed. This text represents the preliminary report data.

[0048] Meanwhile, to address the inherent black-box problem in the decision-making process of deep learning models and thus significantly improve the transparency and clinical credibility of diagnostic results, four other units in this multimodal data processing module work in parallel to generate a comprehensive and interpretable report.

[0049] First, the execution process of the lesion localization data designation unit 144 begins with receiving the output from the lesion coordinate confirmation module 120, namely the confirmed lesion coordinates. This input is a structured coordinate data object that explicitly defines one or more regions locked by the doctor in a coordinate system 40 times the size of the original image. For example, the received confirmed lesion coordinates might be (x:10240, y:20480, width:512, height:512). This data directly reflects the doctor's professional judgment and serves as the gold standard target for all subsequent evidence extraction work. Upon receiving this confirmed lesion coordinate data, it is designated as the lesion localization data.

[0050] Subsequently, the key visual evidence extraction unit 145 receives the lesion localization data and the original analysis results. First, based on the lesion localization data, this unit obtains the corresponding 512x512 pixel high-resolution patch extracted in the detail feature depth analysis module 130. This patch will serve as the canvas for generating visual evidence. Next, the unit processes the gland segmentation mask in the original analysis results. The unit renders this 512x512 binary mask with a preset, high-contrast, semi-transparent color and precisely overlays it onto the aforementioned high-resolution patch. This color and transparency are preset for optimal visual effect; for example, a red with 50% transparency is selected. This setting allows the gland outline identified by the model to be clearly delineated with a striking red, while also allowing the cell morphology and tissue structure details in the original patch below to be observed through the semi-transparent layer. Simultaneously, the unit processes the list of mitotic figure detection results in the original analysis results. It iterates through each item in the list. For each item in the list, such as [123,234,15,15,0.95,"mitotic figure"], the unit draws a brightly colored bounding box with a width and height of 15 pixels at the corresponding position on the canvas tile, i.e., with (123,234) as the top-left vertex. The color of this bounding box is also preset, for example, a bright yellow that is significantly different from the segmentation mask color, to ensure its clear visibility. This process is applied to all detection results in the list. By finally synthesizing the rendered segmentation mask layer with the tile containing all detection boxes, the unit generates a single, highly integrated composite image. This composite image is the key visual evidence, visually showing the extent of the glandular structure identified by the model (red overlay area) and the locations of all potential mitotic figures (yellow bounding boxes) within the area of ​​focus for the doctor.

[0051] Next, the key quantitative indicator extraction unit 146 receives lesion localization data and a microscopic visual feature vector. This vector is a long vector with a fixed dimension, and each element within it has a clearly defined structure, such as [mean gland area: 2500.5, standard deviation of gland area: 800.2, ..., mean gland roundness: 0.65, ..., mitotic count: 1]. This unit uses a set of filtering and mapping logic based on preset rules. These rules are pre-established based on extensive clinical pathology knowledge and statistical data analysis, aiming to associate specific diagnostic conclusions with the most indicative quantitative indicators. This unit also receives structured diagnostic results output by the diagnostic result generation unit 142, such as {"diagnostic type": "adenocarcinoma", "tumor grade": "moderately differentiated", "key event": ["vascular invasion"]}, as triggering conditions for its extraction rules. Upon receiving the above input, this unit first activates the corresponding extraction rules based on the diagnostic result of moderately differentiated adenocarcinoma. For example, a predefined rule might be defined as follows: for the diagnosis of moderately differentiated adenocarcinoma, the most explanatory key quantitative indicators are mitotic count, mean glandular roundness, and standard deviation of glandular area. Subsequently, based on the activated rule, the unit extracts the values ​​of these specific indicators from the input microscopic visual feature vector, indexed by name. It finds the mitotic count dimension in the vector and reads its value as 1. Next, it finds the mean glandular roundness dimension and reads its value as 0.65. Similarly, it finds the standard deviation of glandular area and reads its value as 800.2. Finally, the unit encapsulates these extracted indicator names and their corresponding values ​​into a structured list; this list constitutes the key quantitative indicators. For example, its output is a list containing multiple data objects: [{"Indicator Name":"Nucleus Split Count","Value":1,"Unit":"Numbers / HPF"}, {"Indicator Name":"Mean Gland Roundness","Value":0.65,"Unit":"None"}, {"Indicator Name":"Standard Deviation of Gland Area","Value":800.2,"Unit": "Pixels"}].

[0052] Finally, the evidence integration unit 147 performs the final integration of the aforementioned key visual evidence and key quantitative indicators. Upon receiving the two inputs, the unit initiates the integration process. It first creates a top-level data structure to house all the evidence; this structure can be viewed as a blueprint for generating the user interface evidence panel. This structure predefines different areas, such as a visual evidence area and a quantitative indicator area. Next, the unit directly places the received key visual evidence, i.e., the composite image, into the visual evidence area of ​​this data structure. Subsequently, it processes the list of key quantitative indicators. The unit iterates through each item in the list and populates its content into the quantitative indicator area of ​​the data structure. Simultaneously, the unit's rule engine attaches predefined clinical interpretation text to some indicators to enhance their interpretability. For example, when processing {"indicator name":"mitotic count","value": 1}, the rule engine matches a predefined rule that a mitotic count greater than 0 indicates tumor proliferative activity and generates the corresponding interpretation text. Similarly, for {"Indicator Name":"Mean Glandularity Roundness","Value":0.65}, the rule engine will generate an interpretation text suggesting that the glandular structure is irregular and consistent with the characteristics of a malignant tumor, based on the fact that this value is lower than the normal glandular roundness threshold (e.g., 0.85). Ultimately, this unit encapsulates all the integrated information into a single, structured data object. This data object serves as the interpretability basis. Its content might look like this: {"Evidence Panel":{"Visual Evidence":[Composite Image Data],"Quantitative Indicator":[{"Indicator":"Mitosis Count","Value":"1 / HPF","Interpretation":"Suggests active tumor proliferation"},{"Indicator":"Mean Glandularity Roundness","Value":"0.65","Interpretation":"Suggests irregular glandular structure"},...]}}.

[0053] Specifically, the report file interaction log generation module 150 is used to interactively verify and finalize the preliminary report data, interpretability evidence, and lesion localization data based on physician feedback data, resulting in the final report file and interaction log. That is, while the preceding modules have successfully generated a detailed and formatted preliminary pathology report, artificial intelligence should always be positioned as an auxiliary tool in clinical applications; the final diagnostic responsibility must rest with the licensed pathologist. A static, unmodifiable machine-generated report cannot be integrated into a real clinical workflow, nor can it reflect the minor adjustments or additions that physicians may make based on their comprehensive experience. Therefore, this module is designed to build a crucial bridge between automated preliminary diagnosis and the physician's final approval. It provides a fully functional interactive review environment, enabling pathologists to perform final verification, modification, and approval of the preliminary report, ensuring that their professional judgment is the sole source of the final conclusion. Simultaneously, by accurately recording every interactive operation throughout the review process, an unalterable log is formed, providing transparent quality control and accountability for the diagnostic process.

[0054] In a feasible technical solution, the execution process of the report file interactive log generation module 150 is as follows: the core function of this module is completed collaboratively by its three internal units: the interactive interface generation unit, the report modification and finalization unit, and the log recording unit.

[0055] The workflow of this module begins with the interface generation unit. This unit receives preliminary report data, interpretability evidence, and lesion localization data, and dynamically constructs an integrated review interface based on this information. Continuing with the previous example, an editable text box is displayed on the left side of the interface, which fully loads the report generated from the preliminary report data: "Pathological diagnosis: (rectal, biopsy) adenocarcinoma, moderately differentiated. Microscopic findings...". The right side of the interface displays graphic and textual information provided by the interpretability evidence, such as key visual evidence images marked with tumor areas and mitotic figures, as well as listed key quantitative indicators. Simultaneously, a browser window displaying a full-slice image is embedded at the bottom or side of the interface, and automatically focuses the field of view and highlights the lesion area initially confirmed by the physician based on the lesion localization data.

[0056] When a pathologist operates on this interface, the report modification and finalization unit begins its work. This unit continuously monitors any changes to the content within the text box. For example, after reviewing the report, the pathologist may feel the need to add a sentence to the description regarding the observation of background inflammatory cell infiltration. They may position the cursor on the microscopic observation of intravascular tumor thrombi. After this sentence, they may type, "Numerous lymphoplasmacytic cell infiltrations were observed in the stroma." The unit will immediately capture this text insertion event and its content; these operations are collectively referred to as pathologist feedback data. After the pathologist completes all modifications and clicks the "Approve and Issue" button on the interface, the unit will capture this finalization event and solidify all content within the current text box, forming the final report file. For example, the specific content of this final report file might be: Pathological diagnosis: (rectum, biopsy) adenocarcinoma, moderately differentiated. Microscopic findings: The submitted tissue was colorectal mucosa. Microscopic findings showed atypical glandular infiltrative growth; some glandular structures remained intact, while others presented as cribriform or irregularly branched structures. The cell nuclei showed pseudostratification, moderate atypia, increased nucleoplasm-to-nucleus ratio, and readily visible mitotic figures. Microscopically, intravascular tumor emboli were observed, and abundant lymphoplasmacytic infiltration was seen in the stroma. No definite neurological invasion was observed.

[0057] Meanwhile, the logging unit runs silently in the background throughout the entire interaction, accurately recording every meaningful operation. It's an event-driven logger that records the initial version of the loaded preliminary report data as a baseline at the moment the interactive interface is generated. When the report modification and finalization unit captures the event of a doctor inserting text, the logging unit immediately generates a structured log entry, precisely recording the time of the event, the doctor's identity, the type of action, the location of the modification, and the specific content of the modification. Similarly, if the doctor subsequently deletes a word, a corresponding deletion log entry is generated. When the doctor finally clicks the "Approve and Issue" button, the logging unit records this final event. All these independent log entries generated chronologically are compiled together to form a complete interaction log. This interaction log is essentially a structured, immutable history of operations, completely tracing the evolution of the initial report data into the final report file. Its specific content is as follows: [{"timestamp":"2025-08-19T14:29:00Z","action":"load_report","version":"initial"},{"timestamp":"2025-08-19T14:30:15Z","userID":"Dr.Chen","action":"insert_text","details":{"position":112,"content":",Many lymphoplasmacytic infiltrations were observed in the interstitium"}},{"timestamp":"2025-08-19T14:31:00Z"}]. This content demonstrates a complete interaction log, precisely tracing the report's modification history in chronological order. The record begins at 14:29 with the loading of the initial version of the report, followed by a clear description of the specific modifications Dr. Chen made at 14:30:15, namely, inserting the text "numerous lymphoplasmacytic infiltration seen in the interstitium" at the designated location, and concluding the entire review process with the final draft.

[0058] Finally, the module outputs the final report file generated by the report modification and finalization unit, and the interaction log generated by the log recording unit, which contains all operation records from report loading to final issuance. This final report file can be directly archived into the hospital's electronic medical record system, while the interaction log is stored as important metadata, providing detailed and traceable data support for subsequent medical quality management, process auditing, and even continuous optimization of algorithm models.

[0059] In summary, the multimodal human-computer interaction application system 100 based on the pathologist's perspective, as described in this application, is illustrated. It constructs a closed-loop system that deeply simulates the "global scanning-local focusing-comprehensive diagnosis" workflow of a pathologist. To address the disconnect between existing technologies and clinical practice, the system first generates a candidate lesion heatmap through rapid scanning of whole-slice images, simulating a doctor's low-magnification global browsing and guiding the doctor to interactively identify key areas based on their professional judgment. This significantly improves the processing efficiency of ultra-large images. Subsequently, the system performs high-resolution deep analysis only on lesions confirmed by the doctor and performs multimodal fusion of the extracted microscopic visual features with patient text information to generate a preliminary report with interpretable evidence, solving the problem of multimodal functions deviating from core clinical tasks. Finally, doctors can review and finalize the report through intuitive interaction, ensuring the doctor's dominant position and final decision-making power in the diagnostic process, thereby overcoming the bottlenecks of traditional AI systems' "black box" operation and low clinical acceptance.

[0060] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive. Furthermore, it is not limited to the disclosed implementations, and many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations.

Claims

1. A multi-modal human-computer interaction application system based on the perspective of a pathologist, characterized in that, The method comprises the following steps: a full-slice image scanning module for performing global feature rapid scanning on the acquired full-slice digital image data to obtain a pyramid structure image, a candidate lesion heat map and a candidate lesion list; a lesion coordinate confirmation module for superimposing the candidate lesion heat map on the thumbnail of the pyramid structure image on an interactive interface to display the candidate lesion heat map and the candidate lesion list, so that a pathologist can interactively lock suspicious areas through doctor interaction data including frame selection, list item clicking or text instruction inputting, thereby obtaining confirmed lesion coordinates; a detailed feature deep analysis module for extracting a high-resolution image block from the pyramid structure image based on the confirmed lesion coordinates, and performing detailed feature deep analysis on the high-resolution image block to obtain a microscopic visual feature vector; a multi-modal data processing module for performing multi-modal data fusion on the microscopic visual feature vector and patient information text, and generating a preliminary diagnosis report to obtain preliminary report data, explainability basis and lesion positioning data; a report file interactive log generation module for performing interactive verification and report finalization on the preliminary report data, explainability basis and lesion positioning data based on doctor feedback data to obtain a final report file and an interactive log; The detailed feature deep analysis module comprises: a text instruction judgment and analysis unit for determining whether there is a text instruction in the doctor interaction data, and performing instruction intent analysis on the text instruction to obtain a context analysis instruction in response to the existence of the text instruction; an original analysis result generation unit for performing context-aware model reasoning and original feature extraction on the high-resolution image block based on the context analysis instruction to obtain an original analysis result, wherein the original analysis result comprises a segmentation mask and a detection result; a feature quantization and structured vector packaging unit for performing feature quantization and structured vector packaging on the original analysis result to obtain the microscopic visual feature vector.

2. The multi-modal human-computer interaction application system based on the perspective of a pathologist according to claim 1, characterized in that, The full-slice image scanning module comprises: an image downsampling unit for performing step-by-step downsampling on the full-slice digital image to obtain the pyramid structure image; a low-resolution layer extraction unit for extracting a preset low-resolution layer from the pyramid structure image as a scanning layer; a candidate lesion information generation unit for performing parallel feature extraction and heat map generation on the scanning layer to obtain the candidate lesion heat map and the candidate lesion list.

3. The multi-modal human-computer interaction application system based on the perspective of a pathologist according to claim 2, characterized in that, The candidate lesion information generation unit comprises: a tiling division subunit for tiling and dividing the scanning layer to obtain a tile network composed of scanning tiles; a scanning tile category probability generation subunit for performing feature extraction and prediction on each scanning tile in the tile network based on a lightweight visual model to obtain a category probability vector of each scanning tile; a candidate lesion heat map generation subunit for generating the candidate lesion heat map based on the category probability vector of each scanning tile in the tile network; a candidate lesion list generation subunit for generating the candidate lesion list based on a comparison between the category probability vector of each scanning tile in the tile network and a preset threshold.

4. The multi-modal human computer interaction application system based on pathologist's perspective according to claim 1, wherein, The feature quantization and structured vector packaging unit comprises: A result parameter calculation subunit is configured to calculate morphological parameters and textural parameters of each segmentation mask in the original analysis result to obtain an object index list; A detection result filtering analysis subunit is configured to perform bounding box filtering and statistical analysis on each detection result in the original analysis result to obtain a mitotic figure count; A feature vectorization subunit is configured to perform feature vectorization on the object index list and the mitotic figure count to obtain the microscopic visual feature vector.

5. The multi-modal human computer interaction application system based on pathologist's perspective according to claim 1, wherein, The multi-modal data processing module comprises: A visual-text fusion processing unit is configured to perform multi-modal feature alignment and deep fusion on the microscopic visual feature vector and the patient information text to obtain a visual-text fusion feature vector; A diagnosis result generation unit is configured to input the visual-text fusion feature vector into a multi-task decision head to obtain a diagnosis result, the multi-task decision head comprising a classification head, a grading head, and a key event detection head; A preliminary report data generation unit is configured to perform template-driven report filling on the diagnosis result to obtain the preliminary report data.

6. The multi-modal human-computer interaction application system based on the pathologist's perspective according to claim 5, characterized in that, The multi-modal data processing module further comprises: A lesion positioning data designation unit is configured to designate the confirmed lesion coordinates as the lesion positioning data; A key visual evidence extraction unit is configured to extract key visual evidence from the original analysis result based on the lesion positioning data; An extraction unit is configured to extract key quantitative indicators from the microscopic visual feature vector based on the lesion positioning data; An evidence integration unit is configured to perform evidence integration on the key visual evidence and the key quantitative indicators to obtain the explainability basis.

Images