Sky-eye insight multi-mode man-machine interaction application system based on pathologist view angle

Abstract

The invention relates to the field of man-machine interaction, and particularly discloses a multi-mode man-machine interaction application system for sky-eye insight based on a pathologist visual angle, which is characterized in that firstly, a candidate focus thermodynamic diagram is generated through rapid scanning of a full-slice image, low-power lens global browsing of a doctor is simulated, and the doctor is guided to lock a key area interactively through professional judgment; therefore, the processing efficiency of the oversized image is greatly improved. And then, the system only performs high-resolution deep analysis on the focus confirmed by the doctor, and performs multi-modal fusion on the extracted microscopic visual features and the patient text information to generate a preliminary report with an interpretable basis, so that the problem that the multi-modal function deviates from a clinical core task is solved. Finally, the doctor can check and finalize the report through visual interaction, and the dominant position and the final decision making right of the doctor in the diagnosis process are ensured, so that the bottlenecks of black box operation and low clinical acceptability of a traditional AI system are overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human-computer interaction, and more specifically, to a multi-modal human-computer interaction application system based on the perspective of pathologists. BACKGROUND

[0002] As the "gold standard" of clinical medicine, the accuracy and efficiency of pathological diagnosis are crucial to the development of treatment plans for patients. However, the current pathological diagnosis field is facing severe challenges: on the one hand, there is a severe shortage of pathologists and a huge workload; on the other hand, whole slide images (WSI) are large in size (up to billions of pixels), and manual review is time-consuming and laborious. To alleviate this contradiction, AI-assisted diagnosis technology has emerged, but existing technologies still have some limitations when applied to clinical practice.

[0003] Existing pathological AI-assisted diagnosis systems, although they have made progress in image processing speed and identification of certain specific tasks, their design philosophy is often centered on technical implementation and fails to fully adapt to the actual workflow and thinking patterns of pathologists. In clinical practice, pathologists follow the review habit of "first scanning globally and then focusing locally", that is, first quickly browsing the entire slide under a low-power lens to find suspicious areas, and then switching to a high-power lens to carefully observe the cellular morphology, structure, and other detailed features of the lesions. However, many AI models use strategies such as brute-force computing or grid analysis on the entire high-resolution image, which not only causes a lot of computational resource redundancy, but also contradicts the intuitive workflow of doctors. In addition, existing technologies in multi-modal data fusion often stop at the level of visual and textual feature splicing, failing to deeply integrate and collaboratively analyze the multi-dimensional tasks that doctors are truly concerned about in the diagnosis process, such as tumor grading, staging, and key pathological event detection (such as cancer embolus and nerve invasion). More importantly, the current human-computer interaction mode of the system is generally rigid, with a "black box" one-way result output, and doctors cannot guide, intervene, and verify the analysis process of AI through their familiar medical terminology or regular review operations (such as framing, zooming, and annotating), resulting in a lack of flexibility and explainability in the diagnosis process, making it difficult to gain the full trust and adoption of clinicians. At its root, the above defects stem from the fact that existing technical solutions fail to take "the perspective of pathologists" as the core starting point for system design. These systems lack simulation of the doctor's review habits, deep understanding of the diagnosis logic, and respect for operational intuition, resulting in a gap between technical functions and clinical needs that is difficult to overcome.

[0004] Therefore, how to build a work mode that can deeply simulate the doctor's global insight and local precision, integrate multi-modal information, and provide an intuitive interaction mode for doctors has become a key technical bottleneck for promoting digital pathology AI technology to truly land in clinical practice and improving diagnosis quality and efficiency. SUMMARY

[0005] To solve the above technical problems, the present application is proposed. According to the application, a kind of multi-modal man-machine interaction application system of pathological doctor perspective all-seeing insight is provided, comprising: full slice image scanning module, for the global feature of the obtained full slice digital image data is scanned quickly to obtain pyramid structure image, candidate lesion heat map and candidate lesion list.

[0006] Lesion coordinate confirmation module, for the suspicious area interactive locking based on doctor interaction data to the pyramid structure image, candidate lesion heat map and candidate lesion list to obtain confirmed lesion coordinates.

[0007] Detail feature depth analysis module, for extracting high-resolution tiles from the pyramid structure image based on the confirmed lesion coordinates, and performing detail feature depth analysis on the high-resolution tiles to obtain microscopic visual feature vectors.

[0008] Multi-modal data processing module, for multi-modal data fusion and preliminary diagnosis report generation on microscopic visual feature vectors and patient information text to obtain preliminary report data, explainability basis and lesion positioning data.

[0009] Report file interaction log generation module, for interactive verification and report finalization on preliminary report data, explainability basis and lesion positioning data based on doctor feedback data to obtain final report file and interaction log.

[0010] Compared with the prior art, the multi-modal man-machine interaction application system of pathological doctor perspective all-seeing insight provided by the present application constructs a closed-loop system that deeply simulates the "global scanning-local focusing-comprehensive diagnosis" workflow of pathological doctors. To solve the problem of disconnection between the prior art and the clinic, the system first generates a candidate lesion heat map through fast scanning of full slice images, simulates the doctor's low-power microscope global browsing, and guides the doctor to interactively lock the key area with his professional judgment, which greatly improves the processing efficiency of super large images. Subsequently, the system only performs high-resolution depth analysis on the lesions confirmed by the doctor, and performs multi-modal fusion on the extracted microscopic visual features and patient text information to generate a preliminary report with explainability basis, solving the problem of multi-modal function deviating from the core task of the clinic. Finally, the doctor can review and finalize the report through intuitive interaction, ensuring the doctor's leading position and final decision-making power in the diagnosis process, thereby overcoming the bottleneck of "black box" operation of traditional AI systems and low clinical acceptance. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description thereof taken in conjunction with the accompanying drawings, in which: The accompanying drawings provide exemplary embodiments of the application and serve as an aid in understanding the application. They are a part of and constitute portions of specific embodiments of the application. These drawings are not intended to limit the scope of the application and are merely included to describe and explain illustrative embodiments of the present application. The drawings include the following figures.

[0012] Figure 1 Block diagram of the multi-modal human-computer interaction application system based on the perspective of a pathologist according to an embodiment of the present application.

[0013] Figure 2 Data flow diagram of the multi-modal human-computer interaction application system based on the perspective of a pathologist according to an embodiment of the present application.

[0014] Figure 3 Block diagram of the full-slice image scanning module in the multi-modal human-computer interaction application system based on the perspective of a pathologist according to an embodiment of the present application.

[0015] Figure 4 Block diagram of the detail feature deep analysis module in the multi-modal human-computer interaction application system based on the perspective of a pathologist according to an embodiment of the present application.

[0016] Figure 5 Block diagram of the multi-modal data processing module in the multi-modal human-computer interaction application system based on the perspective of a pathologist according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It should be understood that the drawings and embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0018] It should be understood that the various steps in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0019] The present application is proposed in view of the problems in the prior art. Figure 1 Block diagram of the multi-modal human-computer interaction application system based on the perspective of a pathologist according to an embodiment of the present application. Figure 2This is a schematic diagram of the data flow in the SkyEye Insight multimodal human-computer interaction application system based on a pathologist's perspective, according to an embodiment of this application. Specifically, as shown... Figure 1 and Figure 2 As shown in the embodiment of this application, the Sky Eye Insight Multimodal Human-Computer Interaction Application System 100 based on the pathologist's perspective includes: a whole-slice image scanning module 110, used to perform global feature rapid scanning on the acquired whole-slice digital image data to obtain a pyramid structure image, a candidate lesion heatmap, and a candidate lesion list; a lesion coordinate confirmation module 120, used to interactively lock suspicious areas based on doctor interaction data in the pyramid structure image, candidate lesion heatmap, and candidate lesion list to obtain the coordinates of confirmed lesions; and a detailed feature deep analysis module 130, used to perform deep analysis based on the coordinates of confirmed lesions. The system extracts high-resolution patches from the pyramid structure image and performs in-depth analysis of the detailed features of the high-resolution patches to obtain microscopic visual feature vectors; a multimodal data processing module 140 is used to perform multimodal data fusion of the microscopic visual feature vectors and patient information text to generate a preliminary diagnostic report to obtain preliminary report data, interpretability evidence, and lesion localization data; and a report file interaction log generation module 150 is used to interactively verify and finalize the preliminary report data, interpretability evidence, and lesion localization data based on doctor feedback data to obtain the final report file and interaction log.

[0020] Specifically, the whole-slice image scanning module 110 is used to perform a rapid global feature scan on the acquired whole-slice digital image data to obtain a pyramid structure image, a candidate lesion heatmap, and a candidate lesion list. It is understandable that in pathological diagnostic practice, facing whole-slice digital images with billions of pixels, directly performing high-resolution detailed analysis would not only consume enormous computational resources, leading to low processing efficiency, but also contradict the pathologist's habit of viewing images globally first and then locally. Doctors typically first perform a rapid scan under low magnification to quickly locate suspicious lesion areas from a macroscopic perspective, and then switch to high magnification for detailed observation. Existing technologies often neglect this core workflow, resulting in a disconnect between technology and clinical application. Therefore, setting up a full-slice image scanning module can accurately simulate the doctor's rapid global scanning behavior under low magnification. By performing dimensionality reduction processing and rapid feature extraction on the image, a navigation map containing the location and probability of potential lesions, namely a candidate lesion heatmap and list, can be efficiently generated. This allows subsequent high-precision, high-energy-consuming analysis to focus on a few high-value areas, thereby greatly improving the overall efficiency of the diagnostic process while ensuring that no key information is missed, and keeping the starting point of human-computer interaction consistent with the doctor's operational intuition.

[0021] In a feasible technical solution Figure 3A block diagram of a full slide image scanning module in a pathological doctor perspective based all-seeing insight multi-modal human-computer interaction application system according to an embodiment of the present application. As shown in Figure 3 The full slide image scanning module 110 includes an image down-sampling unit 111 configured to perform step-by-step down-sampling on a full slide digital image to obtain the pyramid structure image; a low resolution layer extraction unit 112 configured to extract a preset low resolution layer from the pyramid structure image as a scanning layer; and a candidate lesion information generation unit 113 configured to perform parallelized feature extraction and heat map generation on the scanning layer to obtain the candidate lesion heat map and the candidate lesion list.

[0022] Specifically, the execution process of the full slide image scanning module 110 is as follows: first, a full slide digital image needs to be obtained. It is also called a digital pathology slide (WSI), which is an ultra-high resolution digital image generated by scanning a physical tissue specimen (e.g., a tissue fixed with formalin, embedded with paraffin, sectioned, and stained with hematoxylin-eosin) on a slide under a high-power objective lens (usually 20x or 40x) with high precision and full field of view by a dedicated full slide scanning device. This image completely reproduces all histological information on the physical slide, and the data volume is extremely large. For example, a typical colorectal cancer tissue section may generate a digital image data with an original resolution of 120000x90000 pixels and a file size of several gigabytes (GB) after scanning under a 40x objective lens.

[0023] Next, the image downsampling unit 111 receives the high-resolution full-slice digital image data and initiates a stepwise downsampling process to generate a pyramid structure of images. This process applies a digital image processing algorithm to reduce the resolution of the image step by step. The algorithm is chosen to reduce the image size while minimizing information distortion, preserving key tissue structures and texture features. Commonly used algorithms include bilinear interpolation, bicubic interpolation, or more complex Lanczos interpolation. Bicubic interpolation is a preferred solution in this embodiment because it strikes a good balance between image quality and computational efficiency. The algorithm calculates the gray value of a new pixel by taking a weighted average of the gray values of the 16 neighboring pixels around the sampling point, generating a downsized image that is smoother and less jagged. In detail, a fixed downsampling factor is set, which is set to 2, corresponding to the conventional relationship between the magnification of the microscope objective, such as 40x, 20x, 10x. The following will take the above-mentioned 120000x90000 pixel original image corresponding to 40x as an example to illustrate. First, the original image is defined as the 0th layer (Level 0) of the pyramid. Next, the image downsampling unit 111 applies the bicubic interpolation algorithm to the 0th layer image with a downsampling factor of 2, processing in both horizontal and vertical directions. When calculating the value of each new pixel in the target image, the algorithm first maps its coordinates back to the coordinate space of the original high-resolution image. Then, the mapping point is centered, and a 4x4 pixel matrix, i.e., a total of 16 neighboring pixel points, around it is selected as the basis for calculation. The core of bicubic interpolation is that it performs weighted summation on the color or gray value of the 16 pixel points, and the weight value is determined based on a cubic polynomial function. The function calculates the contribution of each neighboring pixel point based on its exact distance to the center mapping point, and the closer the pixel point, the higher the weight. In this way, the algorithm not only considers the pixel value itself, but also indirectly utilizes the gradient information of the local pixel value, making the interpolation result continuous in both value and first derivative. This ensures that during the process of reducing the resolution, the edges of the image remain smooth, and the tiny texture details are better preserved, generating a new image with a resolution of 60000x45000 pixels, which is visually equivalent to an image observed under a 20x objective. It is defined as the 1st layer (Level 1) of the pyramid. Then, the unit continues to perform the same downsampling operation on the 1st layer image to obtain an image with a resolution of 30000x22500 pixels, which is equivalent to the 2nd layer (Level 2) under a 10x lens. This process is iteratively executed to generate lower-resolution image layers such as the 3rd layer (5x, 15000x11250 pixels), the 4th layer (2.5x, 7500x5625 pixels), etc.The downsampling process continues until the image resolution is reduced to a preset minimum level, determined by the lowest resolution at which the complete profile of the tissue section can be clearly shown on a standard display, for example equivalent to 1.25x or lower, at which point the image is mainly used to show the overall profile and location of the tissue section on the slide. In this way, the original single high-resolution image is converted into a collection of images at multiple different resolution levels. This structured collection of data is the pyramid structure image. This data structure greatly optimizes the viewing experience, when a region of the image needs to be viewed at a certain magnification (e.g. 10x), the data for the corresponding region can be directly decoded from the 2nd level of the pyramid, without the need for real-time scaling from the vast amount of data in the 0th level, thus enabling smooth zooming and panning operations.

[0024] Next, the low-resolution layer extraction unit 112 extracts a preset low-resolution layer from the pyramid structure image generated in the previous step as the scan layer. The operation of this unit is extraction, and which layer is extracted is determined by a preset value. The preset low-resolution layer level is based on a comprehensive consideration of clinical pathology knowledge and algorithm performance optimization. In this embodiment, for the task of colorectal cancer screening, the preset extraction level is determined to be the 3rd level, which corresponds to a magnification of 5x. The operation of this unit is to locate and read all the image data of this level from the pyramid structure image data file according to the index (i.e. level number 3), thus obtaining an image with a resolution of 15000x11250 pixels. The extracted image is the scan layer. Compared with the original 40x image, the data amount has been significantly reduced, suitable for fast parallel processing; at the same time, compared with lower magnification layers, it still retains sufficient morphological features to distinguish between normal glands and cancerous gland clusters.

[0025] After receiving the scan layer, the candidate lesion information generation unit 113 starts a parallel processing flow to generate a candidate lesion heat map and a candidate lesion list. In a feasible technical solution, the candidate lesion information generation unit 113 includes: a tiling division subunit 1131 for tiling division of the scan layer to obtain a tile network composed of scan tiles; a scan tile class probability generation subunit 1132 for feature extraction and prediction based on a lightweight visual model for each scan tile in the tile network to obtain a class probability vector of each scan tile; a candidate lesion heat map generation subunit 1133 for generating the candidate lesion heat map based on the class probability vector of each scan tile in the tile network; and a candidate lesion list generation subunit 1134 for generating the candidate lesion list based on a comparison between the class probability vector of each scan tile in the tile network and a preset threshold.

[0026] The process is first performed by the tiling subunit 1131. The input received by this subunit is the scan layer determined in the previous stage, such as an image with a resolution of 15000x11250 pixels corresponding to a 5x magnification. Since the subsequent deep learning model processes fixed-size inputs, and in order to achieve large-scale parallel computing, it is not feasible to directly process the entire scan layer. Therefore, the tiling subunit 1131 discretizes the continuous image. It adopts a sliding window strategy with overlap to divide the entire scan layer into a structured network composed of a large number of scan tiles. In a specific embodiment, the size of the tiles is set to 256x256 pixels, which is a good balance between capturing local histological features and controlling computational load. In order to prevent cutting important features at the tile boundary, such as an abnormal glandular structure spanning two tiles, an overlap value of pixels is set during tiling, which is determined by experimental optimization, for example, 32 pixels. That is, each tile has a shared area of 32 pixels in the horizontal and vertical directions with its adjacent tiles. After this processing, the 15000x11250 pixel scan layer is decomposed into a tile network containing thousands of 256x256 pixel scan tiles. This network not only contains the image data of all the tiles, but also accurately records the coordinate position of each tile in the original scan layer.

[0027] Subsequently, the scan patch class probability generation subunit 1132 processes each scan patch in the patch network. A pre-trained lightweight visual model is integrated inside this subunit. To meet the demand of processing thousands of patches in a very short time, in one technical solution of the present application, an efficient MobileNetV3 convolutional neural network architecture is adopted, the core advantage of which lies in the design of depth separable convolution and inverted residual block. Specifically, depth separable convolution decomposes standard convolution into two steps: first, depth convolution, which uses a set of independent filters to extract spatial features from each channel of the input data; then pointwise convolution, which linearly combines the features of each channel output by depth convolution through a 1x1 convolution kernel. This decomposition not only maintains the receptive field and feature extraction capability, but also greatly reduces the parameter quantity and floating point operation count of the model. The inverted residual block is an innovative bottleneck structure, which first expands the channel number of the input feature map through a 1x1 pointwise convolution, then uses a lightweight depth convolution to transform the features in a higher-dimensional space, and finally compresses the channel number back to the original dimension through a 1x1 pointwise convolution and adds it to the original input feature map through a short circuit connection. In this way, with the short circuit connection, the model not only ensures that it can learn rich feature representations, but also effectively prevents the gradient vanishing problem in deep network training. The parameters such as weights and biases of the model are obtained through supervised learning on a large-scale, pathologist-annotated patch dataset. This dataset contains a large amount of 256x256 pixel patches extracted from whole-slide digital images, each of which is assigned a clear class label, such as normal colorectal gland, high-grade intraepithelial neoplasia, adenocarcinoma, mucinous adenocarcinoma, etc. During training, a cross-entropy loss function is used to measure the difference between the model's prediction and the true label, and an Adam optimizer is used to iteratively adjust the model's weights and biases through a backpropagation algorithm until the model reaches the pre-set classification accuracy, sensitivity, and specificity indicators on an independent validation set. In actual execution, each scan patch in the patch network is input into this trained lightweight visual model. As the image data flows through the network backbone stacked with multiple inverted residual blocks, its spatial dimension gradually decreases, while its feature dimension gradually increases, thereby being abstracted into a set of high-dimensional feature maps. At the end of the network, a global average pooling layer is used to convert the two-dimensional feature map into a one-dimensional feature vector, which is then fed into a fully connected layer and a Softmax activation function. The Softmax function is responsible for converting the output values of the fully connected layer into a probability distribution, ultimately outputting a class probability vector. The dimension of this vector is exactly the same as the number of classes set during training, and the sum of all elements in the vector is 1.For example, for an input patch, the output class probability vector can be [Normal Colorectal Gland: 0.02, High-grade Intraepithelial Neoplasia: 0.04, Adenocarcinoma: 0.91, Mucinous Adenocarcinoma: 0.03]. This vector indicates that the model judges the patch to be Normal Colorectal Gland with 2% confidence, High-grade Intraepithelial Neoplasia with 4% confidence, Adenocarcinoma with 91% confidence, and Mucinous Adenocarcinoma with 3% confidence. This process is executed in parallel on multiple computing cores, ensuring high-throughput processing of the entire patch network. The final output is a set of class probability vectors corresponding one-to-one with the patches in the patch network.

[0028] Based on these probability vectors, the candidate lesion heat map generation subunit 1133 begins its work. In order to accurately translate these probability information into a continuous and intuitive visual image, especially in the presence of overlapping regions introduced by the tiling segmentation, the subunit first constructs a floating-point probability map in memory that is identical in size to the original scan layer, i.e. 15000 x 11250 pixels, and initializes all pixel values in this map to zero. Subsequently, the subunit initiates a systematic traversal process that goes through each scan tile in the tile network and its corresponding class probability vector. For each tile, it first reads the exact starting coordinates of the tile on the scan layer from the metadata of the tile network, and extracts the probability value for the target lesion class, e.g. adenocarcinoma, from its probability vector. Then, it plots this probability value onto the probability map. Specifically, for each pixel position covered by the 256 x 256 pixel tile, the subunit reads the existing value at this position in the probability map, compares it with the adenocarcinoma probability value from the current tile, and writes the larger of the two values back to the pixel position. By performing this maximum value projection operation, when a pixel is covered by multiple overlapping tiles, its final value in the probability map will be the highest lesion probability given by all tiles that cover it. This mechanism ensures that even weak but consistent positive signals are effectively preserved, and the signal intensity in high-probability regions is not diluted by neighboring low-probability overlapping regions. After traversing all tiles and fully integrating their probability information into the probability map, the map becomes a data matrix that accurately reflects the lesion likelihood of each pixel. The last step is the visualization rendering. The subunit applies a pre-defined color mapping function that non-linearly or linearly correlates the continuous probability values between 0.0 and 1.0 with a specific color gradient. For example, an optimized color gradient can map 0.0 to a deep blue color, 1.0 to a bright red color, and smoothly transition the intermediate probability values through a series of colors such as cyan, green, yellow, etc. to enhance the human eye's recognition of key probability intervals. This color mapping function is applied to each pixel in the probability map, converting its floating-point probability value to a specific RGB color value. Finally, the rendered collection of pixels constitutes a candidate lesion heat map that is rich in color and information. On this map, the bright red regions indicate the areas with the highest likelihood of cancer, while the blue regions represent the normal tissue background as judged by the model, thus providing the pathologist with an intuitive and comprehensive global view of the lesion distribution.

[0029] Meanwhile, the candidate lesion list generation subunit 1134 converts the set of class probability vectors into a highly structured candidate lesion list. First, a preliminary screening is performed based on a preset threshold. The threshold is scientifically determined by plotting and analyzing the receiver operating characteristic (ROC) curve on an independent test dataset containing a large number of positive and negative samples. In the present embodiment, the threshold can be set to 0.80. The subunit iterates through all the class probability vectors of the tiles and marks the tiles in which the adenocarcinoma class probability value strictly exceeds 0.80 as positive tiles in the internal logic, thereby generating a binary positive tile distribution map covering the entire scan region. On this basis, the subunit applies a connected component analysis algorithm to all the tiles marked as positive. The algorithm systematically aggregates the positive tiles that are spatially connected or adjacent due to overlapping into individual candidate lesion regions with clinical significance, taking 8-neighbor connectivity as the criterion, i.e., a tile is considered adjacent to the tiles above, below, left, right, and the four diagonal directions. After clustering, the subunit calculates and encapsulates a series of key descriptive attributes for each candidate lesion region formed by aggregation. This includes: determining the smallest bounding box that can completely enclose all the tiles in the region and accurately mapping its coordinates back to the coordinate system of the original 40x image through magnification conversion, to facilitate subsequent high-resolution detail observation; calculating the geometric centroid coordinates of the region, also mapped back to the 40x coordinate system, to provide an accurate center point for quick navigation; counting the total number of positive tiles constituting the region, as a quantitative indicator reflecting the size of the lesion region; and calculating the maximum value of the adenocarcinoma probability of all tiles in the region, taking this value as the final confidence score of the candidate lesion, as it represents the most explicit malignant signal in the region. Finally, all these candidate lesion regions containing rich information are organized into a structured data list and strictly sorted from high to low according to the confidence score. The final generated and prioritized list, i.e., the candidate lesion list, provides a clear, quantitative, and focused examination checklist for pathologists.

[0030] Specifically, the lesion coordinate confirmation module 120 is configured to perform interactive locking of suspicious regions based on the doctor interaction data on the pyramid structure image, the candidate lesion heat map and the candidate lesion list to obtain confirmed lesion coordinates. Accordingly, the candidate lesion heat map and the list provide valuable preliminary screening results for pathological diagnosis, but these results are essentially probabilistic predictions based on algorithmic models and may contain false positives or may not fully capture the subtleties of a particular case. In clinical practice, the final diagnosis is in the hands of the pathologist. Therefore, the simple automated output cannot meet the requirements of clinical rigor and reliability. Based on this, in order to establish an interactive bridge between automated analysis and professional judgment of doctors, the present application introduces processing based on doctor interaction data to convert machine-generated candidate information into target information finally determined and confirmed by doctors, ensuring that all subsequent depth analysis with high precision and high resource consumption accurately focuses on the areas of interest to doctors, thereby seamlessly integrating the experience and wisdom of doctors into the diagnosis process, solving the fundamental problem of black box operation and lack of doctor dominance in the prior art.

[0031] In one possible technical solution, the lesion coordinate confirmation module 120 comprises: an image superposition unit 121 configured to superimpose the candidate lesion heat map on the thumbnail of the pyramid structure image in the left image area of the interactive interface; and a candidate lesion list display unit 122 configured to display the candidate lesion list in the right interactive area of the interactive interface.

[0032] Specifically, the execution process of the lesion coordinate confirmation module 120 is as follows: the execution process of the module is carried out in a dedicated interactive interface. The interface is divided into two main areas to simulate the workbench of a pathologist. First, the image superposition unit 121 is responsible for constructing and managing the image area on the left side of the interface. The unit receives the pyramid structure image and the candidate lesion heat map as input. First, a suitable level for overview is selected from the pyramid structure image as the base map, for example, the same level as the scan image used to generate the heat map, i.e., the 3rd level, 5 times, 15000x11250 pixel image. Then, the candidate lesion heat map is superimposed on the bottom map in a semi-transparent manner. The value of the transparency is adjustable, for example, preset to 50%, and the doctor can dynamically adjust it as needed to clearly see the underlying tissue structure while observing the heat map indication. This superimposed display allows the doctor to intuitively correspond the high-risk areas (red areas) prompted by the model to the real tissue morphology. The image area also supports smooth interactive operations, and the doctor can use the mouse wheel for stepless zooming (achieved by decoding different levels of the pyramid structure image in real time in the background) and dragging for panning, completely replicating the operation experience of using a physical microscope.

[0033] Meanwhile, the candidate lesion list display unit 122 is responsible for constructing and managing the interactive area on the right side of the interface. It receives the candidate lesion list as input. It displays each item in the list, i.e. each candidate lesion region, in a structured form. Each row represents a candidate lesion and clearly lists its key information, such as the number, the confidence score like 92%, and the possible preliminary classification like adenocarcinoma. The list is ordered by confidence score from high to low, so that the doctor can focus on the most suspicious regions first. When the interface is constructed, the whole module enters a state of waiting for the doctor’s interaction. The doctor can provide interaction data in a variety of ways that conform to his operational intuition, in order to lock the regions he believes need further analysis. In one possible technical solution, the doctor interaction data includes: the doctor’s selected bounding box on the pyramid structure image or the candidate lesion heat map, the doctor’s click on a certain item in the candidate lesion list, and the doctor’s input of text instructions. Specifically, one way of interaction is that the doctor clicks on a certain item in the candidate lesion list on the right side with the mouse. For example, the doctor clicks on the highest-confidence candidate lesion 1 in the list. This click event is captured by the module and immediately triggers a linkage response: in the image area on the left, the view automatically pans and zooms to center the region corresponding to candidate lesion 1 and marks it with a high-intensity rectangular box. This way of image-text linkage greatly improves the positioning efficiency. Another way of interaction is that the doctor does not rely on the list, but directly operates on the image area on the left. The doctor can use the mouse to directly frame any shape or size of region on the image according to his judgment of the superimposed image. For example, the doctor may find a region on the heat map that is not significantly marked red but its morphology is suspicious, so he manually frames the region. The coordinate information (top-left corner coordinate and width and height) of this framing operation is captured by the module in real time. In addition, more advanced text instruction interaction is also supported. The doctor can input instructions conforming to medical terminology specifications in the text input box in the interactive area, such as analyzing the vascular cancer embolus here. This way of interaction can be used in combination with the previous two ways. No matter which way of interaction the doctor adopts, his operation (click, frame, input text) will be parsed by the module as doctor interaction data. The module converts these interaction data into precise coordinate information. If the doctor clicks on a list item, the bounding box coordinate of the lesion associated with the list item is extracted. If the doctor manually frames, the coordinate of the framed region is directly adopted. These coordinates obtained directly or indirectly from the doctor’s interaction are defined as confirmed lesion coordinates. For example, the doctor clicks on candidate lesion 1, and the bounding box coordinate corresponding to it in the 40x original image coordinate system, e.g. x: 10240, y: 20480, width: 512, height: 512, is locked.

[0034] Specifically, the detailed feature depth analysis module 130 is used to extract high-resolution patches from the pyramid structure image based on the confirmed lesion coordinates, and perform detailed feature depth analysis on the high-resolution patches to obtain a microscopic visual feature vector. It should be understood that the preceding steps successfully simulate the process of a pathologist quickly scanning and identifying suspicious areas under low magnification, but this is only the first step in discovering the problem. To accurately determine the nature of the lesion, such as distinguishing between benign and malignant, and determining the tumor grade, it is necessary to rely on meticulous observation of microscopic details such as cell morphology, nuclear features, and tissue structure under high magnification. This high-resolution information is the cornerstone of diagnosis, but it exists in the form of unstructured pixels. Therefore, this module is designed to simulate the depth analysis and quantitative evaluation process after a pathologist switches to high magnification. That is, the unstructured, high-resolution image region confirmed by the doctor is transformed into a structured, quantifiable numerical vector that comprehensively describes its microscopic pathological characteristics through a series of deep learning models and image analysis algorithms, providing accurate and reliable visual evidence for subsequent multimodal data fusion and automated report generation.

[0035] In a feasible technical solution Figure 4 This is a block diagram of the detailed feature deep analysis module in the "Sky Eye Insight Multimodal Human-Computer Interaction Application System Based on a Pathologist's Perspective" according to an embodiment of this application. Figure 4 As shown, the detailed feature deep analysis module 130 includes: a text instruction judgment and parsing unit 131, used to determine whether there is a text instruction in the doctor's interaction data, and in response to the existence, to perform instruction intent parsing on the text instruction to obtain a context analysis instruction; an original analysis result generation unit 132, used to perform context-aware model reasoning and original feature extraction on the high-resolution image patch based on the context analysis instruction to obtain an original analysis result, the original analysis result including a segmentation mask and detection results; and a feature quantization and encapsulation unit 133, used to perform feature quantization and structured vector encapsulation on the original analysis result to obtain the microscopic visual feature vector.

[0036] Specifically, the execution process of the detail feature depth analysis module 130 is as follows: first, the confirmed lesion coordinates output by the lesion coordinate confirmation module 120 and the pyramid structure image generated by the full slice image scanning module 110 are received. As a doctor confirms a lesion, the confirmed lesion coordinates of the lesion in the 40 times original image coordinate system are (x: 10240, y: 20480, width: 512, height: 512). The highest resolution level, i.e. Level 0, of the pyramid structure image is accessed, which completely saves the original pixel information under 40 times magnification. Based on the received coordinates (10240, 20480), the process locates the starting point in the massive data of Level 0, and reads all pixel data in a rectangular region with a width of 512 pixels and a height of 512 pixels from the starting point. This extraction process is efficiently completed by a decoder supporting random region access of an image file format (such as SVS or tiled TIFF), avoiding loading the entire image of tens of billions of pixels.

[0037] After obtaining the high-resolution patch, the text instruction judgment analysis unit 131 starts working. For example, when the doctor selects the region with the bounding box coordinates (x: 10240, y: 20480, width: 512, height: 512) and inputs the instruction "analyze the gland structure and mitotic figure in this region" in the text box of the interactive interface, the unit first performs a judgment operation to determine whether there is a text instruction in this interactive data. This is a simple check to determine whether the text instruction field is empty. In response to determining that there is a text instruction, i.e., the field is not empty, the unit will start the instruction intent analysis function. This function is undertaken by an embedded, specially trained natural language understanding model. The specific architecture of this model can be a pre-trained language model based on Transformer, such as a BERT model fine-tuned for the biomedical text field. The model is mainly composed of an input embedding layer, multiple layers of stacked Transformer encoders, and a final classification output layer. The input embedding layer is responsible for performing word segmentation on the text instruction, such as analyzing the mitotic figure in this region, and generating an initial vector representation for each word piece that integrates word meaning, position, and segment information. These vectors are then fed into the core Transformer encoder. The encoder can capture complex dependencies between words within the instruction in parallel through its internal multi-head self-attention mechanism, thereby deeply understanding the context of the instruction. Each encoder layer also contains a feedforward neural network to further deepen the representation by performing nonlinear transformations on the features extracted by the attention mechanism. After layer-by-layer processing by multiple encoders, the final output vector is fed into the top classification layer, which maps high-dimensional features to the pre-set intent category space through a fully connected network, thereby completing the analysis from natural language to structured instructions. The weights and biases of the model, such as parameters, are obtained through supervised learning on a large dataset of medical instruction-intent annotation pairs. The dataset is composed of pairs of samples, each containing a natural language instruction that a doctor may input and its corresponding structured intent representation. For example, "analyze the mitotic figure in this region", {"task": "detection", "target": "mitotic_figure"} and "accurately outline the gland structure", {"task": "segmentation", "target": "gland"}. During training, the classification loss (such as cross-entropy loss) between the model's predicted intent and the true label intent is minimized, and the model parameters are iteratively updated using optimizers such as Adam until the model can accurately extract the core task and target entity from natural language. In actual execution, the obtained text instruction "analyze the mitotic figure in this region" is input into this trained model.The model understands the contextual relationship of the words in the sentence through its multi-layer self-attention mechanism, and finally identifies the corresponding core task detection and the corresponding target entity mitotic figure. The model outputs a structured JSON object: {"tasks":[{"type": "segmentation","target":"gland"},{"type":"detection","target":"mitotic_figure"}]}. If the judgment result is that there is no text instruction, that is, the doctor only performs the operation of framing or clicking, without providing any text information, this unit will generate a preset and general context analysis instruction. This default instruction aims to perform a comprehensive basic pathology analysis, such as: {"task":"general_analysis"}.

[0038] Next, the original analysis result generation unit 132 performs targeted model inference and original feature extraction on the high-resolution tile based on the received context analysis instruction. This unit internally maintains a model library, which pre-stores multiple deep learning models trained for different pathological tasks. It is a conditional judgment and model scheduling process based on the context analysis instruction. If the context analysis instruction received by this unit is a clear composite task instruction: {"tasks":[{"type":"segmentation","target":"gland"},{"type":"detection","target":"mitotic_figure"}]} and a 512x512 pixel high-resolution tile. For the first task in the instruction, that is, {"type":"segmentation","target":"gland"}, the unit will load a U-Net model for semantic segmentation from the model library, which aims to accurately outline the gland structure. The model has a symmetric encoder-decoder structure, the encoder gradually extracts deep features and reduces the feature map size through a series of convolution and pooling operations, and the decoder gradually restores the image resolution through upsampling and convolution operations, and fuses the shallow and deep features of the corresponding levels of the encoder through jump connection, so as to realize accurate positioning of the target boundary. The parameters such as weights and biases of the model are trained on a large number of tile data sets with gland outlines carefully outlined by pathologists, by optimizing the Dice loss function and other indicators. When executing, the 512x512 pixel high-resolution tile is input into the U-Net model, and the model output is a 512x512 pixel binary segmentation mask equal to the original tile, where the pixel value of the recognized gland area is 1 and the pixel value of the background area is 0. At the same time, for the second task in the instruction, that is, {"type":"detection","target":"mitotic_figure"}, the unit will load a high-efficiency single-stage target detection model such as YOLOv8, which is specially used for mitotic figure detection. It mainly includes three parts: a backbone network for extracting multi-level features from the input tile, a neck network (such as PANet structure) for fusing different levels of features to enhance the detection ability of different sizes of targets, and a head network for predicting the fused features to output bounding boxes and class probabilities. Similarly, the weights and biases of the model are obtained by supervised training on a large-scale, carefully annotated dataset by experienced pathologists. In actual execution, the 512x512 pixel high-resolution tile is sent into the loaded YOLOv8 model for one forward inference.The model outputs a list of detection results, each of which represents a detected potential karyokinesis image and contains its bounding box coordinates, confidence score, and class label. For example, the output can be [[123, 234, 15, 15, 0.95, "karyokinesis image"], [301, 410, 18, 18, 0.88, "karyokinesis image"]]. When receiving the default {"task": "general_analysis"} instruction, the unit will call multiple models in parallel or serially. For example, it will perform the above-mentioned gland segmentation and karyokinesis image detection tasks simultaneously. Finally, the unit will systematically collect the outputs generated by the two parallel tasks, i.e., the gland segmentation mask generated by the U-Net model and the list of karyokinesis image detection results generated by the YOLOv8 model, to form a unified and structured raw analysis result. This result is generated in a single execution based on the specific composite instruction input by the doctor and explicitly contains pixel-level segmentation masks generated by the segmentation model and detection result lists containing bounding box coordinates and confidence scores generated by the detection model.

[0039] Finally, the feature quantization packaging unit 133 deeply processes this complex raw analysis result, transforming it from qualitative image-level information to quantitative and structured numerical features. In a feasible technical solution, the feature quantization packaging unit 133 includes: a result parameter calculation sub-unit 1331 for calculating morphological parameters and texture parameters of each segmentation mask in the raw analysis result to obtain an object index list; a detection result filtering analysis sub-unit 1332 for performing bounding box filtering and statistical analysis on each detection result in the raw analysis result to obtain a karyokinesis image count; and a feature vectorization sub-unit 1333 for feature vectorization of the object index list and the karyokinesis image count to obtain the microscopic visual feature vector.

[0040] In particular, the execution of this unit begins with receiving the raw analysis results, which contains a 512x512 pixel gland segmentation mask generated by the U-Net model, and a detection result list containing two potential karyokineses [[123, 234, 15, 15, 0.95, "karyokinesis"], [301, 410, 18, 18, 0.88, "karyokinesis"]] generated by the YOLOv8 model. This raw analysis result is distributed to the result parameter calculation subunit 1331 and the detection result filtering analysis subunit 1332 for parallel processing. The result parameter calculation subunit 1331 is responsible for processing the gland segmentation mask in the raw analysis results. This subunit applies a connected component analysis algorithm to scan the 512x512 binary image, identifies each independent connected region composed of pixels with a value of 1, and labels each region as an independent nucleus instance. As identified in this block, there are 5 independent gland structures. Then, for each of the 5 gland instances, the subunit calculates a series of morphological parameters and texture parameters. The calculation of morphological parameters is directly based on the pixel contour of each nucleus instance, including: area (total number of pixels), perimeter, equivalent diameter, circularity (describing the degree of its shape close to a circle), and fractal dimension describing the irregularity of the gland lumen, etc. Specifically, the calculation of the area is to count the total number of pixels that make up the gland instance. The perimeter is determined by tracing the pixel contour of the outermost instance by algorithm and accumulating the length of its boundary. Based on the calculated area, the equivalent diameter is derived, which represents the diameter of an ideal circle with the same area, thus providing a standardized size measurement for irregular glands. The circularity quantifies the compactness of the gland shape through a ratio of area to perimeter, and the closer the value is to 1, the closer the outline of the gland is to a perfect circle. Finally, the fractal dimension is used to measure the complexity and irregularity of the gland boundary by calculating the degree of detail filling of its contour at different scales, a higher fractal dimension value means that the gland boundary is more tortuous and complex, which has important implications in some pathological diagnoses. After completing the calculation of all 5 gland structures, the subunit obtains a list containing 5 entries, each of which is a numerical vector containing all the morphological parameters described above. This list is the object indicator list. At the same time, the detection result filtering analysis subunit 1332 processes the detection result list in the raw analysis results. The input it receives is [[123, 234, 15, 15, 0.95, "karyokinesis"], [301, 410, 18, 18, 0.88, "karyokinesis"]]. The subunit first performs confidence threshold filtering. It uses a pre-set confidence threshold, for example 0.90. The setting of this threshold is based on drawing the precision-recall curve on an independent validation dataset and selecting a balance point that ensures high precision (reduces false positives).In this example, the confidence of the two detections are 0.95 and 0.88, the former passes the threshold, the latter does not, and thus is filtered out. Next, the detection boxes that pass the threshold filtering are subjected to a non-maximum suppression (NMS) algorithm to eliminate duplicate detections of the same target. The NMS algorithm requires an intersection over union (IoU) threshold, for example, 0.5. In this example, since there is only one detection box left, the NMS step does not change the result. After filtering and suppression, the subunit performs statistical analysis on the final remaining valid detection box, i.e., counting. Thus, the final output of this subunit is a single numerical value, i.e., the mitotic figure count, which is 1. Finally, the feature vectorization subunit 1333 collects the outputs of the previous two subunits and performs the final encapsulation task. It receives as input the object indicator list containing 5 gland structure feature vectors, and the mitotic figure count with a value of 1. Since the length of the object indicator list is variable, it needs to be converted into a fixed-length feature representation. To this end, it performs statistical aggregation on each dimension of the feature in the list. For example, for the parameter of the area of all 5 glands, it calculates the mean, standard deviation, median, minimum, and maximum of the 5 area values. Similarly, it performs the same statistical aggregation operation on all other computed morphological parameters, such as circularity, fractal dimension, etc. In this way, regardless of how many glands are in the tile, their overall morphological features are condensed into a fixed number of statistical values. This subunit concatenates all these statistical values aggregated from the gland parameters with the mitotic figure count directly output by the detection result filtering and analysis subunit 1332 in a pre-defined order. For example, the concatenation order can be: [mean of gland area, standard deviation of gland area,..., mean of gland circularity, standard deviation of gland circularity,..., mitotic figure count]. Finally, a fixed-dimension, purely numerical long vector is formed, which is the microscopic visual feature vector.

[0041] Specifically, the multi-modal data processing module 140 is configured to perform multi-modal data fusion and preliminary diagnosis report generation on the microscopic visual feature vector and the patient information text to obtain preliminary report data, explainability basis and lesion positioning data. It can be understood that although the pre-sequencing module has successfully extracted high-resolution visual information into a structured microscopic visual feature vector, the final formation of a pathological diagnosis does not rely solely on observation under a microscope. In clinical practice, a pathologist needs to comprehensively consider visual findings and the clinical background information of a patient (such as age, gender, medical history, tumor markers, etc.) to reach a comprehensive and accurate diagnosis conclusion. Therefore, in order to simulate the core thinking process of a pathologist in making a comprehensive diagnosis decision, the present application needs to perform multi-modal data fusion and preliminary diagnosis report generation to deeply fuse the separated visual evidence and the textual clinical information, and based on the more complete information after the fusion, perform multi-dimensional reasoning conforming to the clinical diagnosis logic, and finally generate a preliminary pathological report containing not only a diagnosis conclusion but also an explainability basis.

[0042] In one possible implementation, Figure 5 A block diagram of a multi-modal data processing module in a multi-modal human-computer interaction application system based on a pathologist's perspective and all-seeing insight according to an embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, the multi-modal data processing module 140 includes a visual-text fusion processing unit 141 configured to perform multi-modal feature alignment and deep fusion on the microscopic visual feature vector and the patient information text to obtain a visual-text fusion feature vector; a diagnosis result generation unit 142 configured to input the visual-text fusion feature vector into a multi-task decision head to obtain a diagnosis result, the multi-task decision head including a classification head, a grading head and a key event detection head; and a preliminary report data generation unit 143 configured to perform template-driven report filling on the diagnosis result to obtain the preliminary report data. Figure 5

[0043] In one possible implementation, the multi-modal data processing module 140 further includes a lesion positioning data specifying unit 144 configured to specify the confirmed lesion coordinates as the lesion positioning data; a key visual evidence extraction unit 145 configured to extract key visual evidence from the original analysis result based on the lesion positioning data; an extraction unit 146 configured to extract key quantitative indicators from the microscopic visual feature vector based on the lesion positioning data; and an evidence integration unit 147 configured to perform evidence integration on the key visual evidence and the key quantitative indicators to obtain the explainability basis.

[0044] ​Specifically, the execution process of the multi-modal data processing module 140 is as follows: the entire workflow begins with the visual-text fusion processing unit 141. The patient information text can be a piece of structured or unstructured text extracted from an electronic medical record system, for example: patient, male, 65 years old, chief complaint of hematochezia for one month. Colonoscopy found a rectal mass, biopsy. No special medical history. Since the visual features are numerical vectors and the patient information is natural language text, there is a huge difference between the two in terms of data structure and semantic space, and direct calculation is not possible. Therefore, the primary task of this unit is to use a pre-trained text encoder to map the patient information text into the same semantic space as the visual features. This text encoder can be a language model based on the Transformer architecture, such as the BioBERT model specially fine-tuned for biomedical field text. The network structure of this model is composed of multiple layers of stacked Transformer encoder layers, each containing a multi-head self-attention mechanism and a feedforward neural network. Its weights and biases, etc. parameters, are pre-trained on a large amount of medical literature, clinical notes, and pathological reports, etc. professional corpus through self-supervised learning tasks such as mask language models. When the above patient information text is input into the model, the model encodes this piece of natural language text into a high-dimensional, fixed-length text feature vector through its deep network structure. After obtaining the numerical visual feature vector and the text feature vector respectively, the unit performs multi-modal feature alignment and deep fusion. In order to solve the problem of dimension mismatch and modal difference, first, through two independent fully connected layers, the visual feature vector and the text feature vector are respectively linearly projected into a common, dimensionally identical latent space. Next, a cross-modal attention mechanism is used to achieve deep fusion. In this mechanism, the projected features of one modality serve as the query (Query), and the projected features of the other modality serve as the key (Key) and value (Value). For example, the projected visual features are used as the query to calculate their relevance to the semantic components in the text features, and the text features are weighted and summed according to these relevance degrees, thereby generating a visual representation rich in relevant clinical context. This process is reversed. This two-way attention interaction allows the features of the two modalities to be aligned and enhanced at a deep level. Finally, the two modality feature vectors enhanced by the attention mechanism are concatenated and fed into a multi-layer perceptron (MLP) composed of several fully connected layers and nonlinear activation functions such as ReLU for final nonlinear transformation and information compression. The final output is a visual-text fusion feature vector, which serves as a highly condensed information carrier that contains both microscopic pathological morphological features and patient macroscopic clinical background information.

[0045] Next, the diagnosis result generation unit 142 receives this vision-text fusion feature vector and inputs it into the multi-task decision head to simultaneously output diagnosis conclusions in multiple dimensions. The multi-task decision head is a collection of multiple parallel sub-networks that are simple in structure but specialized in function. The multi-task decision head specifically includes the following four parts: the classification head: this is a sub-network for determining the nature of the lesion. Its structure is a fully connected layer followed by a Softmax activation function. The role of this fully connected layer is to linearly transform the vision-text fusion feature vector into a vector with a dimension equal to the number of preset disease categories, for example, if it is necessary to distinguish between adenocarcinoma, neuroendocrine tumor, and lymphoma, then the output dimension is 3. The Softmax activation function converts this 3-dimensional vector into a probability distribution, the sum of each item is 1. In this example, its output can be {"adenocarcinoma": 0.98, "neuroendocrine tumor": 0.01, "lymphoma": 0.01}. The grading head: this is a sub-network for judging the degree of tumor malignancy. Its network structure is exactly the same as the classification head, consisting of a fully connected layer and a Softmax activation function. The fully connected layer maps the vision-text fusion feature vector to a vector with a dimension equal to the number of preset grades, such as high, medium, and low, then the output dimension is 3. The Softmax function also converts it into a probability distribution. In this example, its output can be {"high differentiation": 0.1, "medium differentiation": 0.8, "low differentiation": 0.1}. The key event detection head: this is a sub-network for detecting whether there are specific high-risk pathological phenomena. Its structure is a fully connected layer followed by a Sigmoid activation function. Unlike Softmax, the Sigmoid function independently calculates a probability between 0 and 1 for each output node, suitable for scenarios where multiple events may exist simultaneously. The fully connected layer maps the vision-text fusion feature vector to a vector with a dimension equal to the number of key events to be detected, for example, vessel invasion and nerve invasion, then the output dimension is 2. In this example, its output can be {"vessel invasion": 0.85, "nerve invasion": 0.10}. It is worth mentioning that the parameters such as weights and biases of these decision heads are learned together in the end-to-end training process. After completing the forward inference, the unit performs summary processing on the outputs of all decision heads. It selects the class with the highest probability for each head as the final judgment. Finally, the unit integrates these judgment results into a unified and structured diagnosis result. This diagnosis result is a data object containing multiple key-value pairs, whose content clearly includes various aspects of the diagnosis, for example: {"diagnosis type": "adenocarcinoma", "tumor grade": "medium differentiation", "key events": ["vessel invasion"]}.

[0046] Then the preliminary report data generation unit 143 converts the diagnosis result into preliminary report data. The execution of this unit starts with receiving a structured diagnosis result. The input is a data object like {"diagnosis type": "adenocarcinoma", "tumor grade": "moderate", "key events": ["vascular invasion"]}. The unit is a template-driven report filling, which internally contains a logic engine with a pre-set report template library and a set of precise filling rules. These templates and rules are pre-set and solidified according to authoritative pathology report guidelines and the professional knowledge of senior pathologists. First, the unit will select a most matching report template from its template library according to the diagnosis type field in the received diagnosis result. For example, detecting adenocarcinoma and combining the case's metadata, such as the specimen source is rectal biopsy, it will accurately select the colorectal biopsy adenocarcinoma pathology report template. This template is a text framework with multiple pre-defined placeholders, and its structure may be as follows: "Pathological diagnosis: (rectum, biopsy) [diagnosis type], [tumor grade]. Microscopic findings: The submitted specimen is colorectal mucosa tissue. Microscopically, there is an atypical glandular infiltrative growth, [microscopic description of structure]. The cell nuclei are [microscopic description of cells], and mitotic figures are easily seen. [Key event description]. No clear nerve invasion is seen. After selecting the template, the unit starts the filling process. The filling process is divided into direct mapping filling and rule-based conditional content generation. Direct mapping filling handles placeholders that can directly obtain values from input data. For example, it will fill the value adenocarcinoma in the input object's key diagnosis type into the [diagnosis type] placeholder in the template. Similarly, fill moderate into the [tumor grade] placeholder. Rule-based conditional content generation handles more complex descriptive text. The unit's rule engine will select appropriate standard description terms according to the input diagnosis result. For example, for the [microscopic description of structure] and [microscopic description of cells] placeholders, the rule engine will query a pre-set mapping table. This table defines standard descriptions corresponding to different grades. Detecting that the tumor grade is moderate, the rule engine will find the corresponding description, for example, for [microscopic description of structure] select some glandular structures are still present, some are sieve-like or irregular branches, and for [microscopic description of cells] select pseudostratified, moderate atypia, and increased nuclear-cytoplasmic ratio. For [key event description], it will traverse the key event list. Finding that the list contains vascular invasion, it will generate the standard description sentence "Microscopically, there is cancer embolus in the blood vessels." If the list is empty, it will generate "No clear vascular invasion is seen." After all the placeholders are filled, the unit integrates the filled text fragments to generate a complete and coherent report text, for example: "Pathological diagnosis: (rectum, biopsy) adenocarcinoma, moderate. Microscopic findings: The submitted specimen is colorectal mucosa tissue. Microscopically, there is an atypical glandular infiltrative growth, some glandular structures are still present, some are sieve-like or irregular branches. The cell nuclei are pseudostratified, moderate atypia, and increased nuclear-cytoplasmic ratio, and mitotic figures are easily seen. Microscopically, there is cancer embolus in the blood vessels."No definite nerve invasion was seen. This text is the preliminary report data.

[0047] At the same time, in order to solve the inherent black box problem of deep learning model in the decision-making process, thereby significantly improving the transparency and clinical credibility of the diagnosis result, the other four units in the multi-modal data processing module work in parallel to generate a detailed explainability basis.

[0048] Firstly, the lesion positioning data specifying unit 144 receives the output from the lesion coordinate confirmation module 120, i.e. the confirmed lesion coordinates, as the starting point of the execution process of this unit. This input is a structured coordinate data object that clearly defines one or more regions locked by the doctor in the 40 times original image coordinate system. For example, the received confirmed lesion coordinates are (x: 10240, y: 20480, width: 512, height: 512). This data is a direct reflection of the doctor's professional judgment and is the gold standard target for all subsequent evidence extraction work. When receiving this confirmed lesion coordinate data, it is specified as the lesion positioning data.

[0049] Subsequently, the key visual evidence extraction unit 145 receives the lesion localization data and the raw analysis results. First, the unit fetches the corresponding 512x512 high-resolution tile that has been extracted in the detail feature depth parsing module 130 according to the lesion localization data, which will serve as the canvas for generating the visual evidence. Next, the gland segmentation mask in the raw analysis results is processed. The unit renders this 512x512 binary mask in a pre-set semi-transparent color with high contrast, and overlays it precisely on top of the high-resolution tile. The color and transparency are pre-set for optimal visual effect, for example, red with 50% transparency. With such a setting, the model-identified gland contour is clearly outlined in a prominent red color, and the cell morphology and tissue structure details in the underlying original tile can be observed through the semi-transparent layer. Meanwhile, the unit processes the nuclear division figure detection results list in the raw analysis results. It iterates through each item in the list. For each item in the list, for example, [123, 234, 15, 15, 0.95, "nuclear division figure"], the unit draws a 15x15 pixel boundary box in a prominent color at the corresponding position on the canvas tile, with (123, 234) as the top-left corner vertex. The color of the boundary box is also pre-set, for example, bright yellow, which is significantly different from the color of the segmentation mask, to ensure that it is clearly distinguishable. This process is applied to all detection results in the list. By performing the final image composition of the rendered segmentation mask layer and the tile with all the detection boxes drawn, the unit generates a single, highly information-integrated composite image. This composite image is the key visual evidence, which intuitively shows the model-identified gland structure range (red overlaid area) and all potential nuclear division figure locations (yellow boundary boxes) in the area of interest of the doctor.

[0050] Next, the key quantitative indicators extraction unit 146 receives the lesion localization data and the micro-vision feature vector, which is a fixed-dimension long vector with each element inside having a clear definition, such as [adenoid area mean: 2500.5, adenoid area standard deviation: 800.2,..., adenoid circularity mean: 0.65,..., mitotic figure count: 1]. The unit has a set of screening and mapping logic based on preset rules. These rules are pre-established based on a large amount of clinical pathology knowledge and data statistical analysis, aiming to associate specific diagnostic conclusions with the most indicative quantitative indicators. The unit also receives the structured diagnostic results output by the diagnostic result generation unit 142, such as {"diagnosis type": "adenocarcinoma", "tumor grade": "moderately differentiated", "key events": ["vascular invasion"]}, as the triggering condition for its extraction rules. When receiving the above inputs, the unit first activates the corresponding extraction rules according to the adenocarcinoma in the diagnostic results. For example, the preset rules may define that for the diagnosis of moderately differentiated adenocarcinoma, the most explanatory key quantitative indicators are the mitotic figure count, adenoid circularity mean, and adenoid area standard deviation. Subsequently, the unit extracts the numerical values of these specific indicators according to the activated rules, by indexing and extracting the names of these specific indicators from the input micro-vision feature vector. It finds the dimension of the mitotic figure count and reads its value 1 from the vector. Then, it finds the adenoid circularity mean dimension and reads its value 0.65. Similarly, it finds the adenoid area standard deviation and reads its value 800.2. Finally, the unit encapsulates these extracted indicator names and their corresponding numerical values into a structured list, which is the key quantitative indicators. For example, its output is a list containing multiple data objects: [{"indicator name": "mitotic figure count", "value": 1, "unit": "pieces / HPF"}, {"indicator name": "adenoid circularity mean", "value": 0.65, "unit": "none"}, {"indicator name": "adenoid area standard deviation", "value": 800.2, "unit": "pixels"}].

[0051] Finally, the evidence integration unit 147 performs the final integration of the key visual evidence and the key quantitative indicators. Upon receiving the two inputs, it initiates the integration procedure. It first creates a top-level data structure to hold all the evidence, which can be viewed as a blueprint for generating the user interface evidence panel. This structure predefines different areas, such as the visual evidence area and the quantitative indicator area. Next, the unit places the received key visual evidence, i.e., the composite image, directly into the visual evidence area of the data structure. Subsequently, it processes the list of key quantitative indicators. The unit iterates through each item in the list and populates the content into the quantitative indicator area of the data structure. While populating, the rule engine of the unit also attaches pre-defined clinical interpretation texts to some of the indicators to enhance their explainability. For example, when processing {"indicator name": "mitotic figure count", "value": 1}, the rule engine matches a pre-defined rule that a mitotic figure count greater than 0 suggests proliferative activity of the tumor and generates the corresponding interpretation text. Similarly, for {"indicator name": "mean glandularity", "value": 0.65}, the rule engine generates an interpretation text suggesting irregular glandular structure, consistent with the features of a malignant tumor, based on the fact that the value is below the normal glandularity threshold (e.g., 0.85). Finally, the unit encapsulates all the integrated information into a single, structured data object. This data object is the explainability basis. Its content can be as follows: {"evidence panel": {"visual evidence": [composite image data], "quantitative indicators": [{"indicator": "mitotic figure count", "value": "1 per HPF", "interpretation": "suggests proliferative activity of the tumor"}, {"indicator": "mean glandularity", "value": "0.65", "interpretation": "suggests irregular glandular structure, consistent with the features of a malignant tumor"},...]}}.

[0052] Specifically, the report file interactive log generation module 150 is configured to interactively verify and finalize the preliminary report data, the explainability basis and the lesion positioning data based on the doctor feedback data to obtain a final report file and an interactive log. That is, the prequel module has successfully generated a preliminary pathological report with detailed content and standardized format, but artificial intelligence should always be positioned as an auxiliary tool in clinical application, and the final diagnosis responsibility subject must be the licensed pathologist. A static and unmodifiable machine-generated report cannot be integrated into the real clinical workflow, nor can it reflect the fine-tuning or supplement that the doctor may make based on his comprehensive experience. Therefore, the setting of this module is to build a key bridge between the automated preliminary diagnosis and the final doctor's signature responsibility. Provide a fully functional interactive review environment to enable pathologists to make the final verification, modification and issuance of the preliminary report, and ensure that their professional judgment is the only source of the final conclusion. At the same time, by accurately recording each interactive operation in the entire review process, an unalterable log is formed, providing transparent quality control and responsibility traceability basis for the diagnosis process.

[0053] In one possible technical solution, the report file interactive log generation module 150 performs the following process: the core function of this module is completed by the three units inside it: the interactive interface generation unit, the report modification and finalization unit, and the log recording unit.

[0054] The workflow of this module starts with the interactive interface generation unit. This unit receives the preliminary report data, the explainability basis and the lesion positioning data, and dynamically builds an integrated review interface based on these information. Continuing the example above, the left side of the interface will display an editable text box, which contains the full text of the report generated by the preliminary report data: "Pathological diagnosis: (rectum, biopsy) adenocarcinoma, moderately differentiated. Microscopic findings...". The right side of the interface will display the graphic and textual information provided by the explainability basis, such as key visual evidence images marked with tumor areas and nuclear division images, and listed key quantitative indicators. At the same time, a full-slice image browser window will be embedded at the bottom or side of the interface, and according to the lesion positioning data, the field of view will be automatically focused and the lesion area initially confirmed by the doctor will be highlighted.

[0055] When the pathologist operates on this interface, the report modification and finalization unit starts working. This unit continuously listens to any content change in the text box. For example, after reviewing the report, the pathologist thinks that a sentence about the observation of background inflammatory cell infiltration needs to be added in the description. He positions the cursor after the sentence "Intravascular tumor embolus is seen under the microscope." Then he inputs "with massive lymphoplasmacytic infiltration in the stroma." This unit will capture this event of inserting text and its content immediately, which is collectively called pathologist feedback data. After the pathologist finishes all the modifications and clicks the "review and issue" button on the interface, this unit will capture this finalization event and solidify the entire content in the current text box to form the final report file. For example, the specific content of the final report file is as follows: Pathological diagnosis: (rectum, biopsy) adenocarcinoma, moderately differentiated. Microscopic findings: The submitted specimen is colorectal mucosal tissue. Heterotypic glands grow invasively under the microscope, with partial gland structure remaining, partial sieve-like or irregular branching. Nuclei are pseudostratified, with moderate atypia, increased nuclear-cytoplasmic ratio, and mitotic figures easily seen. Intravascular tumor embolus is seen under the microscope, with massive lymphoplasmacytic infiltration in the stroma. No clear nerve invasion is seen.

[0056] Meanwhile, the logging unit runs silently in the background throughout the entire interaction, accurately recording every meaningful operation. It's an event-driven logger that records the initial version of the loaded preliminary report data as a baseline at the moment the interactive interface is generated. When the report modification and finalization unit captures the event of a doctor inserting text, the logging unit immediately generates a structured log entry, precisely recording the time of the event, the doctor's identity, the type of action, the location of the modification, and the specific content of the modification. Similarly, if the doctor subsequently deletes a word, a corresponding deletion log entry is generated. When the doctor finally clicks the "Approve and Issue" button, the logging unit records this final event. All these independent log entries generated chronologically are compiled together to form a complete interaction log. This interaction log is essentially a structured, immutable history of operations, completely tracing the evolution of the initial report data into the final report file. Its specific content is as follows: [{"timestamp":"2025-08-19T14:29:00Z","action":"load_report","version":"initial"},{"timestamp":"2025-08-19T14:30:15Z","userID":"Dr.Chen","action":"insert_text","details":{"position":112,"content":",Many lymphoplasmacytic infiltrations were observed in the interstitium"}},{"timestamp":"2025-08-19T14:31:00Z"}]. This content demonstrates a complete interaction log, precisely tracing the report's modification history in chronological order. The record begins at 14:29 with the loading of the initial version of the report, followed by a clear description of the specific modifications Dr. Chen made at 14:30:15, namely, inserting the text "numerous lymphoplasmacytic infiltration seen in the interstitium" at the designated location, and concluding the entire review process with the final draft.

[0057] Finally, the module outputs the final report file generated by the report modification and finalization unit, and the interaction log generated by the log recording unit, which contains all operation records from report loading to final issuance. This final report file can be directly archived into the hospital's electronic medical record system, while the interaction log is stored as important metadata, providing detailed and traceable data support for subsequent medical quality management, process auditing, and even continuous optimization of algorithm models.

[0058] In summary, the pathological doctor perspective-based all-seeing insight multi-modal human-computer interaction application system 100 based on the embodiments of the present application is illustrated, which constructs a closed-loop system that deeply simulates the pathological doctor's "global scanning-local focusing-comprehensive diagnosis" workflow. To solve the problem of the existing technology being disconnected from the clinic, the system first generates a candidate lesion heat map through fast scanning of the whole slide image, simulates the doctor's low-power microscope global browsing, and guides the doctor to interactively lock the key area with his professional judgment, which greatly improves the processing efficiency of the super large image. Subsequently, the system only analyzes the high-resolution of the lesions confirmed by the doctor, and multi-modal fusion of the extracted microscopic visual features and patient text information is performed to generate a preliminary report with explainable basis, solving the problem of multi-modal function deviating from the core task of the clinic. Finally, the doctor can review and finalize the report through intuitive interaction, ensuring the doctor's leading position and final decision-making power in the diagnosis process, thereby overcoming the bottleneck of traditional AI system "black box" operation and low clinical acceptance.

[0059] The implementations of the present disclosure have been described above with the aid of functional and structural descriptions of specific implementations and appropriate comparisons to related art. The range of the above-described implementations is not limited to what has been specifically described. Many modifications and variations are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the present disclosure can be practiced otherwise than as specifically described.

Claims

1. A multimodal human-computer interaction application system based on a pathologist's perspective, characterized in that, include: The whole-slice image scanning module is used to perform a rapid global feature scan on the acquired whole-slice digital image data to obtain a pyramid structure image, a candidate lesion heat map, and a candidate lesion list; The lesion coordinate confirmation module is used to interactively lock suspicious areas based on doctor interaction data in pyramid structure images, candidate lesion heat maps and candidate lesion lists to obtain the coordinates of confirmed lesions. The detailed feature deep analysis module is used to extract high-resolution patches from the pyramid structure image based on the confirmed lesion coordinates, and perform detailed feature deep analysis on the high-resolution patches to obtain microscopic visual feature vectors; the multimodal data processing module is used to perform multimodal data fusion on the microscopic visual feature vectors and patient information text to generate a preliminary diagnostic report to obtain preliminary report data, interpretability evidence, and lesion localization data; the report file interaction log generation module is used to interactively verify and finalize the preliminary report data, interpretability evidence, and lesion localization data based on doctor feedback data to obtain the final report file and interaction log.

2. The multimodal human-computer interaction application system based on a pathologist's perspective, as described in claim 1, is characterized in that... The full-slice image scanning module includes: an image downsampling unit, used to downsample the full-slice digital image step by step to obtain the pyramid structure image; a low-resolution layer extraction unit, used to extract a preset low-resolution layer from the pyramid structure image as a scanning layer; and a candidate lesion information generation unit, used to perform parallel feature extraction and heat map generation on the scanning layer to obtain the candidate lesion heat map and candidate lesion list.

3. The Sky Eye Insight Multimodal Human-Computer Interaction Application System based on a pathologist's perspective as described in claim 2, characterized in that, The candidate lesion information generation unit includes: a tile segmentation subunit, used to segment the scan layer into tiles to obtain a tile network composed of scan tiles; a scan tile category probability generation subunit, used to perform feature extraction and prediction based on a lightweight visual model on each scan tile in the tile network to obtain the category probability vector of each scan tile; a candidate lesion heatmap generation subunit, used to generate the candidate lesion heatmap based on the category probability vector of each scan tile in the tile network; and a candidate lesion list generation subunit, used to generate the candidate lesion list based on the comparison between the category probability vector of each scan tile in the tile network and a preset threshold.

4. The Sky Eye Insight Multimodal Human-Computer Interaction Application System based on a pathologist's perspective as described in claim 1, characterized in that, The lesion coordinate confirmation module includes: an image overlay unit, used to overlay a heat map of candidate lesions onto a thumbnail of a pyramid structure image in the left image area of ​​the interactive interface; and a candidate lesion list display unit, used to display a list of candidate lesions in the right interactive area of ​​the interactive interface.

5. The multimodal human-computer interaction application system based on a pathologist's perspective, as described in claim 4, is characterized in that... The doctor interaction data includes: a box area selected by the doctor on the pyramid structure image or candidate lesion heat map, an item in the candidate lesion list clicked by the doctor, and text commands entered by the doctor.

6. The multimodal human-computer interaction application system based on a pathologist's perspective, as described in claim 1, is characterized in that... The detailed feature deep analysis module includes: a text instruction judgment and parsing unit, used to determine whether there are text instructions in the doctor's interaction data, and in response to the existence, to perform instruction intent parsing on the text instructions to obtain context analysis instructions; an original analysis result generation unit, used to perform context-aware model reasoning and original feature extraction on the high-resolution image tiles based on the context analysis instructions to obtain original analysis results, the original analysis results including segmentation masks and detection results; and a feature quantization and encapsulation unit, used to perform feature quantization and structured vector encapsulation on the original analysis results to obtain the microscopic visual feature vector.

7. The Sky Eye Insight Multimodal Human-Computer Interaction Application System based on a pathologist's perspective as described in claim 6, characterized in that, The feature quantization and encapsulation unit includes: a result parameter calculation subunit, used to calculate the morphological and textural parameters of each segmentation mask in the original analysis result to obtain an object index list; a detection result filtering and analysis subunit, used to perform detection box filtering and statistical analysis on each detection result in the original analysis result to obtain a kernel splitting image count; and a feature vectorization subunit, used to perform feature vectorization on the object index list and the kernel splitting image count to obtain the microscopic visual feature vector.

8. The multimodal human-computer interaction application system based on a pathologist's perspective, as described in claim 1, is characterized in that... The multimodal data processing module includes: a visual-text fusion processing unit, used to perform multimodal feature alignment and deep fusion on microscopic visual feature vectors and patient information text to obtain a visual-text fusion feature vector; a diagnosis result generation unit, used to input the visual-text fusion feature vector into a multi-task decision head to obtain a diagnosis result, the multi-task decision head including a classification head, a grading head, and a key event detection head; and a preliminary report data generation unit, used to perform template-driven report filling on the diagnosis result to obtain the preliminary report data.

9. The multimodal human-computer interaction application system based on a pathologist's perspective, as described in claim 8, is characterized in that... The multimodal data processing module further includes: a lesion localization data designation unit, used to designate the coordinates of confirmed lesions as the lesion localization data; a key visual evidence extraction unit, used to extract key visual evidence from the original analysis results based on the lesion localization data; an extraction unit, used to extract key quantitative indicators from microscopic visual feature vectors based on the lesion localization data; and an evidence integration unit, used to integrate the key visual evidence and key quantitative indicators to obtain the interpretability basis.

Images