Multi-party collaborative pathology slide reading device and its readable storage medium

Abstract

This invention proposes a multi-party collaborative pathology slide reading device and its readable storage medium. Addressing the issues of synchronization lag and lack of traceability caused by large slides in remote consultations, this solution first constructs a three-dimensional tile database and maintains a state machine on the server side supporting switching between follow and independent modes. It then collects terminal state packets in real time to map the tile matrix index range and determine the gaze point coordinates. Subsequently, based on the mode, viewport, and gaze point, it calculates and pushes a four-level tile priority queue including the gaze point periphery, multi-terminal overlap area, motion prediction preloading, and low-resolution edge placeholder rendering. Finally, it acquires AI-suspected lesion areas and calculates scores by combining multi-terminal dwell time, experience, and other data, generating a decision confidence map and triggering a warning against missed diagnoses. This invention is primarily used for intraoperative remote collaborative pathology slide reading.

Description

Technical Field

[0001] This invention relates to the field of medical informatics and medical image data processing technology, and in particular to a multi-party collaborative pathology image reading device and its readable storage medium based on dynamic scheduling of viewport and gaze point. Background Technology

[0002] Intraoperative frozen section pathological diagnosis is a critical clinical decision in surgery, typically requiring a definitive benign or malignant diagnosis within a short time after specimen receipt, directly determining the extent of surgical resection. With the development of medical informatics, digital full-view digital slide technology is widely used in remote multidisciplinary consultations.

[0003] However, single full-view digital slide images have extremely high resolution, with data volumes typically reaching several gigabytes to tens of gigabytes. Traditional image sharing and transmission methods face significant challenges in collaborative multi-party slide reading. Existing remote slide reading technologies mainly employ screen-sharing video streaming or full-slide downloading. Screen-sharing, due to severe image compression, blurs crucial diagnostic details such as cell nuclei and tissue structures in pathological slides, and experts at the collaborating end cannot perform independent zooming, dragging, or measurement operations, remaining in a passive viewing state. Full-slide downloading, on the other hand, results in extremely long download times due to the large slide files, especially with limited network bandwidth or multiple experts online concurrently, failing to meet the high timeliness requirements of intraoperative frozen section diagnosis.

[0004] Currently, if multiple experts simultaneously perform high-frequency image dragging and zooming operations online, the concurrent streaming pressure on the server will increase exponentially, easily leading to network congestion. This can cause severe pixelation or white patches on the terminal screen, delaying surgical intervention. Furthermore, traditional collaborative remote consultations rely solely on voice or simple on-screen instructions for communication, lacking objective quantitative records of the actual image viewing behavior of experts on multiple devices. This results in a lack of traceable quality control evidence for the final diagnostic decision.

[0005] Therefore, there is an urgent need for a multi-party collaborative pathology slide reading device and its readable storage medium based on dynamic scheduling of viewport and gaze point, in order to solve the problems existing in the current technology. Summary of the Invention

[0006] This invention provides a multi-party collaborative pathology slide reading device and its readable storage medium based on dynamic scheduling of viewport and gaze point. It addresses the technical defects of existing multi-party collaborative pathology slide reading technologies, such as severe image compression due to large data volume when transmitting large slides, limited independent operation of multiple terminals, network lag and loading white blocks caused by high-frequency concurrent interaction, and lack of objective quantitative traceability basis for the entire consultation decision-making process.

[0007] The core technology of this invention is to control the seamless switching of multi-terminal viewing modes through a server-side state machine, and to calculate a four-level tile priority queue in real time based on the tile matrix index range mapped from the state packets of each terminal and the dynamically determined gaze point coordinates, and to perform active push scheduling. At the same time, during the viewing process, multi-terminal interaction data is extracted and gridded weighted cumulative calculation is performed to generate a decision confidence map and trigger an early warning.

[0008] In a first aspect, the present invention provides a multi-party collaborative pathological slide reading method based on dynamic scheduling of the viewport and fixation point, the method comprising the following steps:

[0009] A three-dimensional tile database is constructed, in which the original full-view digital slice image is first transformed into a multi-level image pyramid structure, and each level is divided into tiles of fixed size; The server maintains a state machine, which includes a follow mode and an independent mode, and controls the switching of the viewing mode according to the request sent by the terminal. The status packets reported by each terminal are collected in real time. The two-dimensional pixel boundaries and scaling ratios of the terminal screen in the status packets are mapped to the tile matrix index range of the corresponding level. The coordinates of the terminal's gaze point are determined based on the terminal's operation and interaction behavior. Based on the current viewing mode, tile matrix index range, and gaze point coordinates of the master control terminal and the collaborative terminal, calculate the priority queue of relevant tiles at the current level; Streams are pushed to the corresponding terminals of multiple parties according to priority queues to achieve multi-terminal slice collaborative reading.

[0010] Furthermore, a state machine is maintained on the server side. This state machine includes a follow mode and an independent mode, and controls the switching of the viewing mode based on the requests sent by the terminal, including: By default, it enters follow mode, where the master control takes control of the viewport. The server calculates a unified priority queue based on the master control's coordinates and broadcasts the same tile stream to all collaborating ends. When a request to view independently is received from a collaborating end, the server switches to independent mode, modifies the synchronization flag of the collaborating end, removes the control priority restriction of the collaborating end from the master control end, and establishes an independent push stream queue for the collaborating end. The priority of each tile in its independent push stream queue is determined by the collaborating end's own gaze point.

[0011] Furthermore, mapping the terminal screen boundaries in the state packet to the corresponding tile matrix index range includes: Based on the current scaling ratio in the status packet, match the closest tile level in the 3D tile database; Based on the current screen 2D pixel boundary in the state packet, and combined with the tile size, the calculation is performed by rounding down to map the range of tile 3D matrix indices that need to be loaded at the tile level.

[0012] Furthermore, determining the gaze coordinates of the terminal based on its operational interaction behavior includes: When the terminal's mouse is detected to have been continuously hovering within a set spatial radius area for more than a set time threshold, the coordinates of the mouse are determined to be a valid gaze point. When the user remains stationary for more than the set idle threshold, the gaze point weight is reduced, and the geometric center point of the current viewport is returned as the virtual gaze point.

[0013] Furthermore, calculating the priority queue of relevant tiles at the current level includes: Map the gaze point coordinates of the current master control terminal to the tile matrix, and mark the tiles within the preset neighborhood range around the gaze point coordinates as the first priority; The tiles corresponding to the visible areas with the highest overlap in the viewport matrices of the current master control terminal and each cooperating terminal are marked as second priority; based on the viewport movement trajectory of the master control terminal, motion trend prediction is performed, and the tiles in the predicted future area are marked as third priority; The viewport edge area is marked as the fourth priority. For tiles of the fourth priority, the server does not push high-definition tiles of the current level. Instead, it pulls the low-resolution tiles of the previous level from the cache 3D tile database and sends them to the client so that the client can use the low-resolution tiles to stretch and occupy space. Once the bandwidth is available, the low-resolution tiles will be replaced with high-definition tiles of the current level.

[0014] Furthermore, based on the viewport movement trajectory of the master control terminal, motion trend prediction is performed, and tiles within the predicted future area are marked as third priority, including: Record the coordinates of the center point of the most recently set frame number on the main control terminal, and calculate the movement speed vector of the current viewport; Based on the current position coordinates, the kinematic equations are used to predict the coordinate landing point of the next time period, and the tiles in a specific sector area in the direction of the coordinate landing point are marked as the third priority and added to the push queue for preloading in advance.

[0015] Furthermore, it also includes the step of generating a decision confidence map: The artificial intelligence model is used to identify suspected lesion areas from the original full-view digital slice image; Extract the operation data of the main control terminal and the collaborative terminal during the image viewing process. The operation data includes the dwell time of each terminal in a specific tile area, the seniority weight of the expert corresponding to the terminal, the weight of the current magnification, and the score of the interaction marker event. Based on the operational data, the data of each participating consultant are weighted and accumulated on the same coordinate grid to obtain the gridded confidence score; After normalizing the gridded confidence scores, they are mapped to thermal colors and overlaid on the full-view digital slice thumbnail to generate a consultation decision confidence map. When a low-scoring region in the confidence profile coincides with a suspected lesion region, an automatic warning to prevent missed diagnoses is triggered.

[0016] Secondly, the present invention provides a multi-party collaborative pathological slide reading device based on dynamic scheduling of the viewport and fixation point, comprising: The database construction module is used to build a 3D tile database, in which the original full-view digital slice image is pre-converted into a multi-level image pyramid structure, and each level is divided into tiles of fixed size; The state control module is used to maintain the state machine on the server side. The state machine includes follow mode and independent mode, and controls the switching of the viewing mode according to the request sent by the terminal. The mapping and determination module is used to collect the status packets reported by each terminal in real time, map the two-dimensional pixel boundaries and scaling ratios of the terminal screen in the status packets to the tile matrix index range of the corresponding level, and determine the gaze coordinates of the terminal based on the terminal's operation and interaction behavior. The priority calculation module is used to calculate the priority queue of relevant tiles in the current level based on the current tile reading mode, tile matrix index range, and gaze point coordinates of the master control terminal and the collaborative terminal. The streaming media push module is used to push streams to multiple terminals according to priority queues to achieve multi-terminal slice collaborative viewing.

[0017] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the above-described multi-party collaborative pathological slide reading method based on dynamic scheduling of viewport and fixation point.

[0018] Fourthly, the present invention provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the multi-party collaborative pathological slide reading method based on dynamic scheduling of viewport and fixation point as described above.

[0019] The main contributions and innovations of this invention are as follows: 1. Significantly reduces the instantaneous network bandwidth consumption of large-scale tile transmission: This invention abandons the traditional full-image transmission and passive video streaming mode. It pre-converts the original image into a multi-level image pyramid and divides it into tiles of fixed size. At any time of viewing, it accurately pushes only the core high-definition tiles corresponding to the current viewport boundary and gaze point, which greatly reduces the amount of data sent instantly. It can achieve millisecond-level multi-terminal high-definition synchronous frequency following or independent autonomous viewing in low-bandwidth network environments.

[0020] 2. Achieves flexibility and coordination of primary and secondary modes in multidisciplinary collaborative consultations: This invention maintains follow mode and independent mode through a server-side state machine and supports seamless switching between the two viewing modes. In follow mode, the master control terminal uniformly manages the viewport control, and multiple collaborative terminals follow in unison, ensuring a clear distinction between primary and secondary aspects of the consultation. In independent mode, experts on the collaborative terminals can request the removal of priority restrictions to perform independent examination operations, greatly improving the flexibility of multi-expert, multi-view collaborative image viewing.

[0021] 3. Eliminates white blocks and mosaic effects during slice drag browsing: This invention introduces a trajectory prediction mechanism based on kinematic equations when calculating the priority queue. Tiles in the fan-shaped area in front of the mouse and viewport movement point are marked as third priority and added to the queue in advance for preloading. At the same time, the viewport edge area is marked as fourth priority, and instead, the smaller macroscopic tiles from the previous level are pulled and stretched and blurred on the client side to ensure that no white blocks appear on the screen during high-frequency dragging. When the bandwidth is available, they are smoothly tiled and replaced with high-definition tiles, ensuring the continuity of the viewing experience.

[0022] 4. An objective quantitative traceability system for image reading behavior was constructed to effectively prevent misdiagnosis and missed diagnosis: This invention extracts multi-dimensional behavioral data from experts at each terminal, including their dwell time in specific areas, seniority weight, magnification weight, and explicit interactive marking bonuses. These data are then weighted and accumulated using a formula in a gridded manner, transforming the subjective consultation process into an objective confidence heatmap. When a low-scoring blind spot in the final heatmap overlaps with a suspected lesion area suggested by the AI ​​model, an automatic early warning for preventing missed diagnosis is triggered. This provides traceable data support for pathology quality control management and effectively reduces the missed diagnosis rate.

[0023] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1This is an overall architecture diagram of a multi-party collaborative pathological slide reading method based on dynamic scheduling of viewport and fixation point according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the tile priority scheduling principle of a multi-party collaborative pathological slide reading method based on dynamic scheduling of viewport and fixation point according to an embodiment of the present invention. Figure 3 This is a flowchart of the multi-terminal behavior decision confidence map generation process of the multi-party collaborative pathological slide reading method based on dynamic scheduling of viewport and fixation point according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0026] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0027] Example 1 This embodiment provides a multi-party collaborative pathology slide reading method based on dynamic scheduling of viewport and fixation point. It aims to solve technical problems such as multi-disciplinary joint remote pathology diagnosis scenarios with extremely high timeliness requirements, such as multi-party synchronization lag, network bandwidth bottlenecks, and lack of objective quantitative records of the consultation process caused by the huge amount of data in whole-slide image (WSI) slides, which are caused by the huge amount of data in whole-slide images (WSI).

[0028] This embodiment proposes a multi-party collaborative pathological image reading method based on dynamic scheduling of viewport and fixation point. Its network system architecture mainly includes an image preprocessing module, a multi-terminal state synchronization module, a streaming media dynamic scheduling module, and a behavior graph generation module deployed on the server. The terminal roles involved in this embodiment include: the node initiating the consultation is the main control terminal, corresponding to the primary image reading expert; the nodes invited to participate in the consultation are the collaborative terminals, corresponding to collaborative consultation experts. The main control terminal and the collaborative terminals establish bidirectional data communication with the streaming media scheduling server via the network.

[0029] like Figure 1 As shown, in this embodiment, the overall architecture is divided into three layers, presenting a complete link from "data source" to "server-side scheduling" and then to "multi-terminal presentation": 1. Medical data sources (far left) Content: Raw WSI (full field of view digital slide) files generated by a scanner or microscope.

[0030] Features: It is a 2D pixel-level image, characterized by extremely high resolution (such as 100,000 x 100,000 pixels) and extremely large size (about 15GB), which is also the "culprit" that causes traditional network transmission to lag.

[0031] 2. Streaming media scheduling server (central core) It includes four core engines that work together: 2D-3D tile engine: responsible for "slicing and layering" the 15GB giant tile to build a pyramid-like (Z level, X, Y) three-dimensional index database.

[0032] Terminal state machine: Responsible for managing the viewing mode. It determines whether the current mode is "master-follower mode" (everyone views the same screen) or "independent mode" (an expert zooms in to view other areas).

[0033] Dynamic scheduling engine: This is the core of the streaming. Instead of blindly sending out the entire image, it calculates in real time which tiles should be sent first and which should be sent later based on where everyone is looking (viewport overlap) and where the mouse is moving (prediction vector) (priority calculation).

[0034] Behavioral Graph Engine: Responsible for acting as a "recorder" and "quality controller", quietly recording the subtle clues of multi-terminal expert interactions and calculating the consultation confidence on a gridded coordinate system.

[0035] 3. Multi-party consultation terminal group (far right) The interface showing the specific experts involved in the consultation is displayed: The master control unit (master viewer): responsible for setting the pace; its viewport coordinates and mouse gaze point are the benchmarks for the server to calculate the highest priority.

[0036] Collaboration Expert A: Responsible for coordination, receiving priority tile streams from the server, and synchronizing the screen with the main control terminal.

[0037] Collaboration Expert B: Responsible for the final quality control rendering, it can not only view slices, but also render and overlay the final thermal warning map.

[0038] Specifically, the multi-party collaborative pathological slide reading method based on dynamic scheduling of viewport and fixation point includes the following core steps: Step 1: Construct a 3D tile database Before the consultation begins, the system first preprocesses the raw WSI images using the server-side image preprocessing module. Raw WSI images are typically generated using a dedicated pathology scanner, with a single image resolution of up to 100,000 × 100,000 pixels and a file size of approximately 15GB. The system employs an image pyramid algorithm to downsample the raw WSI images at different magnification levels, generating multiple layers. In this embodiment, five layers are set: 2x, 4x, 10x, 20x, and 40x, where 40x represents the original scan resolution.

[0039] Each layer of the image is further segmented into fixed-size tiles; in this embodiment, the tile size is 256×256 pixels. Each tile is stored in JPEG format, with a compressed size of approximately 50KB. The system uses a triple (Z, X, Y) as the unique identifier for each tile, where Z represents the layer index, X represents the horizontal tile index of that layer, and Y represents the vertical tile index of that layer. The system uses this triple as the primary key to construct a three-dimensional tile database and stores it in a server-side cache or distributed storage system so that the subsequent scheduling module can quickly locate and retrieve tile data based on the index.

[0040] Step 2: Maintain the state machine on the server and control the switching of the viewing mode. See Figure 1 and Figure 2 The server-side multi-terminal state synchronization module maintains a viewing mode state machine, which includes two viewing modes: SyncMode and IndependentMode, and controls the switching of viewing modes according to the requests sent by the terminals.

[0041] 1. Follow Mode: Follow mode is the system default mode. In this mode, the master control terminal takes control of the viewport, and the server uses the coordinate data reported by the master control terminal as a reference to calculate a unified tile priority queue and broadcast the same tile stream to all collaborating terminals, realizing "master control terminal leads the screen, and multiple collaborating terminals follow in unison". This mode is suitable for scenarios in consultation where the lead expert guides the reading rhythm and the collaborating terminals observe synchronously.

[0042] 2. Independent Mode: When a collaborating end needs to view a specific area independently, it sends an independent viewing request to the server. Upon receiving this request, the server switches to independent mode, sets the collaborating end's synchronization flag (Sync_Flag) to false, removes the priority restriction imposed on the master control end, and establishes an independent streaming queue for the collaborating end. In independent mode, the priority of each tile in the collaborating end's independent streaming queue is determined by the collaborating end's own gaze point determination result, no longer affected by the master control end's viewport coordinates. When the collaborating end finishes independent viewing and re-requests to follow the master control end, the server receives its synchronization request and restores Sync_Flag to true, and the collaborating end re-enters follow mode.

[0043] Step 3: Terminal Status Packet Acquisition and 2D-to-3D Mapping Determination Each terminal reports status packets to the server at a fixed frequency; in this embodiment, the reporting frequency is 30 frames per second. Each status packet contains the following data fields: Terminal ID, current screen 2D pixel boundaries (Pixel_X_min, Pixel_Y_min, Pixel_X_max, Pixel_Y_max), current view zoom level (Scale), and current mouse coordinates (Mouse_X, Mouse_Y).

[0044] The server collects status packets reported by each terminal in real time and maps the 2D pixel boundaries and scaling ratios of the terminal screen in the status packets to the corresponding tile matrix index range. The specific mapping logic is as follows: First, based on the current scaling ratio Scale in the status packet, match the closest tile level Z in the 3D tile database. The specific matching rules are as follows: The scale is compared with the magnification ratio corresponding to each level, and the level closest to and not exceeding the current scaling ratio is selected as the current Z level. After determining the Z level, the server divides the current screen 2D pixel boundary in the status packet by the tile size (256 pixels) and rounds down to map the range of tile 3D matrix indices that the current screen viewport needs to load under this Z level. The mapping formula is as follows:

[0045]

[0046]

[0047]

[0048] Therefore, the tile matrix index range corresponding to the terminal visible area under the current Z level is (Tile_X_start to Tile_X_end, Tile_Y_start to Tile_Y_end).

[0049] Taking a typical 4K display (resolution 3840×2160 pixels) as an example, at any scaling level, the screen only needs to render a maximum of about 15×9=135 tiles. Assuming a single JPEG tile is 50KB, the current viewport's high-definition data size is only 6.7MB, which is far smaller than the original image's 15GB size.

[0050] While mapping the viewport area, the server dynamically determines the gaze coordinates of the terminal based on the terminal's user interaction behavior. This embodiment employs gaze determination logic based on mouse interaction behavior without relying on dedicated eye-tracking hardware. 1. Valid Hover Determination: When the system detects that the terminal's mouse has been continuously hovering within a set spatial radius area for more than a set time threshold, the coordinates of the mouse are determined to be a valid gaze point. In this embodiment, the spatial radius is set to 10 pixels, and the time threshold (Thover) is set to 500 milliseconds. That is, when the mouse has been continuously hovering within a circular area with a radius of 10 pixels centered at the current position for more than 500 milliseconds, the system determines and records this point as the current gaze point coordinates (Focus_X, Focus_Y).

[0051] 2. Dynamic Degradation and Centering Backup: When a user only performs static browsing without mouse operation, and the mouse remains stationary for more than a set idling threshold, the system automatically attenuates the gaze point weight. In this embodiment, the idling threshold (Tidle) is set to 3 seconds. That is, when the mouse remains stationary for more than 3 seconds, the system determines that the current user is in a browsing state without a clear focus target. At this time, the geometric center point of the current viewport is returned as the virtual gaze point, calculated as follows:

[0052]

[0053] This fallback mechanism ensures that even if the user does not actively interact with the system, the system can still continuously output valid gaze coordinates for subsequent tile priority calculations, thus avoiding interruptions in the streaming schedule.

[0054] Step 4: Calculate the priority queue of relevant tiles at the current level. See Figure 2 The streaming media dynamic scheduling module, based on collected multi-terminal status data, current viewing mode, tile matrix index range, and gaze point coordinates, does not directly send out complete images. Instead, it calculates the priority P of all relevant tiles at the current Z level and pushes tile data to each terminal according to the priority queue from high to low. This embodiment defines a four-level priority queue: 1. First Priority (P1, Extremely High Priority): After mapping the gaze coordinates of the current master terminal to the tile matrix, tiles within a preset neighborhood of the gaze coordinates are marked as first priority. In this embodiment, the neighborhood is set as a 3×3 matrix, i.e., 9 tiles centered on the tile where the gaze is located. In follow mode, P1 priority applies to all collaborating terminals; in independent mode, the P1 priority of a collaborating terminal is determined by its own gaze coordinates. P1 tiles will be pushed to the corresponding terminal with the highest resolution and fastest speed, ensuring that the core areas of focus for experts are presented in high-definition detail in real time.

[0055] 2. Second Priority (P2, High Priority): Tiles corresponding to the visible areas with the highest overlap in the viewport matrices of the current master control terminal and each collaborating terminal are marked as second priority. The system compares the intersection of the viewport range of the master control terminal and the viewport ranges of each collaborating terminal, and marks the tiles of common interest to multiple terminals within the intersection area as second priority, to ensure that all participants obtain a synchronized high-definition experience in the same frequency area.

[0056] 3. Third Priority (P3, Medium Priority): This level is for trajectory prediction streaming. The system predicts motion trends based on the viewport movement trajectory of the master control terminal, marks tiles within the predicted future area as third priority, and adds them to the streaming queue for pre-loading. The specific prediction algorithm is as follows: The server records the center point coordinates of the master control terminal for the most recent set number of frames (set to 10 frames in this embodiment). Let the center point coordinates of the i-th frame be (C_Xi, C_Yi), and the time interval between two adjacent frames be Δt, then the current viewport movement speed vector is:

[0057]

[0058] Based on the current position coordinates, the coordinates of the next time period T_pred (T_pred = 1 second in this embodiment) are predicted using kinematic equations:

[0059]

[0060] After mapping the coordinate landing point (P_X, P_Y) to the tile matrix, the system marks the tiles within a specific sector in the direction of the landing point's movement as having the third priority. In this embodiment, the angle of the specific sector is set to 120 degrees, and the radius is set to 5 tile widths. When the main control terminal's movement direction changes abruptly, the tiles preloaded in P3 may not be used immediately because the predicted landing point deviates from the actual position. However, since their data volume is small, the system will recalculate and adjust the P3 region after the next round of status packet updates, achieving dynamic correction.

[0061] 4. Fourth Priority (P4, Low Priority): Tiles in the viewport edge area are marked as fourth priority. For fourth priority tiles, the server does not push high-resolution tiles of the current level at the current viewing time. Instead, it pulls the corresponding low-resolution tiles of the previous level (Z-1 or Z-2 level) from the cache of the 3D tile database and sends them to the client. The client then uses the Canvas rendering engine to stretch (upscale) the low-resolution tiles to fill the corresponding area as placeholders. Due to the inherent blurring effect of the stretching operation, it can serve as a reasonable placeholder visually, ensuring that no abrupt white blank areas appear when the user quickly drags or zooms the screen. When the system network bandwidth is idle (for example, when the server detects that the current network bandwidth utilization is lower than a set threshold), the server smoothly pushes high-resolution tiles of the current level to replace them.

[0062] The aforementioned P1 to P4 four-level priority scheduling strategy allows the system to prioritize the transmission of only a small number of high-definition tiles (usually less than 1MB) in the core area of ​​the gaze point at any given time. Combined with the low-resolution tiles occupying the edge areas, this achieves a smooth viewing experience under extremely low bandwidth conditions. The amount of data transmitted instantly is less than one millionth of the original image size, and the overall bandwidth usage is reduced by more than 90% compared to traditional streaming media.

[0063] Step 5: Push the stream according to the priority queue and generate the decision confidence map simultaneously. The streaming media dynamic scheduling module pushes streams to the terminals corresponding to the multi-party collaboration according to the priority queue, thereby realizing multi-terminal slice collaborative viewing. (See also...) Figure 3 The method in this embodiment also includes a decision confidence graph generation and a missed diagnosis warning mechanism that are executed synchronously by the behavior graph generation module: 1. Acquiring Suspected Lesion Areas Using Artificial Intelligence: The system first acquires suspected lesion area data generated by the artificial intelligence model after identifying and predicting lesions in the original WSI image through a data interface. This data serves as a reference benchmark for subsequent early warning judgments based on the confidence map.

[0064] 2. Extraction of Operation Behavior Data: During the image viewing process, the server continuously extracts and records the operation data of the main control terminal and the collaborating terminal, specifically including the following parameters: Dwell Time T: The effective viewing time of each terminal within a specific tile area, in milliseconds. The system only counts the effective viewing time when the terminal's viewport covers the tile area and the user is active (mouse is moving or has been confirmed to hover); momentary passing by is not counted.

[0065] The qualification weight We for the expert corresponding to the terminal is a static preset value. In this embodiment, W is the qualification weight of the chief physician or the main control terminal. e Set to 1.5, W of the associate chief physician e Set to 1.2, the attending physician's W e Set to 1.0.

[0066] Magnification weight W z Nonlinear dynamic mapping is employed to reflect the different diagnostic values ​​of different magnifications in pathological diagnosis. In this embodiment, W z The mapping rules are as follows: 2x to 4x (for finding lesions), W z =1; 10x to 20x (for observing tissue structure), W z =2; 40x (used to observe key diagnostic features such as cell mitotic figures), W z =5.

[0067] Event Bonus M: If explicit interactive behaviors such as measurement, circling, or text annotation occur in the tile area, the system directly awards a high bonus score. In this embodiment, the value of M is set to 50.

[0068] 3. Confidence Score Calculation: Based on the operational data, the system performs a weighted summation of the data from each participant in the consultation within the same coordinate grid (i.e., the same tile area) to obtain the confidence score for that grid. The calculation formula is as follows:

[0069] Where n is the number of terminals participating in the consultation. Let be the dwell time of the i-th terminal within this grid area. The weight of the current magnification factor for the i-th terminal. Let be the qualification weight of the expert corresponding to the i-th terminal. Add points to the marking event of the i-th terminal in this area.

[0070] 4. Atlas Generation and Early Warning Against Missed Diagnoses: The system normalizes the confidence scores of each grid and maps them to thermal colors (e.g., high-scoring areas are mapped to dark warm colors to represent high magnification and deep consensus, while low-scoring areas are mapped to cool colors to represent blind spots). This is then overlaid on the WSI thumbnail base map to generate a visualized consultation decision confidence atlas. When a low-scoring area in the confidence atlas spatially overlaps with the previously acquired suspected lesion area, the system determines that there is a risk of missed diagnosis and automatically triggers an early warning message. A quality control warning is issued on the interface of the multi-party collaborative terminal, reminding the consultation experts to supplement the image review for that blind spot.

[0071] Example 2 Based on the same inventive concept as in Embodiment 1 above, this embodiment provides a multi-party collaborative pathological slide reading device based on dynamic scheduling of viewport and fixation point. The device is deployed on a streaming media scheduling server and specifically includes the following functional modules: 1. Database Construction Module: Used to construct a 3D tile database. The original full-view digital slice image is pre-converted into a multi-level image pyramid structure, and each level is divided into tiles of fixed size, which are stored in a database with the level and coordinate index as the primary key.

[0072] 2. State control module: used to maintain the state machine on the server side. The state machine includes a follow mode and an independent mode, and controls the switching of the viewing mode according to the request sent by the terminal.

[0073] 3. Mapping Determination Module: This module collects status packets reported by each terminal in real time, maps the two-dimensional pixel boundaries and scaling ratios of the terminal screen in the status packets to the tile matrix index range of the corresponding level, and determines the gaze coordinates of the terminal based on the terminal's operation and interaction behavior.

[0074] 4. Priority Calculation Module: This module is used to calculate the priority queue of relevant tiles at the current level based on the current viewing mode, tile matrix index range, and gaze point coordinates of the master control terminal and the collaborating terminal.

[0075] 5. Streaming media push module: used to push the stream to the terminal corresponding to the multi-party collaboration according to the priority queue, so as to realize multi-terminal slice collaborative viewing.

[0076] In addition, the device also includes a behavior map generation module, which is used to obtain suspected lesion areas generated by the artificial intelligence model through lesion identification of the original full-view digital slice image; extract multi-terminal operation data and perform gridded weighted accumulation calculation to obtain gridded confidence scores; normalize the scores and map them into thermal color overlay to generate a consultation decision confidence map; and automatically trigger a warning to prevent missed diagnoses when the low-score area in the map overlaps with the suspected lesion area.

[0077] Example 3 This embodiment also provides an electronic device, see reference. Figure 4 It includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0078] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0079] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0080] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0081] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the multi-party collaborative pathological slide reading methods based on dynamic scheduling of viewport and fixation point in the above embodiments.

[0082] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0083] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0084] Input / output device 408 is used to input or output information.

[0085] Example 4 This embodiment also provides a readable storage medium storing a computer program, which includes program code for controlling and executing a process, the process including the multi-party collaborative pathological slide reading method based on dynamic scheduling of viewport and fixation point according to Embodiment 1.

[0086] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0087] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0088] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 3 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0089] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A multi-party collaborative pathological slide reading method, characterized in that, Includes the following steps: A three-dimensional tile database is constructed, in which the original full-view digital slice image is first transformed into a multi-level image pyramid structure, and each level is divided into tiles of fixed size; The server maintains a state machine, which includes a follow mode and an independent mode, and controls the switching of the viewing mode according to the request sent by the terminal. The status packets reported by each terminal are collected in real time. The two-dimensional pixel boundaries and scaling ratios of the terminal screen in the status packets are mapped to the three-dimensional matrix index range of the tiles at the corresponding level. The gaze coordinates of the terminal are determined based on the terminal's operation and interaction behavior. Based on the current viewing mode of the master terminal and the collaborative terminal, the range of the tile 3D matrix index, and the coordinates of the gaze point, calculate the priority queue of relevant tiles in the current level; Pushing streams to the terminals corresponding to the multi-party collaboration according to the priority queue to achieve multi-terminal slice collaborative viewing; The mapping of the terminal screen's two-dimensional pixel boundaries in the status packet to the corresponding level's tile three-dimensional matrix index range includes: Based on the current scaling ratio in the status packet, match the closest tile level in the 3D tile database; Based on the current screen two-dimensional pixel boundary in the state packet, and combined with the size of the tile, the floor function is calculated to map the range of the three-dimensional matrix index of the tile that needs to be loaded under the tile level. Determining the gaze coordinates of the terminal based on its user interaction behavior includes: When the terminal's mouse is detected to have been continuously hovering within a set spatial radius area for more than a set time threshold, the coordinates of the mouse are determined to be a valid gaze point. When the user remains stationary for more than the set idle threshold, the gaze point weight is reduced, and the geometric center point of the current viewport is returned as the virtual gaze point. Calculating the priority queue of relevant tiles at the current level includes: Map the gaze point coordinates of the current master control terminal to the tile matrix, and mark the tiles within the preset neighborhood range around the gaze point coordinates as the first priority; The tiles corresponding to the visible areas with the highest overlap in the viewport matrices of the current master control terminal and each cooperating terminal are marked as second priority; based on the viewport movement trajectory of the master control terminal, motion trend prediction is performed, and the tiles in the predicted future area are marked as third priority; The viewport edge area is marked as the fourth priority. For tiles of the fourth priority, the server does not push high-definition tiles of the current level. Instead, it pulls the low-resolution tiles of the previous level from the cache of the 3D tile database and sends them to the client so that the client can use the low-resolution tiles to stretch and occupy space. Once the bandwidth is available, the low-resolution tiles will be replaced with high-definition tiles of the current level.

2. The multi-party collaborative pathological slide reading method as described in claim 1, characterized in that, The server maintains a state machine, which includes a follow mode and an independent mode, and controls the switching of the viewing mode according to the request sent by the terminal, including: By default, it enters follow mode, where the master control takes control of the viewport. The server calculates a unified priority queue based on the master control's coordinates and broadcasts the same tile stream to all collaborating ends. When a request to view independently is received from a collaborating end, the server switches to independent mode, modifies the synchronization flag of the collaborating end, removes the control priority restriction of the collaborating end from the master control end, and establishes an independent push stream queue for the collaborating end. The priority of each tile in its independent push stream queue is determined by the collaborating end's own gaze point.

3. The multi-party collaborative pathological slide reading method as described in claim 1, characterized in that, Based on the viewport movement trajectory of the master control terminal, motion trend prediction is performed, and tiles within the predicted future area are marked as third priority, including: Record the coordinates of the center point of the most recently set frame number on the main control terminal, and calculate the movement speed vector of the current viewport; Based on the current position coordinates, the kinematic equations are used to predict the coordinate landing point of the next time period, and the tiles in a specific sector area in the direction of the coordinate landing point are marked as the third priority and added to the push queue for preloading in advance.

4. The multi-party collaborative pathological slide reading method as described in claim 1, characterized in that, It also includes the step of generating the decision confidence map: The suspected lesion region is obtained by the artificial intelligence model through lesion identification of the original full-view digital slice image; Extract the operation data of the main control terminal and the collaborative terminal during the image viewing process. The operation data includes the dwell time of each terminal in a specific tile area, the seniority weight of the expert corresponding to the terminal, the weight of the current magnification, and the score of the interaction marker event. Based on the operational data, the data of each participating consultant are weighted and accumulated on the same coordinate grid to obtain the gridded confidence score; The gridded confidence scores are normalized and mapped to thermal colors, which are then superimposed on the full-view digital slice thumbnail to generate a consultation decision confidence map. When a low-scoring region in the confidence profile coincides with a suspected lesion region, an early warning for preventing missed diagnoses is automatically triggered.

5. An apparatus for implementing the multi-party collaborative pathological slide reading method according to any one of claims 1 to 4, characterized in that, include: The database construction module is used to build a 3D tile database, in which the original full-view digital slice image is pre-converted into a multi-level image pyramid structure, and each level is divided into tiles of fixed size; The state control module is used to maintain the state machine on the server side. The state machine includes a follow mode and an independent mode, and controls the switching of the viewing mode according to the request sent by the terminal. The mapping determination module is used to collect the status packets reported by each terminal in real time, map the two-dimensional pixel boundaries and scaling ratios of the terminal screen in the status packets to the three-dimensional matrix index range of the tiles at the corresponding level, and determine the gaze point coordinates of the terminal based on the terminal's operation and interaction behavior. The priority calculation module is used to calculate the priority queue of relevant tiles in the current level based on the current viewing mode of the master terminal and the collaborative terminal, the three-dimensional matrix index range of the tile, and the coordinates of the gaze point. The streaming media push module is used to push the stream to the terminal corresponding to the multi-party collaboration according to the priority queue, so as to realize multi-terminal slice collaborative viewing.

6. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the multi-party collaborative pathological slide reading method according to any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program including program code for controlling a process to execute the process, the process including the multi-party collaborative pathological slide reading method according to any one of claims 1 to 4.

Images