Pathological image multi-terminal synchronous browsing and collaborative diagnosis method and device and medium

Abstract

The application provides a pathological image multi-terminal synchronous browsing and collaborative diagnosis method and device and medium, aiming at the sensitive data leakage risk, video stream transmission detail distortion and multi-person collaborative labeling conflict problem in pathological image remote diagnosis, a pyramid tile mapping is constructed by a server in a medical private network; a master control end listens to a viewport operation and extracts a differential increment of a multi-dimensional state vector, and broadcasts the incremental data not containing the video stream through a private WebRTC data channel; a collaborative end searches for visible tiles and performs lossless rendering locally by using graphic hardware acceleration; and a bidirectional hash deduplication and medical permission weighting operation conversion algorithm is combined to solve the concurrent labeling conflict. The application is mainly used for low-delay lossless linkage browsing and remote collaborative diagnosis of super-large field of view digital pathological sections.

Description

Technical Field

[0001] This invention relates to the fields of digital pathology, computer collaborative work, and medical information technology, and in particular to a method, device, and medium for simultaneous browsing and collaborative consultation of pathological images across multiple terminals. Background Technology

[0002] Clinical pathology diagnosis, as the authoritative basis for disease confirmation, places extremely high demands on the interpretation of subcellular microstructures in tissue sections. Whole-slide images (WSI) typically have billions of pixels per slide, with data volumes ranging from gigabytes to tens of gigabytes. In practical applications of remote pathology consultations, the following two existing technological solutions are primarily relied upon: One approach is a screen-sharing solution based on a general video conferencing platform connected to the public internet. This solution involves real-time screen capture of the pathology image browsing screen at the main control terminal, followed by video encoding and compression. The lossy video stream is then uploaded to a public cloud server and forwarded to each collaborating terminal for decoding and playback. Each collaborating terminal can only passively view the one-way broadcast video feed and cannot independently explore or roam. This mode requires access to the external public internet, resulting in sensitive data containing patient health information flowing through an external cloud, posing a risk of information leakage. Furthermore, the lossy compression algorithm used in video encoding causes severe distortion of high-frequency details essential for pathological diagnosis, such as cell nuclear chromatin texture and cytoplasmic boundaries. Additionally, the continuous transmission of high-resolution video streams consumes significant dedicated bandwidth and results in high network latency.

[0003] Secondly, there are dedicated pathology systems based on traditional DICOM / PACS or WebSocket polling / full data transmission. In multi-device collaborative browsing, these systems typically distribute slide data or poll for synchronized coordinates when viewport panning or zooming occurs. In hospital environments with cross-network segments and high packet loss rates, data throughput overhead is enormous, with panning / zooming delays reaching hundreds of milliseconds, easily leading to network congestion and screen stuttering. Furthermore, when multiple devices are simultaneously conducting online consultations, these systems often employ exclusive locking mechanisms to restrict operation to a single user at any given time, or lack collaborative conflict handling mechanisms tailored to clinical hierarchies. This results in overlapping elements, data corruption, and inconsistent visuals when multiple devices are drawing, measuring, or erasing / modifying the same pathological area. Summary of the Invention

[0004] This invention provides a method, device, and medium for multi-terminal synchronous browsing and collaborative consultation of pathological images. It addresses the problems of existing remote pathological collaboration technologies that rely on external public networks to transmit video streams or full image data, which pose risks such as leakage of patient privacy, loss of image microstructure, large network overhead and synchronization delay, and lack of conflict prevention and consistency guarantee mechanisms that match the authoritative clinical medical judgments for multi-terminal concurrent annotation.

[0005] The core technology of this invention is to preprocess the full-view digital pathological image into multi-resolution pyramid tiles and establish a coordinate mapping relationship. The main control terminal abstracts the viewport operation into a multi-dimensional state vector and broadcasts it to the collaborative terminal in incremental form through a WebRTC data channel deployed in a private medical network. The collaborative terminal retrieves and renders the corresponding tile locally based on the vector. Under the premise of completely excluding video stream transmission, pixel-level lossless synchronous linkage of multi-terminal screens is achieved. At the same time, an operation conversion algorithm that integrates medical permission weights is introduced to solve the multi-terminal concurrent annotation conflict.

[0006] In a first aspect, the present invention provides a method for multi-terminal synchronous browsing and collaborative consultation of pathological images based on private WebRTC and multi-dimensional state incremental synchronization, applied to a collaborative consultation system deployed within a private medical network. The method includes the following steps: On the server side, the full-view digital pathology images are preprocessed into multi-resolution image pyramid tiles, and a mapping relationship between the screen coordinate system and the tile metadata is established. The main control unit monitors the viewport operations on the canvas in real time, abstracts the current viewport state into a multi-dimensional state vector that includes the normalized floating-point coordinates of the viewport center point in the global physical coordinate system of the digital pathology image, the absolute magnification of the current slice, the canvas rotation angle, and the local timestamp, and calculates the state difference increment of the multi-dimensional state vector at a preset sampling frequency. The master control unit broadcasts state differential increments to the collaborating units within the medical private network via a WebRTC data channel deployed privately within the medical private network. These state differential increments do not include video stream data. After receiving the state differential increment, the collaborating end retrieves the set of tile coordinates corresponding to the currently visible area based on the state differential increment, obtains the corresponding lossless tiles from the local cache pool or asynchronously requests them from the server, and performs lossless rendering locally on the collaborating end using graphics hardware acceleration to achieve pixel-level lossless synchronous linkage browsing and collaborative consultation of multiple terminal screens.

[0007] Furthermore, the preset sampling frequency is 30Hz; the state difference increment of the multidimensional state vector is generated by the master control terminal by calculating the difference values ​​of the parameters of each dimension in the multidimensional state vector at adjacent sampling times.

[0008] Furthermore, the collaborating end retrieves the set of tile coordinates corresponding to the currently visible area and obtains the lossless tiles, including: The calculated coordinates of the target viewport center point and the absolute magnification are mapped to the corresponding image pyramid level, and the set of visible tiles corresponding to the current viewport region is determined based on the mapping relationship. The system checks if the corresponding tile exists in the local cache pool. If not, it asynchronously requests the uncached tile from the server using the HTTP / 3 transport protocol of the User Datagram Protocol and performs pixel-level lossless rendering locally using WebGL hardware acceleration.

[0009] Furthermore, before performing lossless rendering, the collaborative approach also includes: Based on the received two adjacent sets of time-stamped viewport state vectors and their corresponding velocity vectors, the collaborative end uses the bicubic Hermite spline interpolation algorithm to calculate continuous viewport transition motion trajectories locally, and drives the rendering process with the viewport transition motion trajectories to complete the viewport switching transition in a smooth manner.

[0010] Furthermore, the method also includes a bidirectional hashing deduplication step: When a viewport change operation or annotation operation occurs at the data sending end, the operator identifier, operation timestamp, region of interest coordinates and operation type are merged and a data hash fingerprint is generated by hash algorithm. The data hash fingerprint is then retrieved from the local cache pool at the sending end. If the same data hash fingerprint is found, it is determined to be a duplicate operation and the current network transmission is terminated. If no match is found, the data hash fingerprint and the corresponding data are written to the local cache pool at the sending end and network transmission is triggered. After receiving data, the data receiving end recalculates the data hash fingerprint and compares it with the local cache pool of the receiving end. If a match is found, it is determined to be duplicate data in multi-path propagation and is discarded directly. If a match is not found, it is written to the local cache pool of the receiving end before subsequent merging and rendering are performed.

[0011] Furthermore, the method also includes pre-caching and offline synchronization steps: The pre-caching thread running in the background of the collaboration end calculates the viewport movement speed vector of the master control end in real time. Based on the viewport movement speed vector and the look-ahead time adjustment coefficient, it predicts the future viewport landing area and requests the pyramid tile corresponding to the viewport landing area from the server in advance and stores it in the local cache pool. When the terminal detects a network connection interruption, it temporarily stores the locally generated annotation operations and browsing path data in the local cache pool in the form of a queue. When the network connection is restored, it performs differential compression on the temporarily stored offline data and synchronizes it to the intranet database and other collaborative terminals in the order of timestamps to perform merging.

[0012] Furthermore, the method also includes steps for resolving conflicts in multi-person collaborative annotation: Pre-configure medical permission weights based on clinical professional title levels for each consultation account logging into the system; When multiple terminals are detected to be performing concurrent annotation operations on the overlapping area of ​​the canvas within a preset concurrent time window, the comprehensive priority score of each concurrent operation is calculated based on the medical permission weight of each terminal account and the time attribute of each operation. The weight of medical permission in the comprehensive priority score is higher than that of time attribute. The operation with the highest overall priority score is retained by the operation transformation algorithm. The operations with lower overall priority scores covered by the retained operations are transformed and presented locally on the collaboration side by reducing the display opacity and performing spatial exclusive adaptive offset. At the same time, the operations with lower overall priority scores are pushed into the historical version stack. When concurrent operations have the same medical permission weight, the deterministic execution sequence and primitive transformation of each operation are determined based on the hash value of the unique operation identifier of each operation.

[0013] Secondly, this invention provides a pathological image multi-terminal synchronous browsing and collaborative consultation device based on private WebRTC and multi-dimensional state incremental synchronization, deployed within a private medical network, comprising: The pyramid mapping module is used to preprocess full-view digital pathology images into multi-resolution image pyramid tiles on the server side and establish a mapping relationship between the screen coordinate system and the tile metadata. The viewport status monitoring module is used by the main control terminal to monitor the viewport operation on the canvas in real time. It abstracts the current viewport status into a multi-dimensional state vector that includes the normalized floating-point coordinates of the viewport center point in the global physical coordinate system of the digital pathology image, the absolute magnification of the current slice, the canvas rotation angle, and the local timestamp. It also calculates the state difference increment of the multi-dimensional state vector at a preset sampling frequency. The status increment broadcast module is used to broadcast status differential increments to collaborating terminals within the medical private network via a WebRTC data channel deployed privately within the medical private network. The status differential increments do not include video stream data. The local lossless rendering module is used to retrieve the set of tile coordinates corresponding to the currently visible area based on the state difference increment after the collaborative end receives the state difference increment. It then obtains the corresponding lossless tiles from the local cache pool or asynchronously requests them from the server. Finally, it performs lossless rendering locally on the collaborative end using graphics hardware acceleration to achieve pixel-level lossless synchronous browsing and collaborative consultation of multiple terminal screens.

[0014] Thirdly, the present invention provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned method for multi-terminal synchronous browsing and collaborative consultation of pathological images based on proprietary WebRTC and multi-dimensional state incremental synchronization.

[0015] Fourthly, the present invention provides a readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for multi-terminal synchronous browsing and collaborative consultation of pathological images based on private WebRTC and multi-dimensional state incremental synchronization.

[0016] The main contributions and innovations of this invention are as follows: 1. The end-to-end system of this invention is deployed entirely privately on the hospital intranet, local area network or medical dedicated network, and is physically isolated from the external public network. It does not rely on third-party public cloud servers to transfer data, cuts off the path of leakage of patient privacy information, and meets the requirements for medical data compliance.

[0017] 2. This invention abandons the lossy video stream screen sharing mode. The main control terminal only broadcasts viewport status metadata. Each cooperating terminal loads local tiles on demand based on the pre-built pyramid tile mapping relationship, and performs high-fidelity lossless reconstruction locally using graphics hardware acceleration. This eliminates the blurring of cell microstructure and edge color overflow caused by video encoding, ensuring accurate presentation of high-frequency details of lesions.

[0018] 3. In this invention, since the state vector data packet only contains hundreds of bytes of metadata such as position, magnification, and angle, the network throughput requirement is reduced by more than two orders of magnitude compared to traditional video streaming transmission. Combined with bidirectional hash deduplication transmission control with pre-verification at the sending end and post-verification at the receiving end, redundant rendering and repeated network packet transmission are intercepted, reducing the synchronous browsing latency in the same network environment to 60-180ms. Furthermore, the interpolation smoothing mechanism eliminates the sense of screen shift caused by large-span viewport switching.

[0019] 4. The tile retrieval of this invention adopts a transmission protocol with headless blocking and forward error correction characteristics, combined with a pre-caching mechanism based on viewport velocity vector prediction, which effectively eliminates the white screen waiting phenomenon during fast roaming; when the network is unexpectedly interrupted, offline annotation data is temporarily stored in a local structured cache queue, and after the network is restored, it is automatically and losslessly merged in time sequence after differential compression, ensuring the continuity of consultation in weak network environment.

[0020] 5. This invention assigns the dominant weight of expert clinical professional title permissions to the operation conversion model, so that when concurrent overlapping annotations cause coverage conflicts, operations with higher medical permission weights are authoritatively retained, while operations with lower medical permission weights are adaptively downgraded in display, spatially offset, and added to the version stack. This not only avoids canvas chaos caused by multi-terminal concurrent conflicts and ensures the final absolute consistency of the distributed canvas, but also completely retains the annotation derivation history of users with different permission levels for audit backtracking.

[0021] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is an architecture diagram of a remote digital pathology collaborative consultation system according to an embodiment of the present invention; Figure 2 This is a timing diagram of the status synchronization of a remote digital pathology collaborative slide reading system according to an embodiment of the present invention; Figure 3 This is a timing diagram of a bidirectional hash deduplication transmission decision-maker according to an embodiment of the present invention. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0024] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0025] Example 1 like Figure 1 As shown, the system of the pathological image multi-terminal synchronous browsing and collaborative consultation device provided in this embodiment of the invention is deployed throughout the entire chain in a private medical network environment such as a hospital intranet, local area network, or medical security network. It implements physical isolation from the external public network at the physical level, preventing sensitive patient health information from flowing through external public cloud servers. The system is logically divided into four core layers: Front-end Interaction and Local Rendering Layer: The multi-terminal user interface is built on the Vue or React front-end framework. This layer integrates a high-performance digital pathology pyramid image rendering engine, utilizing WebGL graphics hardware acceleration to load and render full-view digital pathology images (WSI) pyramid tiles on demand and without loss of quality locally on each consultation terminal; the canvas interaction layer is built on SVG or Canvas, supporting professional pathology annotation tools such as pen annotation, erasing, panning, zooming, and measuring rulers; a local cache pool is established on the browser side (using the IndexedDB structured database) to cache historical annotation operation records (including coordinates, type, timestamp, etc.) and the hash fingerprints of corresponding image slices; at the same time, a Service Worker thread is introduced to silently pre-cache pathology image slices of adjacent layers and adjacent coordinates in the background. This layer may also include, for example, Figure 1 The lightweight front-end AI model shown is an optional auxiliary module that can be integrated into the system front-end.

[0026] State Incremental Synchronization and Network Communication Layer: A privately deployed Web Real-Time Communication (WebRTC) architecture is adopted. A WebRTC signaling server, media forwarding server, and LAN STUN / TURN server are built within the medical private network LAN. Lightweight viewport control metadata and labeled incremental data are transmitted between consultation terminals via the privately deployed WebRTC data channel (RTCDataChannel, based on the SCTP protocol). In the transmission of slice tile image data and the establishment of consultation signaling, an HTTP / 3 (QUIC) transport protocol based on User Datagram Protocol (UDP) is used. Leveraging its connection migration, headless blocking, and forward error correction features, the continuity of slice data transmission is ensured even in weak network environments with a packet loss rate greater than 20%.

[0027] The collaborative conflict resolution and control engine layer includes a real-time viewport change listener, a viewport smooth transition interpolator, a bidirectional hash deduplication transmission decision-maker, and a medical authority factor-weighted operational transformation (OT) algorithm engine. The real-time viewport change listener captures the master's translation and scaling operations on the canvas and converts them into high-precision state vectors; the viewport smooth transition interpolator performs smooth viewport transitions locally on the collaborative end; the bidirectional hash deduplication transmission decision-maker performs sender-side hash pre-verification and receiver-side hash post-verification to filter duplicate data; and the weighted OT algorithm engine is used for medical-level authority arbitration and consistency transformation of concurrent annotation operations by multiple users.

[0028] Private data and tile storage layer: This includes a WSI image pyramid server deployed on the local area network (LAN) and a MongoDB consultation and annotation database deployed in the hospital's intranet central computer room. The LAN-based WSI image pyramid server stores WSI tile files in multi-resolution pyramid format (e.g., .svs, .ndpi format) and maintains a tile index mapping table; the MongoDB database stores consultation history sessions, annotation vector data, a global hash fingerprint database, and audit logs containing expert digital signatures, supporting full version history backtracking.

[0029] In this embodiment, the pyramid tile preprocessing and mapping index establishment are as follows: like Figure 2 As shown, on the pathology image server side of the private data and slide storage layer, the original ultra-large field-of-view digital pathology images are preprocessed into a multi-resolution image pyramid structure and then tiled. The physical resolution of each level of the image pyramid decreases sequentially: level 0 is 40× original acquisition resolution, level 1 is 20× scaling, level 2 is 10× scaling, and so on. Each generated slide tile is configured with a unique tile metadata triple:

[0030] in, This represents the physical resolution levels of the image pyramid. The coordinates () represent the grid coordinates of the tile in the corresponding level of the two-dimensional grid.

[0031] The system establishes a mapping index table between the screen display coordinate system and tile metadata on the pathology image server and each terminal. In the front-end rendering engine of each consultation terminal, the system monitors the translation and scaling operations of the canvas view in real time, dynamically converting the screen display coordinate system into the global physical coordinate system of the digital pathology image. Based on the screen coordinate system and tile index mapping table, the system quickly retrieves the set of visible tiles within the Region of Interest (ROI) corresponding to the current viewport. :

[0032] in, and These represent the starting and ending boundary coordinates of the current viewport in the global physical coordinate system of the digital pathology image, respectively. This represents the nearest pyramid level mapped to the current viewport magnification. The system loads only currently visible tiles that are not yet cached locally, significantly reducing network bandwidth consumption during initial loading.

[0033] In this embodiment, the viewport multidimensional state vector abstraction and incremental broadcasting are as follows: During collaborative consultations, the controlling party (e.g., holding a controlling token) holds primary control. Figure 2 The user in terminal A) performs translation, scaling, or rotation operations on the pathology slide canvas. The master terminal listens for changes in the canvas view and abstracts the current viewport presentation state into a five-dimensional state vector:

[0034] The physical meaning and value constraints of each dimension parameter are as follows: ( This represents the normalized high-precision floating-point coordinates of the current viewport center point in the global physical coordinate system of the digital pathology image; It represents the absolute magnification of the current canvas slice. It is a continuous floating-point number and is mapped to the nearest neighbor level of the image pyramid during rendering through an interpolation algorithm. This represents the rotation angle of the current canvas relative to the original slice's reference direction, with a value range of []. ); This represents the local high-precision timestamp (in milliseconds) at which the viewport state was generated on the master control end.

[0035] After the main control terminal (terminal A) detects a view pan / zoom operation, it searches for the tiles {level, x, y} of the currently visible area in real time. For tiles that are not found in the local cache, it sends a request to the local area network tile server via the HTTP / 3 protocol. Figure 2 Step 3), the server returns a lossless tile image ( Figure 2 Step 4), the main control unit performs rendering via local WebGL hardware acceleration ( Figure 2 Step 5), and generate the five-dimensional state vector at the current time. ( Figure 2 Step 6).

[0036] The master control unit's real-time viewport change listener captures interactive operations at a preset sampling frequency of 30Hz and calculates the state difference increment of the five-dimensional state vector at adjacent sampling times. Since the data packet length of each calculated state difference increment is typically less than 100 bytes, the master control unit utilizes a WebRTC data channel (RTCDataChannel, based on the SCTP protocol) deployed privately within a medical private network. Figure 2 Step 7), broadcast the lightweight state differential increment to the collaborating end within the medical private network (e.g., Figure 2(Middle Collaboration Terminal B). This differential incremental broadcast process does not include any screen recording video stream data. Its network overhead is reduced by approximately 99.9% compared to transmitting 1080P / 4K video streams, and it avoids color edge overflow and detail loss caused by video encoding, achieving pixel-level lossless synchronization.

[0037] In this embodiment, the acquisition of tiles on the collaborative end and the local lossless rendering are as follows: like Figure 2 As shown in steps 8 to 14, the collaborating terminal (terminal B) receives the broadcast from the master terminal via the WebRTC data channel. After state difference increment ( Figure 2 Step 8), execute the following collaborative rendering process: 1. Target Viewport Reconstruction and Tile Retrieval: Based on the received state difference increment and the previously known viewport state, the collaborating end reconstructs and calculates the current target viewport state locally, and resolves the viewport center coordinates. Absolute magnification z and rotation angle The collaborating end maps the target viewport center coordinates and magnification to the corresponding image pyramid level, and determines the set of visible tile grid coordinates covered by the current target viewport based on the mapping index table. }( Figure 2 Step 9), where This represents the tile grid coordinates of the target viewport region at the corresponding pyramid level. The collaborating end prioritizes querying the local IndexedDB cache pool. Figure 2 (Step 10) If the corresponding tile image has been found in the cache pool, it is read directly from the local machine to achieve zero network requests and second-level loading.

[0038] 2. Asynchronous request misses tile: For visible tiles that are not found in the local cache pool, the collaborating end initiates an asynchronous tile retrieval request to the local area network tile server based on the HTTP / 3 protocol. Figure 2 Step 11), the server returns lossless tile image data ( Figure 2 Step 12).

[0039] 3. Bicubic Hermite spline interpolation for smooth transition: To avoid screen flickering and visual dizziness caused by instantaneous jumps during large-span translations or rapid scaling on the master end, the collaborative end performs smooth transitions locally using a viewport smooth transition interpolator. Figure 2 Step 13). The interpolator is based on the viewport state vectors received from two sets of adjacent timestamps. and The continuous viewport transition motion trajectory is calculated locally using the bicubic Hermite spline interpolation algorithm:

[0040] in, As a normalized transition factor, satisfying ,in This represents the current interpolation rendering time on the local end of the collaborative device. and These represent timestamps. and The viewport parameter vector contains coordinates, magnification, and angle; and These are the velocity vectors during the switching between the old and new viewports. They can be calculated locally by the cooperating end based on the differential rate of change of the state vectors of adjacent viewports over time, or they can be directly included in the broadcast state differential control metadata.

[0041] 4. Local WebGL hardware-accelerated rendering: The co-working end uses interpolated motion trajectories. Drive smooth updates to the viewport matrix, utilize local GPU and WebGL hardware acceleration capabilities to stitch and reconstruct the acquired lossless tiles and render them onto the local Canvas. Figure 2 Step 14) yields a high-fidelity synchronized browsing screen that is pixel-perfectly consistent with the main control screen.

[0042] The bidirectional hash deduplication transmission control mechanism is as follows: To save network bandwidth and eliminate redundant rendering during synchronous browsing and multi-user annotation, the system deploys hash verification cache modules at both the data sending and receiving ends, constructing a bidirectional hash deduplication transmission decision mechanism, the timing of which is as follows: Figure 3 As shown: 1. Construction of a multimodal unified data structure: When the master or collaborating end performs a viewport panning, tile loading, or primitive annotation operation, the system automatically generates a multimodal unified data structure. Figure 3 Step 1, the example operation notation in the diagram is as follows ,Right now :

[0043] in, The user account identifier for the currently operating user; A local high-precision timestamp generated by the operation; These are the geometric coordinates of the region of interest targeted by the current operation. For specific operation types and primitive geometric attributes; For data hash fingerprints, the calculation rules satisfy:

[0044] That is, by combining the operator identifier, timestamp, region of interest coordinates and operation type using the SHA-256 algorithm, a unique data hash fingerprint is generated.

[0045] 2. Sender-side buffer pre-verification: The sender (e.g., ...) Figure 3 Before transmitting data via the WebRTC data channel, terminal A first retrieves the IndexedDB cache pool on the local browser. Figure 3 Step 2). If an identical hash record already exists in the local IndexedDB, it indicates that the operation has already been transmitted or there has been no substantial state change. The decision-maker directly determines it as duplicate data and terminates the current network transmission. Figure 3 Branch A1), blocking invalid packet transmission; if the local cache pool does not contain the packet. Then execute branch A2, which will... Write the corresponding data to the local IndexedDB and trigger network transmission.

[0046] 3. Verification after buffering at the receiving end: The receiving end (e.g., ...) Figure 3 After terminal B receives the data through the WebRTC data channel, Figure 3 Step 3) does not directly call the rendering pipeline, but instead recalculates the hash fingerprint based on the data content and compares and verifies it with the local IndexedDB cache pool on the receiving end. Figure 3 Step 4). If the match is the same. Records that are considered duplicate data due to multipath propagation are discarded directly. Figure 3 If the data is not hit (B1), the redraw process will not be initiated; if the data is not hit, B2 will be executed, and the data and hash will be written to the local IndexedDB of the receiving end, and subsequent operations transformation (OT) merging and canvas rendering will be submitted.

[0047] Through this bidirectional hash deduplication mechanism, the effective data transmission volume of the system is reduced by 40% to 60%, and the incidence of canvas rendering conflicts caused by repeated operations on multiple terminals is reduced from 15% to below 2%.

[0048] In this embodiment, the Service Worker pre-caching and offline synchronization mechanism is as follows: To overcome consultation interruptions caused by network jitter and temporary network outages, the system introduces a pre-caching and offline synchronization mechanism: 1. Motion velocity vector calculation and future landing point prediction: A ServiceWorker running in the background of the collaborative browser is permanently resident in an independent thread, which monitors and calculates the movement velocity vector of the main control viewport in real time. ,in These represent the rates of change of the horizontal and vertical coordinates of the viewport center point over time. Based on the current velocity vector's direction and magnitude, and combined with a preset look-ahead time adjustment coefficient... (A positive real number used to control the advance displacement of the predicted landing point relative to the current viewport along the movement direction), predicting the viewport landing point area that the master control terminal will move to in the future. :

[0049] in, This represents the current visible viewport space. This refers to the predicted future viewpoint area. A preset look-ahead time adjustment coefficient, with time as its physical dimension, is used to calculate the lead amount from the viewport movement velocity vector to the displacement prediction increment. In actual engineering implementation, The value of can be adaptively and dynamically adjusted based on the local area network's round-trip time (RTT) and the average response time of pyramid slice preloading. The recommended value range is . (Preferred) For example, in a typical embodiment, select This value is equivalent to causing the Service Worker to proactively request future requests along the current viewport roaming direction. The image tiles of the expected landing area ensure that when the main control terminal moves into the area, the collaborating terminal can directly and seamlessly access the local cache to read the tiles, eliminating the white screen waiting during the fast roaming process.

[0050] 2. Offline Queue Storage and Differential Compression Recovery: When a terminal detects a network failure or is in an offline consultation state, the system's Service Worker completely saves all labeled operations and browsing paths of the current terminal in the form of a first-in-first-out operation queue in the local IndexedDB, avoiding the loss of offline operations. Once the WebRTC data channel or LAN connection is restored, the system automatically activates the synchronization engine, extracts the temporarily stored offline queue data, performs compression processing on the offline data using a differential algorithm (reducing the data volume by approximately 70%), and seamlessly synchronizes it to the MongoDB central database and other collaborating terminals in absolute timestamp order for merging, ensuring the continuity of consultations in weak network environments.

[0051] In this embodiment, the collaborative annotation weighted OT algorithm that integrates clinical professional title and medical authority weights is as follows: In collaborative consultation and annotation scenarios, multiple consultation terminals can simultaneously draw lesions or erase areas on the pathological image canvas. To ensure the final consistency of local annotation data across all terminals and to align with clinical decision-making mechanisms, this invention employs a weighted operation transformation (OT) algorithm that incorporates medical permission factors. 1. Clinical professional title medical permission weighting: The system assigns a medical permission weight based on the clinical professional title to each consultation account that logs into the consultation system. :

[0052] In a specific embodiment, the chief physician's authority weight is configured as follows: Associate chief physician is configured as The attending physician is configured as follows The number of resident physicians is .

[0053] 2. Concurrency Conflict Determination and Comprehensive Priority Scoring: The system sets a preset concurrency time window threshold. (Specific value) When different terminals are detected in When concurrent annotation or erasure operations occur within the time window targeting the same physical coordinates or overlapping bounding box regions of the canvas, the system initiates conflict detection and calculates a comprehensive priority score P for each operation:

[0054] in, The preset concurrent time window threshold; The weight of the operator's clinical professional title and medical authority. This is the millisecond-level timestamp when the annotation operation was generated. The starting baseline timestamp created for the current consultation session; and The preset adjustment weight coefficients are set, and the constraint relationship is satisfied. This constraint ensures that medical professional title authority plays an absolutely dominant role in the determination, while minor temporal order only serves as a supplementary arbitration function, preventing minor network transmission jitter from causing user accounts with lower medical authority to preemptively perform operations that override the professional judgment of user accounts with higher medical authority.

[0055] In specific project implementation, to avoid manipulating timestamps Extremely close to the start time of consultation creation When the denominator approaches zero, the numerical value diverges. The denominator term in the formula is configured with a preset small time smoothing bias constant. (e.g., configuration) ,Right now ); or, base time It can be dynamically set as the current concurrent time window. The start timestamp, at this time This represents the millisecond-level relative timing deviation of the operation within the current concurrent time window. Meanwhile, the preset adjustment coefficient satisfies the constraint relationship. Medical access score item The time factor consistently holds a dominant position in the overall score, while it only provides a slight biased reference when the medical authority weights are exactly the same. This prevents user accounts with lower medical authority weights from taking precedence over user accounts with higher medical authority weights due to slight network transmission jitter, thus ensuring the numerical stability and medical authority of the multi-terminal score calculation.

[0056] 3. Conflict operation transformation matrix handling: Cross-permission overlap conflict handling: If annotations from user accounts with different permission levels overlap spatially or conflict in primitive modification (e.g., a user account with lower medical permission weight delineates a lesion, while a user account with higher medical permission weight performs erasure or modification in the same area), the system compares their comprehensive priority score P, retains the operation with the highest score P as the authoritative primitive, and renders it normally on the canvas of each terminal. Operations with lower scores are transformed and updated by higher-permission operations: The annotation primitives corresponding to the lower medical permission weight automatically undergo spatial exclusive adaptive offset on the local canvas of each terminal, and their rendering opacity is automatically reduced to a preset threshold (e.g., reduced to 50% semi-transparent presentation). At the same time, the operation record is pushed into the historical version stack of the MongoDB database for subsequent auditing and backtracking.

[0057] Concurrent processing with the same permissions: If the concurrent operations have the same medical permission weight R (i.e., the same professional title level), the system extracts the unique operation identifier (operation ID) from each operation structure, calculates its hash value, and performs deterministic primitive drifting and transformation based on the numerical partial order relationship of the operation identifier hash value. This ensures that each terminal executes primitive rendering according to the same deterministic rules, guaranteeing that all collaborative terminals achieve an absolutely unified rendering result in a distributed, decentralized, and uncoordinated environment.

[0058] Specifically, the present invention provides a pathological image multi-terminal synchronous browsing and collaborative consultation device based on private WebRTC and multi-dimensional state incremental synchronization. This device is deployed in a medical private network environment, and its specific functional modules are constructed as follows: The pyramid mapping module is deployed on the local area network slicing server. It is used to preprocess full-view digital pathology images into multi-resolution image pyramid tiles on the server and store them within the medical private network, establishing a mapping relationship between the screen coordinate system and the pyramid tile metadata.

[0059] Viewport State Monitoring Module: Deployed on the main control terminal, it is used to monitor viewport operations on the canvas in real time, abstract the current viewport state into a multi-dimensional state vector, and calculate the state difference increment of the multi-dimensional state vector at a preset sampling frequency.

[0060] State Increment Broadcast Module: Deployed at the state increment synchronization and network communication layer, it is used to broadcast state differential increments to collaborating terminals within the medical private network via a WebRTC data channel deployed privately within the medical private network. The state differential increments do not include video stream data.

[0061] Local lossless rendering module: Deployed on each collaborative terminal, it is used to retrieve the set of tile coordinates corresponding to the currently visible area based on the state difference increment after the collaborative terminal receives the state difference increment, obtain the corresponding lossless tile from the local cache pool or asynchronously request it from the server, and perform lossless rendering locally on the collaborative terminal using graphics hardware acceleration.

[0062] Bidirectional hash deduplication module: Distributed at the data sending and receiving ends, it is used to generate data hash fingerprints when viewport change operations or annotation operations occur, retrieve the local cache pool at the sending end to filter duplicate transmissions, and compare the local cache pool at the receiving end to discard duplicate data from multiple propagation.

[0063] Pre-caching and offline synchronization module: running in the background Service Worker thread of each terminal, it is used to calculate the movement speed vector of the viewport of the master terminal, and pre-request the pyramid tiles corresponding to the future landing area from the server according to the prediction formula to store them in the local cache pool; and temporarily store offline operations in the form of queue when the network is interrupted, and synchronize them to the intranet database and other terminals after the network is restored through differential compression.

[0064] Weighted Collaborative Arbitration Module: Deployed in the collaborative conflict resolution and control engine layer, it is used to calculate a comprehensive priority score based on medical permission weights and time attributes when concurrent annotation operations on overlapping areas of the canvas occur within a preset concurrent time window, and to resolve conflicts using an operation conversion algorithm.

[0065] Example 2 Based on the same concept, this invention also proposes a method for multi-terminal synchronous browsing and collaborative consultation of pathological images based on private WebRTC and multi-dimensional state incremental synchronization, including the following steps: Step S101: The server preprocesses the WSI image to be consulted into multi-resolution pyramid tiles and establishes a mapping index table between the screen coordinate system and the tile metadata {level,x,y}. Step S102: The main control terminal monitors the translation / zoom operations on the canvas in real time, converts the screen coordinate system to the WSI global physical coordinate system, retrieves the currently visible tile set, and loads only the visible and uncached tiles; Step S103: The master control terminal abstracts the current viewport state into a five-dimensional state vector. The state differential increment is calculated at a preset sampling frequency of 30Hz and broadcast to the collaborating end through a private WebRTC data channel; Step S104: The collaborating end receives the state differential increment, reconstructs the target viewport state, calculates the target visible tile coordinate set, queries the local IndexedDB cache, and asynchronously requests the server for the missing tiles via HTTP / 3 protocol. Step S105: The collaborative end calculates the continuous transition motion trajectory based on the state vector and velocity vector of the adjacent viewport using the bicubic Hermite spline interpolation algorithm, and performs local pixel-level lossless rendering through WebGL hardware acceleration to achieve synchronized browsing with the main control end at the pixel level; Step S106: The operation data generated during the synchronous browsing and annotation process is verified by a bidirectional hash deduplication mechanism and then merged and rendered on multiple devices using the OT algorithm; Step S107: When concurrent annotation conflicts occur on multiple terminals, arbitration is carried out using the weighted OT algorithm based on medical permissions to achieve global consistency of the annotation canvas on multiple terminals; Step S108: The annotation and browsing data during the network interruption are temporarily stored in a queue. After the network is restored, they are synchronously merged in the order of timestamps after differential compression.

[0066] In summary, this invention operates within a private medical network, physically isolated from the external public network. Sensitive pathological data does not flow through external networks, eliminating the risk of data leakage. Within the same local area network environment, the end-to-end latency for multi-terminal synchronous browsing is consistently between 60 and 180 ms, significantly lower than existing video conferencing screen sharing methods (latencies typically exceeding 200 ms) and polling full-data synchronization methods (latencies reaching 300-500 ms under high concurrency). The length of a single incremental status data packet is less than 100 bytes, reducing overall network overhead by more than 50% compared to continuous transmission of 1080P / 4K video streams (typically requiring 2-8 Mbps bandwidth). Rendering is entirely based on local pyramid lossless tile reconstruction, eliminating cell microstructure blurring and color overflow caused by video encoding transcoding, ensuring pixel-level lossless high-fidelity presentation. Combined with bidirectional hash deduplication and medical permission-weighted OT algorithms, the time for a single complex collaborative consultation is reduced from over 40 minutes to less than 20 minutes.

[0067] Example 3 This invention also provides an electronic device, including a memory and a processor. The memory may be a high-speed random access memory (RAM) or a non-volatile memory (e.g., disk storage devices, flash memory devices, or other non-volatile solid-state storage devices); the memory stores a computer program; the processor is configured to run the computer program to execute the various steps of the pathological image multi-terminal synchronous browsing and collaborative consultation method based on proprietary WebRTC and multi-dimensional state incremental synchronization described in the above method embodiments.

[0068] This invention also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, such as a disk, optical disk, read-only memory (ROM), or random access memory (RAM). The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the entire process of the pathological image multi-terminal synchronous browsing and collaborative consultation method based on private WebRTC and multi-dimensional state incremental synchronization described in the above method embodiments.

[0069] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 1 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0070] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for simultaneous multi-terminal browsing and collaborative consultation of pathological images, characterized in that, The method, applied to a collaborative consultation system deployed within a private medical network, includes the following steps: On the server side, the full-view digital pathology images are preprocessed into multi-resolution image pyramid tiles, and a mapping relationship between the screen coordinate system and the tile metadata is established. The main control unit monitors the viewport operation on the canvas in real time, and abstracts the current viewport state into a multi-dimensional state vector that includes the normalized floating-point coordinates of the viewport center point in the global physical coordinate system of the digital pathology image, the absolute magnification of the current slice, the canvas rotation angle, and the local timestamp, and calculates the state difference increment of the multi-dimensional state vector at a preset sampling frequency. The master control terminal broadcasts the state differential increment to the collaborating terminal within the medical private network through a WebRTC data channel privately deployed within the medical private network, wherein the state differential increment does not include video stream data; After receiving the state difference increment, the collaborating end retrieves the set of tile coordinates corresponding to the currently visible area based on the state difference increment, obtains the corresponding lossless tiles from the local cache pool or asynchronously requests them from the server, and performs lossless rendering locally on the collaborating end using graphics hardware acceleration to achieve pixel-level lossless synchronous linkage browsing and collaborative consultation of multiple terminal screens.

2. The method for simultaneous multi-terminal browsing and collaborative consultation of pathological images as described in claim 1, characterized in that, The preset sampling frequency is 30Hz; the state difference increment of the multidimensional state vector is generated by the master control terminal by calculating the difference values ​​of each dimension parameter in the multidimensional state vector at adjacent sampling times.

3. The method for simultaneous multi-terminal browsing and collaborative consultation of pathological images as described in claim 1, characterized in that, The collaborative terminal retrieves the set of tile coordinates corresponding to the currently visible area and obtains lossless tiles, including: The calculated coordinates of the target viewport center point and the absolute magnification are mapped to the corresponding image pyramid level, and the set of visible tiles corresponding to the current viewport region is determined based on the mapping relationship. The system checks if the corresponding tile exists in the local cache pool. If not, it asynchronously requests the uncached tile from the server using the HTTP / 3 transport protocol of the User Datagram Protocol and performs pixel-level lossless rendering locally using WebGL hardware acceleration.

4. A method for simultaneous multi-terminal browsing and collaborative consultation of pathological images as described in claim 1 or 2, characterized in that, Before performing lossless rendering, the method further includes: The collaborative end calculates continuous viewport transition motion trajectories locally using the bicubic Hermite spline interpolation algorithm based on the received two adjacent sets of time-stamped viewport state vectors and their corresponding velocity vectors, and drives the rendering process with the viewport transition motion trajectories to complete the viewport switching transition in a smooth manner.

5. The method for simultaneous multi-terminal browsing and collaborative consultation of pathological images as described in claim 1, characterized in that, The method also includes a bidirectional hash deduplication step: When a viewport change operation or annotation operation occurs at the data sending end, the operator identifier, operation timestamp, region of interest coordinates and operation type are merged and a data hash fingerprint is generated by hash algorithm. The data hash fingerprint is then retrieved from the local cache pool at the sending end. If the same data hash fingerprint is found, it is determined to be a duplicate operation and the current network transmission is terminated. If no match is found, the data hash fingerprint and the corresponding data are written to the local cache pool at the sending end and network transmission is triggered. After receiving data, the data receiving end recalculates the data hash fingerprint and compares it with the local cache pool of the receiving end. If a match is found, it is determined to be duplicate data in multi-path propagation and is discarded directly. If a match is not found, it is written to the local cache pool of the receiving end before subsequent merging and rendering are performed.

6. The method for simultaneous multi-terminal browsing and collaborative consultation of pathological images as described in claim 1, characterized in that, The method also includes pre-caching and offline synchronization steps: The pre-caching thread running in the background of the collaboration terminal calculates the viewport movement speed vector of the master terminal in real time. Based on the viewport movement speed vector and the look-ahead time adjustment coefficient, it predicts the future viewport landing area and requests the pyramid tile corresponding to the viewport landing area from the server in advance and stores it in the local cache pool. When the terminal detects a network connection interruption, it temporarily stores the locally generated annotation operations and browsing path data in the form of a queue to the local cache pool. When the network connection is restored, differential compression is performed on the temporarily stored offline data, and the data is synchronized to the intranet database and other collaborative terminals in chronological order according to timestamps for merging.

7. The method for simultaneous multi-terminal browsing and collaborative consultation of pathological images as described in claim 1, characterized in that, The method also includes steps for resolving conflicts in collaborative annotation among multiple users: Pre-configure medical permission weights based on clinical professional title levels for each consultation account logging into the system; When multiple terminals are detected to be performing concurrent annotation operations on the overlapping area of ​​the canvas within a preset concurrent time window, a comprehensive priority score for each concurrent operation is calculated based on the medical permission weight of each terminal account and the time attribute of each operation, wherein the weight of the medical permission weight in the comprehensive priority score is higher than the weight of the time attribute. The operation with the highest overall priority score is retained by the operation transformation algorithm. The operations with lower overall priority scores covered by the retained operations are transformed and presented locally on the collaboration side by reducing the display opacity and performing spatial exclusive adaptive offset. At the same time, the operations with lower overall priority scores are pushed into the history version stack. When concurrent operations have the same medical permission weight, the deterministic execution sequence and primitive transformation of each operation are determined based on the hash value of the unique operation identifier of each operation.

8. A device for simultaneous browsing and collaborative consultation of pathological images across multiple terminals, characterized in that, Deployed within a private healthcare network, including: The pyramid mapping module is used to preprocess full-view digital pathology images into multi-resolution image pyramid tiles on the server side and establish a mapping relationship between the screen coordinate system and the tile metadata. The viewport status monitoring module is used by the main control terminal to monitor the viewport operation on the canvas in real time. It abstracts the current viewport status into a multi-dimensional state vector that includes the normalized floating-point coordinates of the viewport center point in the global physical coordinate system of the digital pathology image, the absolute magnification of the current slice, the canvas rotation angle, and the local timestamp. It also calculates the state difference increment of the multi-dimensional state vector at a preset sampling frequency. The status increment broadcast module is used to broadcast the status differential increment to the collaborating end in the medical private network through the WebRTC data channel deployed privately in the medical private network, wherein the status differential increment does not include video stream data; The local lossless rendering module is used to retrieve the set of tile coordinates corresponding to the currently visible area based on the state difference increment after the collaborative end receives the state difference increment, obtain the corresponding lossless tiles from the local cache pool or asynchronously request them from the server, and perform lossless rendering locally on the collaborative end using graphics hardware acceleration to achieve pixel-level lossless synchronous linkage browsing and collaborative consultation of multi-terminal screens.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the method for multi-terminal synchronous browsing and collaborative consultation of pathological images as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the method for synchronous browsing and collaborative consultation of pathological images across multiple terminals as described in any one of claims 1 to 7.

Images