Image tile rendering and video memory scheduling method and system and readable storage medium thereof

Abstract

The application provides an image tile rendering and video memory scheduling method and system and a readable storage medium thereof, and belongs to the technical field of computer graphics. In order to solve the problems of rendering delay and video memory overflow during super high resolution image browsing, the application collects the speed and acceleration of the viewport coordinate at a high frequency, dynamically calculates a prediction time window and outputs a predicted viewport bounding box; the intersection tile priority score is calculated and a preloading queue is generated; a background asynchronous thread decodes the image according to the queue and silently uploads the image to the video memory of the graphics processor, meanwhile, the reverse predictive de-weighting elimination is performed on the resident tile based on the motion direction; finally, the main rendering thread directly hits the ready texture in the video memory to complete the drawing. The application can be used for smooth browsing of super resolution images such as digital pathology.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and digital image processing technology. Specifically, it relates to a method for dynamic pre-rendering of image tiles and low-level video memory scheduling management of super-resolution images (such as WSI panoramic pathological slides) during interactive browsing. Specifically, it is an image tile rendering and video memory scheduling method, system, and readable storage medium. Background Technology

[0002] In fields such as digital pathology and remote sensing mapping, super-resolution images (such as WSI panoramic slices at the tens of gigapixel level) generate extremely large amounts of data, with single images typically exceeding gigabytes in size. To enable real-time viewing of such images on devices with limited computing power, the industry typically employs an image pyramid model, which cuts the images into massive amounts of standard-sized image tiles at different scaling levels for storage.

[0003] In existing technologies, mainstream super-resolution image browsing systems generally adopt a passive "lazy loading" strategy. This means that the system only triggers disk I / O reading, CPU / GPU decoding, and VRAM uploading processes for the corresponding tiles when the user's viewport moves to a specific coordinate area. However, due to the inherent physical latency of I / O and decoding of high-resolution image tiles, this "load as you see" approach inevitably leads to large areas of rendered "blank tiles" on the screen when the user quickly drags or zooms the viewport, severely reducing the smoothness of interaction. Furthermore, existing lazy loading solutions typically lack a proactive cache management mechanism. When the viewport switches frequently, old tiles are not cleared in time, and the loading of new tiles is frequently blocked, easily causing VRAM jitter, and even leading to memory overflow (OOM) and system crashes. Summary of the Invention

[0004] This application provides an image tile rendering and video memory scheduling method, system and its readable storage medium. The existing passive on-demand loading mechanism lacks forward-looking viewport prediction and fine-grained video memory management, which can easily cause video memory overflow during high-speed interaction. Moreover, due to I / O and decoding delays, it can lead to serious problems such as white blocks in screen rendering and UI lag.

[0005] The core technology of this invention is to adaptively adjust the preload time window and spatial weight by extracting the viewport motion vector and tangential acceleration in real time, and combine it with the background zero-copy silent video memory upload and reverse predictive elimination mechanism to achieve precise tilting of computing resources toward the actual motion intention and zero-latency scheduling of video memory.

[0006] In a first aspect, this application provides an image tile rendering and video memory scheduling method, the method comprising the following steps:

[0007] Acquire user interaction input and frequently collect viewport coordinate data, and calculate viewport motion vector data accordingly. The motion vector data includes at least movement speed and acceleration. Based on the moving speed and acceleration, the prediction time window is dynamically calculated, and the prediction viewport bounding box within the prediction time window is determined using a preset trajectory prediction model. In the image pyramid hierarchical network, a set of tiles to be loaded that intersect with the predicted viewport bounding box is determined. Based on the distance from the tile center to the viewport center and the angle between the tile position and the direction of motion, combined with dynamic adaptive weights related to the movement speed, the priority score of each tile in the set of tiles to be loaded is calculated. A task queue is generated based on priority scores. The tiles are decoded in an asynchronous background thread, and the decoded pixel data is uploaded to the graphics processor's memory in a non-blocking and silent manner to generate textures. At the same time, predictive elimination is performed on the resident tiles in the memory based on the current motion direction. When the viewport moves to the corresponding area, the main rendering thread directly extracts the tile textures that are ready in the video memory to complete the rendering.

[0008] Furthermore, the process of dynamically calculating the prediction time window based on movement speed and acceleration includes: Extract the projection of acceleration onto the direction of movement velocity to calculate tangential acceleration, which is used to characterize the user's motion intention; Based on the sign and magnitude of the tangential acceleration, the motion intention is divided into the acceleration phase, the constant speed fine-tuning phase, and the emergency stop phase. For different stages, a piecewise mapping function is used to dynamically adjust the prediction time window. In the acceleration stage, the prediction time window is extended to reserve tiles in advance, while in the emergency stop stage, a nonlinear decay function is applied to shorten the prediction time window by a sudden drop, thereby truncating the preloading task.

[0009] Furthermore, the process of calculating the tile priority score by incorporating dynamic adaptive weights related to movement speed includes: Assign a first weight to the topological distance between the tile center and the viewport center, and assign a second weight to the angle between the tile position and the direction of motion, with the sum of the first weight and the second weight being a constant; The system monitors the movement speed in real time and performs dynamic adaptive adjustment: when the speed is greater than the high-speed threshold, the second weight is increased and the first weight is decreased to prioritize loading the tiles directly in front of the movement; when the speed is less than the low-speed threshold, the first weight is increased and the second weight is decreased to prioritize loading the tiles around the center of the viewport. To prevent the center tile of the viewport from being mistakenly eliminated due to its low weight, the values ​​of the first and second weights are strictly limited to a preset closed interval that is greater than zero and less than one.

[0010] Furthermore, the preset trajectory prediction model is a lightweight Kalman filter model, whose state vector contains position, velocity, and acceleration information. When an abrupt stop is detected to produce a large reverse jerk, the acceleration variance term corresponding to the process noise covariance matrix in the model is dynamically increased, forcing the model gain to converge to the current stationary observation value to avoid incorrect predictions caused by inertia.

[0011] Furthermore, the process of silently uploading the decoded pixel data to the graphics processor's video memory in a non-blocking manner to generate textures includes: Create an auxiliary shared context in a background asynchronous thread that communicates with the main rendering thread for resources; By using pixel buffer objects in conjunction with a direct memory access controller, the decoded pixel data is transferred to video memory in a non-blocking manner as an asynchronous hardware stream. After the transfer is complete, the corresponding texture state is marked as ready only through the auxiliary shared context, and drawing instructions that trigger screen redraw are prohibited from being called during this process.

[0012] Furthermore, the process of performing predictive replacement of resident tiles in memory based on the current motion direction includes: Real-time monitoring of video memory usage; when video memory usage reaches a dynamically set eviction threshold, triggering the eviction mechanism and establishing a doubly linked list of resident tiles. Calculate the cosine of the direction angle between the position vector of the tile residing in the video memory and the current viewport motion direction vector; The location of each stationary tile is defined based on the cosine of the directional angle, and its survival weight is recalculated. The tile with the lowest survival weight is moved to the end of the linked list. The main rendering thread releases the memory handle of the tail tile of the linked list during the vertical synchronization interval of the next frame.

[0013] Furthermore, the process of recalculating its survival weight includes: When a stationary tile is located in a remote area directly behind the movement, its survival weight is penalized and downgraded. When a resident tile is in the reserved area directly ahead of the movement, downgrading is prohibited even if the tile is at the end of the list, in order to ensure the image cache on the path that the user is about to reach.

[0014] Furthermore, the water level threshold is dynamically calculated and determined based on the total physical video memory of the graphics processor after obtaining the total physical video memory during the initialization phase.

[0015] Secondly, this application provides an image tile rendering and video memory scheduling system, including: The input monitoring and feature extraction module is used to acquire viewport coordinate data at high frequency and calculate motion vector data including movement speed and acceleration; The trajectory and viewport prediction module is used to dynamically adjust the prediction time window based on movement speed and acceleration, and output the prediction viewport bounding box using the trajectory prediction model; The priority scheduling module is used to determine the tiles to be loaded that intersect with the predicted viewport bounding box, and calculate the priority score of each tile by combining dynamic adaptive weights associated with the movement speed to generate a task queue. The background asynchronous processing and video memory management module is used to perform image decoding and silent video memory upload based on pixel buffer objects in the background thread according to the queue, and to perform predictive video memory eviction on resident tiles in combination with the motion direction. The main rendering engine module is used to directly call the textures in the ready state in the video memory for zero-latency rendering when the viewport actually arrives.

[0016] Thirdly, this application provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the image tile rendering and video memory scheduling method described above.

[0017] The main contributions and innovations of this invention are as follows: 1. Effectively eliminate rendering white blocks and achieve seamless and smooth roaming: Utilizing a high-precision trajectory prediction model and a dynamic time window Δt, the system completes background decoding in advance before the viewport arrives. Combining PBO (Pixel Buffer Object) and DMA (Direct Memory Access) technology, hardware-level silent uploading of pixel data is achieved, pre-empting image rendering time. This allows the main thread to directly hit the texture handle in the Ready state when the viewport arrives, achieving zero-latency on-screen display and significantly improving the "white block" problem.

[0018] 2. Decoupling the UI main rendering thread to eliminate frame drops and stuttering: Through a multi-threaded asynchronous architecture and state decoupling design, the extremely computationally intensive image decoding and VRAM transfer tasks are completely offloaded to the background asynchronous scheduler. Since no blocking drawing instructions are invoked, the UI main thread focuses on a high-speed rendering loop (V-Sync), ensuring that the system's interactive frame rate remains stably locked above 60fps even when faced with large-scale, drastic image dragging.

[0019] 3. Overcoming memory bottlenecks and achieving high-level scheduling to prevent overflow: A predictive LRU (Least Recently Used) memory eviction strategy is introduced, abandoning the traditional pure time-based LRU. By mathematically calculating the angle between a tile and the opposite direction of movement, tiles are precisely demoted and evicted from the area directly behind and far away. Simultaneously, combined with dynamic weight calculations based on the "tunneling effect" during rapid dragging, invalid tiles on the sides are deprived of loading priority. This allows the system to maintain a cache hit rate as high as 96.5% even with extremely low physical memory usage (e.g., only 1GB), effectively eliminating memory thrashing and overflow risks.

[0020] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the system architecture of the image tile rendering and video memory scheduling method according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the intersection calculation of motion vectors and the predicted viewport according to an embodiment of this application; Figure 3 This is a schematic diagram of a predictive memory replacement strategy based on the direction of motion according to an embodiment of this application. Figure 4 This is a general flowchart of the image tile rendering and video memory scheduling method according to an embodiment of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0023] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0024] To address the technical shortcomings of existing "on-demand loading" strategies in browsing ultra-large-scale images, such as rendering white blocks, stuttering, and even memory jitter or overflow caused by physical delays in disk I / O reading, CPU / GPU decoding, and video memory uploading, this invention provides an image tile rendering and video memory scheduling method, system, and its readable storage medium.

[0025] The technical solution of this invention deeply integrates the shared context asynchronous transmission mechanism of pure computer graphics with the memory / video memory scheduling algorithm, realizing the "zero-perception" smooth loading of tiles.

[0026] Example 1 This application aims to propose an image tile rendering and video memory scheduling method, specifically, referring to... Figure 1 and Figure 4 The method includes the following steps: Step S1: High-frequency acquisition and feature extraction of viewport motion vector data (i.e. Figure 4 (High-frequency acquisition viewport coordinates) The system's main rendering engine monitors user input devices (such as mouse dragging and touchpad swiping) through an input listening module, and frequently collects the screen coordinates of the current viewport center point at fixed time intervals (e.g., every 16ms, corresponding to a 60Hz screen refresh rate). Based on the continuous time-series coordinate data, the system calculates the instantaneous state in real time, extracting the current viewport's movement speed magnitude v, movement velocity direction, and acceleration vector. At the same time, basic speed and acceleration thresholds are set to filter out minute hand tremors and accurately identify translation or scaling motion intentions.

[0027] Step S2: Future viewport prediction based on a lightweight temporal model (i.e. Figure 4 (Predictive model and adaptive parameter mapping in the text) Combination Figure 2 As shown, the system does not use a fixed preload time, but rather a dynamically adjusted prediction time window. Using a lightweight temporal model, the "predicted viewport bounding box" that the future viewport will reach in the WSI panoramic coordinate system is calculated. This step includes the core physical parameter mapping mechanism of this invention: First, calculate the intended characteristic value, i.e., the tangential acceleration. The system uses the velocity vector of the current viewport. and acceleration vector The projection of acceleration onto the velocity direction is calculated using the following formula:

[0028] in, and These represent the velocity components of the current viewport movement speed in the horizontal (x-axis) and vertical (y-axis) directions in the screen's two-dimensional coordinate system, respectively. and These represent the acceleration components of the current viewport movement acceleration in the horizontal (x-axis) and vertical (y-axis) directions, respectively. The magnitude of the velocity vector represents the absolute scalar velocity of the viewport being dragged by the user.

[0029] when At this time, it indicates that the user is in the "accelerated throw" viewpoint; when This indicates that the user is in the process of "suddenly braking" and searching for the target lesion area; when At this time, it represents constant speed cruising or fine-tuning.

[0030] Subsequently, the system presets a reference time window. (Preferred time: 0.3 seconds) Minimum window (Preferred time: 0.1 seconds) and maximum window (Preferred 0.8 seconds), and the following piecewise nonlinear mapping function is executed to calculate the prediction time window. : 1. If (In the accelerated throwing phase), a linear gain extension window is used: ; 2. If (During the constant speed or fine-tuning phase), maintain the reference window: ; 3. If (During the emergency braking phase), an exponential descent is used to shorten the window:

[0031] in, The base of the natural logarithm (a mathematical constant) is used to construct the natural exponential decay model in physics. Acceleration threshold. The preferred setting is 5000 pixels per square second, which is the critical value for emergency braking. The preferred setting is 8000 pixels per square second, with a linear gain coefficient. Set as The exponential decay coefficient λ is set to .

[0032] The aforementioned asymmetric model with exponential decay not only pre-books background worker threads during acceleration, but more importantly, it leverages [its capabilities] during emergency stops. Its exponential drop characteristic instantly... Compress to 0.1 seconds, and immediately send a truncation command to the background asynchronous scheduler to prevent memory overflow caused by inertia to continue loading useless tiles ahead.

[0033] In calculation Then, it is input into a preset trajectory prediction model to generate a predicted viewport bounding box. Preferably, this prediction model is a lightweight Kalman filter model, whose (system) state vector It is set as a six-dimensional vector containing position, velocity, and acceleration:

[0034] The observation vector is a two-dimensional vector:

[0035] To support the unique acceleration-based dynamic adjustment preloading mechanism of this invention, the system state equation of the lightweight Kalman filter is set as follows: Combined with the base time corresponding to the UI rendering frame rate Its core state transition matrix F is strictly limited to:

[0036] Here, the subscript k represents the current discrete time step, and the subscript k-1 represents the previous discrete time step. The vector represents the process noise of the system at time k, and its covariance is the process noise covariance matrix Q. The observation noise vector represents the system at time k, and its covariance corresponds to the observation noise covariance matrix R.

[0037] Meanwhile, since the front-end input device can only acquire the two-dimensional coordinates of the screen, its observation equation is set as follows: The dimension-reduced observation matrix H is strictly constrained as follows:

[0038] To achieve a high degree of coordination between the algorithm and the underlying video memory hardware, this invention performs specific dynamic modifications and constraints on the noise covariance matrix of the aforementioned Kalman filter model: Firstly, addressing the issue of accidental memory loading due to slight hand tremors in high-DPI mice on high-resolution screens, this invention imposes a low-level constraint on the observation noise covariance matrix R. It initializes it as a diagonal matrix. ,in and The value is taken as 2 to 5 pixels equivalent to the front-end UI viewport, representing the variance of the observation position error of the front-end input device in the x-axis and y-axis directions of the screen, respectively. This forms a physical debouncing valve at the graphics level, ensuring that background video memory preloading scheduling is triggered only when a real and intentional dragging action occurs.

[0039] Secondly, to overcome the instantaneous memory overflow defect caused by prediction inertia in conventional Kalman filter models, this invention configures the process noise covariance matrix Q as a dynamic variable strongly coupled with the kinematic state. Specifically, when the user suddenly stops dragging, i.e., when a large reverse jerk (the derivative of acceleration, Jerk, also known as jerkiness or pulsation) is generated, if the constant Q is maintained, the model will continue to predict a long trajectory forward due to inertia. Therefore, this invention sets the variance term corresponding to acceleration in the Q matrix to a dynamic value proportional to the current input jerk. When an abrupt stop is detected, the system instantaneously increases the Q value, forcing the Kalman gain to converge rapidly to the observation coordinates where the system is currently stationary. This sudden change in the underlying mathematical parameters completely eliminates inertial prediction based on historical velocity, thereby instantly cutting off the invalid preloading tasks of the background worker thread and greatly protecting the memory resources of the current viewport focus area.

[0040] Preferably, the prediction model uses a simplified version of the LSTM network. To ensure that the main thread is not blocked, the network structure is strictly limited: 1. The input dimension is fixed at (T, 4), corresponding to the past T time steps ( Viewport increment feature sequence ; 2. The output dimension is fixed at (1,2), directly representing the relative offset of the viewport center within the future prediction time window. ; 3. The network depth is strictly limited to a single-layer LSTM unit structure to eliminate forward latency in temporal computation. The number of hidden layer units is limited to between 32 and 64. The output of the last time step of the single-layer LSTM is connected to a fully connected layer containing 2 neurons, without using any activation function.

[0041] In this topology, taking 64 hidden units as an example, the formula for calculating the total number of parameters in this model is:

[0042] These constraints limit the model's total parameters to less than 20,000, its memory footprint to less than 100KB, and the physical limitation of a single forward inference cycle to between 0.5 and 1 millisecond. This explicit network topology constraint ensures that the UI main thread can simultaneously complete model inference, tile intersection detection, and VRAM texture mapping within one refresh cycle (16.6ms), completely avoiding UI stuttering and frame drops caused by the introduction of AI models.

[0043] Step S3: Tile intersection detection and dynamic priority weight calculation (i.e. Figure 4 (Tile intersection detection and dynamic scoring in China) After obtaining the predicted viewport bounding box output in step S2, the system does not blindly load all image data within that region, but instead combines... Figure 2 The diagram shown illustrates the calculation of the intersection of the motion vector and the predicted viewport. In the corresponding level of the WSI panoramic image pyramid model, it calculates all the sets of tiles that spatially intersect with the bounding box.

[0044] To maximize the use of limited GPU bus bandwidth and video memory resources during high-speed roaming and avoid rendering latency caused by I / O queuing, the system dynamically prioritizes each tile in the set. The specific priority score calculation formula is as follows:

[0045] in, This represents the topological distance variable between the tile center and the predicted viewport center (the closer the distance, the higher the base score for this item). The variable represents the cosine of the angle between the tile position and the current viewport motion direction vector (tiles directly in front of the moving object score the highest, followed by those to the side, and those behind the moving object score the lowest). Topological distance weights; Weights are assigned to the direction of motion.

[0046] Conventional preloading schemes typically include and Setting it to a fixed ratio (e.g., 0.5 each) presents an irreconcilable physical contradiction in the actual engineering of ultra-high resolution image browsing: if the fixed weight focuses on distance, the system will waste computing power to load invalid tiles on the side of the viewport during extremely fast dragging, causing the target tile in front to fail to load; if the fixed weight focuses on direction, low-speed fine-tuning will cause loading blind spots in other directions around the center of the viewport.

[0047] To address this, the present invention introduces a weight adaptive dynamic adjustment mechanism based on instantaneous scalar velocity v. and It is not a static constant, but a dynamic adaptive variable strongly coupled with the user's instantaneous drag speed v. Its core logic is: the faster the movement speed, the greater the directional weight. The higher the center distance, the slower the movement speed, and the greater the weight of the center distance. The higher the threshold, the better. The system sets a low-speed threshold. (e.g., 500px / s) and high-speed threshold (For example, 3000px / s), and perform the following dynamic mapping: 1. When (During the low-speed fine-tuning period): =0.3, =0.7. The system smoothly reverts to "center-radial mode", prioritizing the loading of video memory on the tiles around the center of the viewport and ensuring their resident status.

[0048] 2. When At the time (during the high-speed dragging period): =0.7, =0.3. The system scheduling strategy instantly switched to "aggressive tunnel mode", which heavily favored and prioritized loading the tiles directly in front of the device, depriving the side tiles of loading priority, and avoiding invalid I / O occupying video memory.

[0049] 3. When Time (in the smooth acceleration period): Perform linear or smooth nonlinear interpolation mapping between 0.3 and 0.7, and strictly satisfy the following conditions: =1.0- .

[0050] It is particularly important to emphasize that this invention will, at the underlying architecture level, and The value range of is strictly defined within a closed interval of 0.3 to 0.7, while its extreme values ​​of 0 or 1.0 are absolutely prohibited. This limitation of the parameter boundary solves a crucial flaw: if allowed... If the score is reduced to 0, when the user suddenly stops during rapid dragging, the tile at the center of the viewport (i.e., the closest in absolute distance) will be mistakenly released by the low-return (LRU) memory policy due to its 0 score, resulting in a severe "white block in the center" phenomenon. Retaining a minimum base weight of 0.3 ensures that even in the most aggressive "tunnel prediction mode," the system can still maintain basic rendering of the current viewport core area.

[0051] After completing the dynamic adaptive weight allocation and scoring, the system strictly sorts the set of tiles to be loaded according to their Score values ​​from high to low, thereby generating a "preload priority task queue" to be sent to the background Worker thread pool (i.e., the background asynchronous thread pool). Figure 4 The multi-threaded hardware scheduling and silent processing engine in the system.

[0052] Step S4: Priority-based background silent scheduling and video memory management After generating the "preloaded task queue," to significantly improve the UI stuttering and frame drops caused by the main thread handling tile loading in traditional solutions, this invention designs a low-level hardware and software coordination mechanism that starts a background asynchronous worker thread pool to process the queue without blocking the main rendering thread (UI main thread). This step mainly includes two core execution logics: "zero-copy silent video memory upload" and "predictive LRU memory eviction." The specific underlying mechanism is as follows: Figure 3 and Figure 4 The background asynchronous thread pool module is shown. The background asynchronous scheduler processes the task queue without blocking the main rendering engine's UI thread.

[0053] On the one hand, the zero-copy silent upload mechanism based on shared context and PBO (i.e. Figure 4 (S4.2 in the text) The background asynchronous worker thread reads the image stream from the disk or network based on the priority of the task queue, and calls the CPU multi-thread or GPU hardware decoder to decode the image into raw pixel data (e.g., RGBA format).

[0054] To achieve "silent upload" to the graphics processor's video memory (VRAM) and avoid blocking the main UI thread, the system establishes an auxiliary shared context in the background worker thread pool that communicates with the main rendering engine (such as WebGL or OpenGL). Specifically, this is OffscreenCanvas in the WebGL environment and wglShareLists in the OpenGL environment. After completing tile decoding, the background thread never calls traditional texture interfaces (such as glTexImage2D) that would block the main thread. Instead, it writes the decoded pixel data to a pixel buffer object (PBO). Using a direct memory access controller (DMA), the system non-blockingly transfers the data to the GPU's video memory to generate texture objects in the form of an asynchronous hardware stream.

[0055] After the transmission is complete, state decoupling and silent logic are implemented: the auxiliary shared context only marks the state corresponding to the video memory texture as "Ready" (i.e., ...). Figure 4 In section S4.3, this process strictly prohibits the invocation of any screen drawing instructions (DrawCall) or buffer swapping instructions. This ensures that the underlying decoding and memory movement of tiles have no physical interference with the drag-and-drop interaction of the front-end user.

[0056] On the other hand, a predictive memory eviction strategy (Predictive LRU) that incorporates real-time motion physics features is used. Figure 4(S4.1 in the text) To address the technical challenge of memory overflow (OOM) caused by the accumulation of numerous preloaded tiles during ultra-high resolution image browsing, this invention proposes a "predictive weighted elimination mechanism" that incorporates real-time motion physics vectors, building upon the traditional time-based LRU (Least Recently Used) algorithm. The specific implementation logic is as follows: 1. Adaptive dynamic calibration of video memory threshold: The system abandons the fixed percentage threshold and instead dynamically queries the total physical video memory of the current host machine's GPU when initializing the rendering engine (e.g., through the WebGL extension WEBGL_debug_renderer_info). Based on this, the system adaptively calculates and sets two dynamic thresholds: Safe water level If the value is lower than this, the eviction mechanism will not be triggered to preserve the cache for users to quickly roll back; Expulsion water level When the video memory usage reaches this extreme level, a predictive eviction operation is immediately triggered.

[0057] 2. The rigorous mathematical definition of LRU doubly linked lists and "reverse direction": The system maintains a doubly linked list residing in tiles, with the head of the list being the most recently visited node and the tail being the least recently visited node. When a tile is touched... Instead of brute-force truncation from the tail, the system starts a lightweight background traversal thread to recalculate the "survival weight" of the tiles in the latter half of the linked list (e.g., the last 50%). The system obtains the screen coordinates of the current viewport center. With instantaneous velocity vector Simultaneously, iterate through the linked list nodes to obtain the center coordinates of each resident tile in the global coordinate system. This allows us to construct the position vector of the tile relative to the current viewport position. .

[0058] Next, the system calculates the position vector of the stationary tile. With the current velocity vector The cosine value of the included angle:

[0059] Here, the subscript i represents the i-th resident tile node traversed in the doubly linked list of memory replacements. Represents the absolute linear physical distance (i.e., position vector) from the center of the i-th stationary tile to the current viewport center. (Modulus length).

[0060] according to Figure 3 Region division of the planar view of the video memory resident state: 1. If (i.e., an angle greater than 120 degrees, located in the strictly opposite direction, directly behind and far from the region), the system assigns its survival weight. Punitive downgrading (e.g., multiplying by a factor of 0.1); 2. If (i.e., the side-rear blind zone with an angle between 90 degrees and 120 degrees), its survival weight is multiplied by an appropriate penalty coefficient (preferably 0.5). 3. If (i.e., the reserved area directly in front with an angle of less than 90 degrees) indicates that even if the tile is at the end of the linked list based on the pure time dimension, it is also located on the path that the user is about to reach in spatial physics. The system absolutely prohibits it from being downgraded, and even assigns a residence compensation coefficient (preferably 1.5) to ensure the residence of the memory in front.

[0061] After recalculation, the background thread performs a quick sort on the latter half of the linked list based on survival weights, forcibly moving the tile node with the lowest absolute weight to the physical tail of the list. Finally, during the V-Sync interval of the next frame, the main rendering thread safely calls the underlying graphics API to release the memory handles of the last N tiles of the linked list, until the overall memory usage smoothly drops back to a safe level. .

[0062] Step S5: Main thread performs on-demand high-speed rendering loop (V-Sync) like Figure 4 As shown in the S5 module, when the user's actual viewport reaches the preloaded area, the UI main rendering thread determines the tiles to be displayed in the ultra-fast rendering loop. Since the heavy load of decoding and video memory uploading mentioned above has been silently completed in the background, the main rendering engine directly hits the GPU texture handle in the "Ready" state, instantly completes shader sampling, and achieves truly seamless, zero-perceptible tile display.

[0063] To demonstrate the specific technical effectiveness of this invention in ultra-large-scale image browsing scenarios, the method and its underlying hardware and software co-architecture are particularly applicable (but not limited to) ultra-high-resolution images with the following physical specifications: WSI panoramic slice total resolution: 10000×10000 pixels to 200000×200000 pixels (i.e., hundreds of megapixels to hundreds of gigapixels).

[0064] Image Pyramid model hierarchy: contains 8 to 16 resolution scaling levels (Level 0 is the highest resolution base layer).

[0065] Image tile cutting specifications: The standard size of a single tile is preferably 256×256 pixels or 512×512 pixels. After being decompressed into RGBA raw pixel data, each tile occupies approximately 256KB (for 256-pixel specification) or 1MB (for 512-pixel specification) of contiguous physical space in video memory.

[0066] Client hardware environment: The experimental benchmark of this invention runs on a general commercial PC or medical workstation environment, equipped with a conventional quad-core or higher CPU, a general SSD solid-state drive, and a dedicated graphics card or integrated graphics card with a video memory (VRAM) capacity between 2GB and 8GB.

[0067] To objectively verify the significant advancements of the proposed "Pre-rendering and Silent Memory Scheduling Method Based on Viewport Motion Vector Prediction" compared to existing technologies, a standardized "panning and zooming" stress test was conducted on a pathological WSI slide with dimensions of 100,000×100,000 pixels (approximately 5GB file size) and a tile size of 512×512 pixels under the aforementioned limited hardware environment (test machine: Intel i5 quad-core processor, 16GB RAM, integrated graphics card with 2GB shared video memory, 1080P monitor).

[0068] The test was divided into three groups: Control group A (traditional on-demand loading): the traditional "lazy loading" strategy (i.e., I / O and decoding only begin after the viewport is reached).

[0069] Control group B (conventional fixed-direction preloading): only conventional fixed-weight (w1=0.5, w2=0.5) geometric direction preloading is used, and it does not have underlying silent memory scheduling (i.e., the main thread glTexImage2D is used for blocking upload) and reverse elimination strategy (pure time LRU).

[0070] Experimental Group C (Invention): The invention adopts the complete scheme of the invention, which includes "dynamic time window Δt", "dynamic weights w1, w2 (non-linear Sigmoid mapping)", "multi-threaded PBO silent memory upload" and "predictive reverse memory replacement (PredictiveLRU)".

[0071] The following data was collected during a 60-second high-frequency random drag test (average sliding speed v > 3000px / s): Table 1 below: Table 1

[0072] As shown in the table above, the present invention achieved a synergistic effect that was unexpected in control group B: 1. Complete decoupling of I / O and rendering, locking in 60fps: Although control group B could partially predict direction, due to the lack of a low-level PBO silent upload mechanism, the main thread still had to bear the heavy workload of moving video memory, resulting in a severely damaged frame rate (41fps) and frequent frame drops (82 times / minute). This invention, through complete low-level decoupling, maintains a stable frame rate of 59.8fps, effectively eliminating UI stuttering caused by video memory I / O.

[0073] 2. Low Memory Overhead with High Prediction Hit Rate (Out of Memory Prevention): In control group B, fixed geometry prediction caused peak memory usage to spike to 1950MB (approaching the 2GB physical limit) during rapid dragging, with up to 42% of bandwidth wasted on tiles in the wrong direction (memory jitter). In contrast, the dynamic weight allocation mechanism proposed in this invention, combined with "reverse-direction PredictiveLRU de-weighting and elimination," allows the system to maintain a high cache hit rate of 96.5% while using only 1120MB of memory. This demonstrates that this invention effectively solves the performance bottleneck of rapid roaming of ultra-large-scale images without relying on massive memory hardware, through pure algorithmic and scheduling innovation.

[0074] Example 2 Based on the same concept, this application also proposes an image tile rendering and video memory scheduling system, including: The input monitoring and feature extraction module is used to acquire viewport coordinate data at high frequency and calculate motion vector data including movement speed and acceleration; The trajectory and viewport prediction module is used to dynamically adjust the prediction time window based on movement speed and acceleration, and output the prediction viewport bounding box using the trajectory prediction model; The priority scheduling module is used to determine the tiles to be loaded that intersect with the predicted viewport bounding box, and calculate the priority score of each tile by combining dynamic adaptive weights associated with the movement speed to generate a task queue. The background asynchronous processing and video memory management module is used to perform image decoding and silent video memory upload based on pixel buffer objects in the background thread according to the queue, and to perform predictive video memory eviction on resident tiles in combination with the motion direction. The main rendering engine module is used to directly call the textures in the ready state in the video memory for zero-latency rendering when the viewport actually arrives.

[0075] Example 3 This embodiment also provides a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including the image tile rendering and video memory scheduling method according to Embodiment 1.

[0076] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0077] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0078] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 4 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0079] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image tile rendering and video memory scheduling method, characterized in that, Includes the following steps: The system acquires user interaction input and frequently collects viewport coordinate data, thereby calculating viewport motion vector data, which includes at least movement speed and acceleration. The projection of the acceleration onto the direction of the movement velocity is extracted to calculate the tangential acceleration, which is used to characterize the user's movement intention; Based on the positive / negative state and magnitude of the tangential acceleration, the motion intention is divided into an acceleration phase, a constant speed fine-tuning phase, and an emergency stop phase. For different phases, a piecewise mapping function is used to dynamically adjust the prediction time window. Specifically, the prediction time window is extended during the acceleration phase to reserve tiles in advance, while a nonlinear decay function is applied during the emergency stop phase to shorten the prediction time window by a sudden drop, thereby truncating the pre-loading task. A preset trajectory prediction model is used to determine the prediction viewport bounding box within the prediction time window. The trajectory prediction model is a lightweight Kalman filter model, and its state vector contains position, velocity, and acceleration information. When an emergency stop action is detected to generate a large reverse jerk, the acceleration variance term corresponding to the process noise covariance matrix in the model is dynamically increased, forcing the model to converge to the current stationary observation value to avoid erroneous predictions caused by inertia. In the image pyramid hierarchical network, a set of tiles to be loaded that intersect with the predicted viewport bounding box is determined. A first weight is assigned to the distance from the tile center to the viewport center, and a second weight is assigned to the angle between the tile position and the direction of movement. The sum of the first weight and the second weight is a constant. The speed of movement is monitored in real time, and dynamic adaptive adjustment is performed: when the speed is greater than a high-speed threshold, the second weight is increased and the first weight is decreased to prioritize loading tiles directly in front of the movement; when the speed is less than a low-speed threshold, the first weight is increased and the second weight is decreased to prioritize loading tiles around the viewport center. To prevent tiles at the viewport center from being mistakenly eliminated due to excessively low weights, the values ​​of the first weight and the second weight are strictly limited to a preset range greater than zero and less than one. Combining the dynamic adaptive weights related to the movement speed, the priority score of each tile in the set of tiles to be loaded is calculated. A task queue is generated based on the priority scores. Tiles are decoded in a background asynchronous thread, and an auxiliary shared context that communicates with the main rendering thread is created in this thread. Using a pixel buffer object and a direct memory access controller, the decoded pixel data is silently and non-blockingly uploaded to the graphics processor's video memory (VRAM) via a hardware asynchronous stream to generate textures. After the transmission is complete, the corresponding texture state is marked as ready only through the auxiliary shared context, and drawing instructions that trigger screen redraws are prohibited during this process. Simultaneously, predictive eviction is performed on resident tiles in the VRAM based on the current motion direction. This predictive eviction process includes: real-time monitoring of VRAM usage; when the VRAM usage reaches a dynamically set threshold... When the water level threshold is reached, an elimination mechanism is triggered, and the doubly linked list of maintained resident tiles is traversed. The cosine of the direction angle between the position vector of the resident tile in the video memory and the current viewport motion direction vector is calculated. The orientation region of each resident tile is defined based on the cosine of the direction angle, and its survival weight is recalculated. When the resident tile is in the far-off region directly behind the movement, its survival weight is penalized and downgraded. When the resident tile is in the reserved region directly in front of the movement, downgrading is prohibited even if the tile is at the end of the linked list, in order to protect the image cache on the path that the user is about to reach. The tile with the lowest survival weight is moved to the end of the linked list. The main rendering thread releases the video memory handle of the tile at the end of the linked list during the vertical synchronization interval of the next frame. When the viewport moves to the corresponding area, the main rendering thread directly extracts the ready tile textures from the video memory to complete the rendering.

2. The image tile rendering and video memory scheduling method as described in claim 1, characterized in that, The expulsion water level threshold is dynamically calculated and determined based on the total physical video memory of the graphics processor after the initialization phase is obtained.

3. A system for implementing the image tile rendering and video memory scheduling method according to claim 1 or 2, characterized in that, include: The input monitoring and feature extraction module is used to acquire viewport coordinate data at high frequency and calculate motion vector data including movement speed and acceleration; The trajectory and viewport prediction module is used to dynamically adjust the prediction time window based on the moving speed and the acceleration, and output the prediction viewport bounding box using the trajectory prediction model; The priority scheduling module is used to determine the tiles to be loaded that intersect with the predicted viewport bounding box, and calculate the priority score of each tile by combining the dynamic adaptive weight associated with the movement speed to generate a task queue. The background asynchronous processing and video memory management module is used to perform image decoding and silent video memory upload based on pixel buffer objects in the background thread according to the queue, and to perform predictive video memory eviction on resident tiles in combination with the motion direction. The main rendering engine module is used to directly call the textures in the ready state in the video memory for zero-latency rendering when the viewport actually arrives.

4. A readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program including program code for controlling a computer execution process, the process including the image tile rendering and video memory scheduling method according to any one of claims 1 or 2.

Images