An image rendering method, device and storage medium

Abstract

An image rendering method, device and storage medium are provided. A client requests a coordinate set corresponding to an image block visible in a current viewport range from a server through a single network connection, without waiting for the entire picture to be downloaded. The first visible block is decoded and rendered immediately, greatly shortening the first screen time. During decoding, only the image block data in the current viewport is decoded according to the format identifier in the extracted data frame, thereby improving the efficiency of image rendering. In the case of on-demand loading, the amount of invalid data transmission approaches zero, significantly reducing network bandwidth consumption. In addition, the server and the client use a single long connection to transmit frame data streams, greatly reducing protocol overhead, improving first screen loading speed and transmission stability in weak networks, and achieving efficient transmission with extremely low protocol overhead.

Description

Technical Field

[0001] This article relates to the field of image processing technology, and in particular to an image rendering method, device and storage medium. Background Technology

[0002] With the rapid development of web front-end technologies, browsers are no longer merely windows for displaying information, but have gradually evolved into core platforms supporting complex image processing, massive data visualization, and high-precision geographic information systems. In application scenarios such as medical image diagnosis, satellite remote sensing maps, ultra-large-scale engineering drawings, and financial data dashboards, users need to render ultra-large images with resolutions of tens of thousands or even hundreds of thousands of pixels on the browser side. However, traditional web image loading and rendering mechanisms face severe performance bottlenecks and user experience challenges when dealing with such "giant images."

[0003] Traditional Canvas rendering requires waiting for all complete image data to be downloaded before decoding and drawing can begin. This method has obvious drawbacks. When a user focuses on a local area of ​​the image, the data of all invisible areas is still being downloaded. This not only causes a serious waste of network bandwidth, but also results in an excessively long time from when the user initiates a request to when they see the first contentful paint (FCP). Especially when network conditions are poor, a long period of blank screen seriously affects the user experience.

[0004] Therefore, there is an urgent need for an image rendering method to solve the problem of loading large-scale graphs on the current web. Summary of the Invention

[0005] To address the aforementioned technical problems, this paper provides the following technical solutions: Firstly, an image rendering method is provided, the method comprising: Obtain the viewport range of the current display area, determine the coordinate set of one or more image blocks visible within the current viewport based on the viewport range, and send a data request containing the coordinate set to the server through the single network connection; The server receives a frame data stream, which is obtained by the server encapsulating the image blocks corresponding to the coordinate set in response to the data request. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. Each data frame is extracted sequentially from the frame data stream. Based on the format identifier in the extracted data frame, the image block data in the data frame is decoded, and the decoded image block is rendered onto the display area according to the coordinate information.

[0006] According to an image rendering method provided in this application, the client maintains a cache pool, and after decoding the image block data in the data frame to obtain the decoded image blocks, the method further includes: Using the coordinate information of the image block as the key and the decoded image block data as the value, the decoded image block is stored in the cache pool; When the number of image blocks stored in the cache pool reaches the preset capacity limit, the image block data that has been accessed the least within a set time period is removed from the cache pool.

[0007] According to an image rendering method provided in this application, the method further includes: When the viewport range of the display area changes, the image block data that is no longer visible within the viewport range after the change is removed from the cache pool.

[0008] According to an image rendering method provided in this application, the step of rendering the decoded image block onto the display area based on coordinate information includes: During rendering, the coordinate set of image blocks within the viewport range of the current display area is scanned, and the cache pool is checked to see if the decoded image block corresponding to the coordinate set exists. If there are missing coordinates of decoded image blocks that are not in the cache pool, then the missing coordinates are added to the missing queue for missing compensation. If the cache pool contains all the decoded image blocks corresponding to the coordinate set, then all the decoded image blocks are rendered onto the display area according to the coordinate information.

[0009] According to an image rendering method provided in this application, the missing data compensation includes: When an image block in the missing queue meets the compensation triggering condition, a compensation request is initiated to the server through an independent network connection. The compensation request is used to obtain the missing image block data. After receiving the missing image block data returned by the server, it is stored in the cache pool and rendered to the display area.

[0010] According to an image rendering method provided in this application, the compensation triggering condition includes: The expected arrival time of the image block corresponding to the missing coordinates in the frame data stream transmitted on the single network connection exceeds a preset time threshold; or The sequence number of the data packet in the frame data stream corresponding to the missing coordinates of the image block is less than the sequence number of the currently received maximum data packet.

[0011] According to an image rendering method provided in this application, the step of initiating a compensation request to the server via an independent network connection includes: A compensation request is constructed using the Hypertext Transfer Protocol and concurrently executed with the frame data stream transmission on the single network connection, the compensation request containing the missing coordinates; Send the compensation request to obtain the missing image patch data.

[0012] According to an image rendering method provided in this application, the step of initiating a compensation request to the server via an independent network connection includes: The client obtains an index file pre-generated by the server, which records the byte offset index of each image block in the complete image file; The index file is queried based on the missing coordinates, and the compensation request containing the byte range is constructed based on the corresponding byte offset index. Send the compensation request to obtain the missing image patch data.

[0013] According to an image rendering method provided in this application, constructing the compensation request containing a byte range based on the corresponding byte offset index includes: When there are multiple target image blocks in the missing queue, the target image blocks are image blocks that are spatially adjacent and physically continuously distributed in their storage files. The byte ranges corresponding to the multiple target image blocks are merged into a continuous byte range. Construct a compensation request that covers the merged byte range.

[0014] According to an image rendering method provided in this application, the method further includes: After receiving the missing image block data returned after the missing compensation process, calculate the hash value of the missing image block data and compare it with the hash check value of the corresponding data frame header in the original frame data stream; If they match, the missing image patch data is stored in the cache pool and rendered. If there is a discrepancy, a new compensation request will be initiated or the missing coordinates will be added back to the missing coordinate queue.

[0015] According to an image rendering method provided in this application, during the process of storing the missing image block data into the cache pool and rendering, the missing image block has the highest rendering priority, and the missing image block is decoded and drawn to the corresponding coordinate position on the display area.

[0016] According to an image rendering method provided in this application, the frame header of the data frame further includes a sharpness level field, wherein the sharpness level identifier is used to indicate the data precision level of the image block, and the step of rendering the decoded image block onto the display area according to the coordinate information includes: Monitor the status of user interactions; Request image blocks of different resolution levels based on the interactive operation status; When a user is detected to be performing a drag or zoom operation, an image block carrying a first resolution level identifier is requested from the server; when a user operation stops for more than a preset time threshold, an image block carrying a second resolution level identifier is requested from the server, wherein the image block data size corresponding to the first resolution level is smaller than the image block data size corresponding to the second resolution level.

[0017] According to an image rendering method provided in this application, the monitoring of the user's interactive operation state includes: Listen for scroll events or mouse wheel events; When the frequency of event triggering exceeds the frequency threshold within the preset time window, it is determined that the user is performing a drag or zoom operation. If the event is not detected within the preset time threshold, it is determined that the user operation has stopped.

[0018] According to an image rendering method provided in this application, after requesting image patches of different resolution levels based on the interactive operation state, the method further includes: When an image block of the first resolution level is received, the received image block is used as a placeholder image and drawn at the corresponding coordinate position; When a second-resolution image block is received at the same coordinate position, the rendered first-resolution image block is replaced with the second-resolution image block in the animation frame callback.

[0019] Secondly, an image rendering method is provided for application on a server, the method comprising: In response to a data request sent by a client, the data request includes the coordinate set, which is the set of coordinates of one or more image blocks visible within the current viewport determined by the client based on the viewport range of the current display area; the data request is sent by the client to the server through the single network connection. The image blocks corresponding to the coordinate set are encapsulated into a frame data stream. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. Send to the client via the single network connection.

[0020] Thirdly, an image rendering method is provided, applied to a system including a client and a server, the method comprising: The control client is used to obtain the viewport range of the current display area, determine the coordinate set of one or more image blocks visible within the current viewport based on the viewport range, and send a data request containing the coordinate set to the server through a single network connection; The control server is used to encapsulate the image blocks corresponding to the coordinate set into a frame data stream after responding to the data request. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. The client is controlled to receive the frame data stream sent by the server. The client is controlled to extract each data frame sequentially from the frame data stream, decode the image block data in the extracted data frame according to the format identifier, and render the decoded image block onto the display area according to the coordinate information.

[0021] Fourthly, an apparatus is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any one of the image rendering methods described in the first, second, or third aspects above.

[0022] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the image rendering method described in the first, second, or third aspects above.

[0023] This paper provides an image rendering method, device, and storage medium, which have the following beneficial effects: The client obtains the viewport range of the current display area, determines the coordinate set of one or more visible image blocks within the viewport based on the viewport range, and sends a data request containing the coordinate set to the server through the single network connection. In other words, the client only requests the image blocks visible within the current viewport range, without waiting for the entire image to download; decoding and rendering can begin immediately upon the arrival of the first visible block, significantly shortening the first-screen loading time. During decoding, based on the format identifier in the extracted data frame, only the image block data within the current viewport is decoded, thereby improving image rendering efficiency. Furthermore, in the case of on-demand loading, the amount of invalid data transmission approaches zero, significantly reducing network bandwidth consumption. In addition, the server and client utilize a single long-lived connection for frame data stream transmission, greatly reducing protocol overhead, improving first-screen loading speed and transmission stability under weak network conditions, achieving highly efficient transmission with extremely low protocol overhead.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this document. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings.

[0026] Figure 1 This is a flowchart illustrating an image rendering method provided in this article; Figure 2 This is another flowchart illustrating one of the image rendering methods presented in this article; Figure 3 This is a flowchart illustrating the missing compensation logic in an image rendering method provided in this article; Figure 4 This is another flowchart illustrating the missing compensation logic in an image rendering method presented in this article; Figure 5 This is a schematic diagram of a computer-readable storage medium provided in this disclosure; Figure 6 This is a schematic diagram of the structure of a computing device provided in this disclosure. Detailed Implementation

[0027] This document describes several technical solutions with different concepts. Each concept has one or more embodiments, and these different concepts can be combined to form more embodiments. Those skilled in the art, after reading this document, can combine these different concepts to obtain new technical solutions, and these new technical solutions should also fall within the scope of this document.

[0028] The technical solutions of these different concepts will be introduced in turn below. Some concepts may appear in multiple technical solutions of different concepts. For these concepts, this article will explain them when they first appear and will not repeat them in the following text.

[0029] To address the aforementioned technical issues, this paper provides an image rendering system. The image rendering system includes a server and a client.

[0030] On the server side, the image to be rendered is divided into multiple image blocks, each image block is compressed and encoded separately, and a corresponding format identifier is generated for each image block. The format identifier is used to indicate the compression encoding method used for the image block. Each image block, its corresponding format identifier, and the coordinate information of the image block in the image to be rendered are encapsulated into a data frame. The transport layer utilizes a single network connection to perform real-time compression and transmission of frame data streams. Specifically, the server sequentially concatenates multiple data frames into a frame data stream and sends it to the client through a single network connection.

[0031] The client is configured to receive the frame data stream through the single network connection, extract each data frame sequentially from the frame data stream, call the decoder corresponding to the format identifier in the extracted data frame to decode the image block data in the data frame according to the format identifier, and obtain the decoded image block; and render the decoded image block onto the display area according to the coordinate information.

[0032] This embodiment aims to encapsulate massive image slices into a continuous byte stream by constructing a custom binary stream protocol (HIS, Hyper Tile Stream), transmitting it via a single long connection, and combining streaming GZIP / Deflate compression to achieve highly efficient transmission with extremely low protocol overhead. Simultaneously, a compression format identifier is introduced into the binary frame header, allowing the server to dynamically select the encoding method based on the pixel block content and automatically route to the corresponding decoder during client-side streaming parsing. This enables hybrid encoding transmission, thereby achieving differentiated compression for different image content within the same data stream, balancing image quality and bandwidth.

[0033] In other words, this application uses a dynamic format routing binary stream protocol and hybrid encoding to enable the same data stream to adaptively adopt the optimal encoding according to the content characteristics of different image regions. This solves the contradiction between image quality and bandwidth caused by the rigidity of single file format in traditional solutions, as well as the handshake overhead and header redundancy problems caused by a large number of independent HTTP requests.

[0034] This article also provides an image rendering method. The following embodiments will be used to illustrate the image rendering method provided in this article in conjunction with the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating an image rendering method according to an exemplary embodiment. The image rendering method is applied to a client.

[0036] The method includes steps 101 and 103: In step 201, the viewport range of the current display area is obtained, the coordinate set of one or more image blocks visible within the current viewport is determined based on the viewport range, and a data request containing the coordinate set is sent to the server through the single network connection.

[0037] To address bandwidth waste, memory overflow, and slow initial screen loading, some feasible implementations require the client to first determine the extent of the current display area within the original image coordinate system. Since the user may have already performed translation / drag and zoom operations on the image, the client maintains a view transformation matrix or a simple offset and scaling ratio. By reading the current transformation parameters and combining them with the width and height of the display area itself, the client can calculate the currently visible rectangular area in the original image, i.e., the viewport. For each pixel position within the viewport, its corresponding image patch index is determined, ultimately resulting in a unique set of image patch coordinates visible within the current viewport. The client then sends an explicit data request using the single network connection previously established with the server.

[0038] After receiving the on-demand request from the client, the server parses out the coordinate set. The server has either stored or can generate the image patch data corresponding to these coordinates internally. The server only encapsulates the image patches corresponding to the coordinate set specified in the request into data frames, concatenates these frames sequentially into a frame data stream, and then sends it back to the client through the same single network connection.

[0039] This embodiment improves the rendering experience, and the client can also include an outer ring of blocks at the edge of the viewport as preload.

[0040] This embodiment differs from traditional transmission modes. Based on the currently visible display area and zoom level, it calculates the index of visible pixel blocks and only requests, decodes, and draws image blocks within the visible area. Therefore, the data stream received by the client corresponds entirely to the data required by the current viewport, with invalid data transmission approaching zero. This reduces bandwidth consumption and significantly decreases unnecessary data transmission and decoding overhead, thereby reducing the amount of data required for the first screen and the transmission time, achieving WYSIWYG rendering within seconds.

[0041] In step 202, a frame data stream sent by the server is received. The frame data stream is obtained by the server encapsulating the image blocks corresponding to the coordinate set after responding to the data request. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block.

[0042] The client receives frame data streams through a single network connection. These data frame streams are generated by the server in response to the data request. Specifically, the server divides the image to be rendered into multiple image blocks, compresses and encodes each image block separately, and generates a corresponding format identifier for each image block. The format identifier indicates the compression and encoding method used for that image block.

[0043] Reference Figure 2 , Figure 2 This is another flowchart illustrating an image rendering method according to an exemplary embodiment. The server first acquires the original large-size image to be rendered, such as a satellite image or medical image. The image to be rendered is divided into N×M fixed-size pixel blocks (tiles, such as 256x256 pixels) according to a preset block size. For each image block, the server encodes it, converting the original pixel data of the image block into encoded data of a specific format to reduce its size.

[0044] As an example, the compression encoding method includes at least two of the following: lossless encoding formats and lossy encoding formats. The lossless encoding format includes raw data encoding such as Raw, which retains all details. The lossy encoding formats include, but are not limited to, JPEG encoding, WebP encoding, etc., which have high compression ratios but may lose some details.

[0045] To address the issues of image quality degradation or bandwidth waste caused by a single encoding format, differentiated compression for different image content is implemented within the same data stream to balance image quality and file size. In some feasible implementations, generating a corresponding format identifier for each image block includes: the server selecting a corresponding compression encoding method based on the image content complexity of each image block, and generating a corresponding format identifier based on the compression encoding method.

[0046] In this embodiment, lossless coding or lossy coding with low compression ratio is selected for high-complexity regions, and lossy coding with high compression ratio is selected for low-complexity regions.

[0047] Image content complexity can be measured based on information entropy, spatial frequency, or edge density metrics. For example, the probability distribution entropy of pixel grayscale values ​​within an image block can be calculated. A higher information entropy value indicates richer pixel variations, such as in photographs or natural textures. A lower information entropy value indicates a more uniform pixel distribution or the presence of numerous repetitive patterns, such as in text, lines, or large color blocks. Therefore, the server selects the corresponding compression encoding method based on the relationship between the entropy value of each image block and a preset threshold, and generates a corresponding format identifier based on the selected compression encoding method. Specifically, when the entropy value exceeds the preset threshold, lossless encoding or lossy encoding with a compression ratio lower than the preset compression ratio threshold is selected; when the entropy value does not exceed the preset threshold, lossy encoding with a compression ratio higher than or equal to the preset compression ratio threshold is selected.

[0048] For example, if an image block contains a large amount of high-frequency information such as text, lines, or edges, lossless compression such as Raw should be selected to ensure clarity.

[0049] For example, if the image block contains low-frequency information such as natural textures, gradients, or photo areas, lossy compression such as JPEG or WebP can be selected to reduce the file size.

[0050] After determining the encoding method, the server calls the corresponding encoder to actually compress the image blocks. After compression, the server generates a corresponding format identifier. This format identifier is a one-byte flag value, such as 1 for lossless Raw, 2 for JPEG, and 3 for WebP. This format identifier is written into the fixed frame header of the data frame. During subsequent parsing, the client does not need to know this in advance; it only needs to read the format identifier to determine which decoder to use, thus automatically switching the decoding logic to decode the image blocks.

[0051] In some cases, the coordinates of the image patch within the original image to be rendered are also recorded, including row and column numbers and pixel offsets. This facilitates subsequent extraction of the image patch data.

[0052] This embodiment introduces a compression format flag (FormatFlag) into the data frame, allowing the server to dynamically select the encoding method based on the pixel block content and automatically route to the corresponding decoder during client-side streaming parsing, thus achieving hybrid encoding transmission. The client switches decoders indiscriminately based on the format flag in the frame header, thereby achieving an optimal balance between compression ratio, image quality, and transmission efficiency within the same file.

[0053] In response to the data request, the server encapsulates the image block corresponding to the coordinate set, its corresponding format identifier, and the coordinate information of the image block in the image to be rendered into a data frame. Multiple data frames are sequentially concatenated into a frame data stream and sent to the client through a single network connection.

[0054] In this embodiment, each image block is packaged into an independent binary data frame, which consists of a fixed-length frame header and a variable-length data payload. For example, each frame contains an independent 16-byte header and payload, which supports dynamic frame size control (4KB-80KB) to avoid large packet blocking and ensure smooth rendering in weak network conditions.

[0055] As an example, the frame header at least includes a start-of-frame identifier (also known as the magic number), image patch coordinates, a sharpness rating field, a format identifier field, a payload length field, and a hash checksum field, supporting the mixed transmission of pixel patches of different compression formats (such as lossless Raw and lossy JPEG) in a single continuous stream. The data payload is the compressed encoded data of that image patch, and the start-of-frame identifier is used by the client to quickly locate the beginning of a data frame from a jumbled byte stream. To facilitate easy differentiation and parsing by the client, the data frame is designed as follows, as shown in the table below, which is an example of the data frame structure.

[0056] The header identifier (magic number) is a 4-byte value, 0x48545331 (corresponding to the ASCII character "HTS1"), used to mark relevant information about the current stream and tell the parser that the current data is not garbled.

[0057] The location information Tile_x and Tile_y, each two bytes (2 bytes unit16), represent the row and column of the current shard. After receiving the data, the client needs to verify whether x and y are within the visible window. If they are, the data is parsed immediately; otherwise, it is temporarily cached.

[0058] Image information (level) (2 bytes, unit16) defines the image sharpness level (Level) with values ​​of 1, 2, and 3. A value of 1 indicates the original image is being requested, 2 indicates half-sharpness, and 3 indicates quarter-sharpness. When the user interacts frequently, using lower-sharpness data allows for faster response. When the user stops interacting, the original image is then requested for replacement. This addresses image loading speed issues to some extent and improves the user experience.

[0059] The decoder identifier (format) (1 byte) is defined as 1, 2, 3. For example: 1: (raw): no compression loss; 2: (JPG): calls the JPG decoder; 3: ...

[0060] Compression flags (1 byte) indicate whether the current data has been compressed and which compression algorithm was used.

[0061] PayloadLen: Defines the length of the current frame data and plays a major role in data navigation.

[0062] To ensure fast image display, image size needs to be limited. The ideal size is 20k-40k, with a maximum of 80k. Larger images can cause buffering issues (e.g., frequent image viewing and scrolling, or the inability to load subsequent data).

[0063] The verification (hash) (4 bytes) is used to verify whether the current image is correct.

[0064] Core content analysis: Binary streams are just 0 / 1 byte streams. Browsers cannot directly render binary streams because they cannot know what they represent.

[0065] Image decoder: Each format involves compressing the binary stream using special data logic calculations. Ultimately, this results in data that a browser can directly parse.

[0066] A custom HIS (HyperTile Stream, binary stream protocol) protocol is built on the server side. HIS is a continuous byte stream, a continuous binary data stream formed by sequentially concatenating multiple data frames, i.e., a frame data stream. To further compress the transmission volume, all data frames are subjected to streaming GZIP or Deflate compression, that is, a standard GZIP or Deflate generic compression layer is wrapped around the binary frame data stream. This structure eliminates the overhead of HTTP headers for each slice, merges hundreds of requests into a single handshake, and significantly reduces protocol latency.

[0067] At the transport layer, a persistent, single network connection is pre-established between the server and the client. In this embodiment, the single network connection includes, but is not limited to, HTTP Stream and WebSocket.

[0068] Once the connection is established, the server continuously transmits frame data streams to the client in real time as segmented binary byte streams through a single network connection.

[0069] Unlike traditional HTTP slicing methods that require establishing a separate TCP connection for each tile, this embodiment only requires a handshake and TLS negotiation once. All subsequent data frames are transmitted over the same connection, eliminating the handshake overhead and header redundancy caused by numerous independent HTTP requests, achieving highly efficient transmission with extremely low protocol overhead. Furthermore, the above protocol is universal, based on the standard HTTP Stream API and GZIP standard, requiring no plugins and compatible with all modern browsers.

[0070] In step 203, each data frame is extracted sequentially from the frame data stream. The image block data in the extracted data frame is decoded according to the format identifier in the data frame, and the decoded image block is rendered onto the display area according to the coordinate information.

[0071] To address the issues of traditional solutions where decoding large images on the main thread blocks UI rendering, causing page lag, and the long First Screen Time (FCP) due to the need to wait for the entire image to download before decoding and rendering, this embodiment utilizes the browser's native DecompressionStream API to construct a non-blocking pipeline, connecting network reception, streaming decompression, and protocol packet parsing to achieve a zero-wait mode where data is processed as soon as it arrives.

[0072] As an example, a multi-level streaming pipeline was constructed, consisting of a Network Stream (for reading), a Decompression Stream (for decompression), and a Custom Parser Stream (for parsing) sequentially connected. It utilizes the browser's underlying C++ implementation for GZIP decompression, rather than JavaScript-level computation, significantly reducing CPU usage.

[0073] Based on this, in some feasible implementations, the process of sequentially extracting each data frame from the frame data stream includes: performing streaming processing on the frame data stream through a network reading stream, a decompression processing stream, and a custom parsing stream connected in sequence; wherein, the decompression processing stream calls the browser's native streaming decompression interface to perform non-blocking decompression, and the custom parsing stream maintains an internal buffer, caches the decompressed byte stream, and divides it into independent data frames one by one according to the frame structure definition.

[0074] Continue to refer to Figure 2 When the client receives a binary frame data stream sent by the server through a single network connection, the network read stream obtains the raw byte stream from the single network connection and pushes it to the next level.

[0075] The decompression processing stream calls the browser's native streaming decompression interface to perform streaming decompression on the original byte stream, eliminating common transport layer compressions such as GZIP or Deflate, and restoring the original binary frame sequence, i.e., the decompressed byte stream. In this embodiment, the decompression processing stream performs decompression operations at a low level (usually implemented in C++) without blocking the JavaScript main thread. Whenever the network read stream produces a certain amount of compressed data, the decompression processing stream attempts to decompress it and pushes the decompressed byte data to the next level. If the compressed data arrives in multiple parts, the decompression processing stream maintains its internal state and automatically handles compressed blocks that span multiple segments.

[0076] It should be noted that the decompression processing stream is based on the browser's native API interface, which is used to decompress compressed data streams (such as GZIP) in real time without the need to introduce third-party libraries.

[0077] The custom parsing stream receives the decompressed byte stream, identifies and segments each independent data frame, and passes each complete data frame down, for example, into a decoding queue. Based on the format identifier in the extracted data frame, the decoder corresponding to the format identifier is called to decode the image block data in the data frame, and the decoded image block is obtained.

[0078] In some feasible implementations, to improve efficiency, an internal buffer is maintained at a custom parsing level. After the decompressed byte stream is cached, independent data frames are identified and segmented one by one according to the frame structure definition of the data frames. Specifically, the frame header of the data frame includes at least a frame start identifier and a payload length field. The step of extracting each data frame sequentially from the frame data stream includes: identifying the frame start identifier from the internal buffer in real time as byte data is written to the internal buffer; after the frame start identifier is identified, extracting the complete data frame from the internal buffer according to the payload length field, and retaining the remaining data in the internal buffer for further processing.

[0079] Each time the custom parsing stream obtains a data block from the decompression processing stream, it executes the following logic: Step a11: Append new bytes of data to the internal buffer each time they are received; Step a12: Scan the header of the internal buffer, attempting to match the start-of-frame identifier. If a match fails, slide forward byte by byte until the next start-of-frame identifier is found; Step a13: After a successful match, read the payload length field, such as coordinate information, format identifier, payload length, hash check value, etc. Step a14: Based on the payload length, check if there are enough bytes in the buffer to form a complete data frame. If not, wait for more data. Step a15: If it is a complete data frame, extract the complete data frame from the internal buffer, strip it, and send it to the decoding queue. At the same time, remove the extracted bytes from the internal buffer, and keep the remaining bytes in the internal buffer to wait for the next frame; Step a16: Repeat the above process until all data frames have been extracted.

[0080] This streaming segmentation process ensures that the client can begin decoding and rendering immediately after receiving the first complete frame, without waiting for all the data of the entire image, significantly reducing initial screen latency. For example, the first visible pixel block can be displayed on the screen within microseconds.

[0081] In some feasible implementations, the frame header further includes a hash check value. After extracting the complete data frame from the internal buffer based on the payload length field, the method further includes: calculating the hash value of the data payload in the complete data frame and comparing it with the hash check value; if they match, the complete data frame is sent to the decoding queue, or if they do not match, the complete data frame is discarded and the next frame start identifier is matched from the internal buffer.

[0082] When encapsulating each data frame, the server calculates the hash value of the data payload in that frame, i.e., the hash value of the encoded data after image patch compression, and stores this hash value in the hash checksum field of the frame header. The hash algorithm can be cyclic redundancy check, message digest algorithm, or secure hash algorithm. After the custom parsing stream extracts a complete data frame from the internal buffer based on the payload length field in the frame header, it does not immediately send the frame to the decoding queue, but instead performs the verification process.

[0083] Specifically, the frame header is read from the beginning of the extracted complete data frame, and the hash checksum is extracted based on the predefined field offset. For all bytes in the data payload immediately following the frame header in the same frame, the hash value is calculated using the same hash algorithm as the server, yielding the actual hash value. Then, the calculated actual hash value is compared bit-by-bit with the hash checksum extracted from the frame header. If the two hash values ​​match, it means that the data frame has not been damaged or tampered with during transmission, and the data is complete and reliable. The client sends the complete data frame to the decoding queue, waiting for the decoder to decode and finally render it. The frame is marked as valid, and subsequent processing proceeds normally. If the two hash values ​​do not match, it means that an error occurred in the data payload of the data frame during transmission, such as bit flips or byte misalignments caused by network packet loss. In this case, the data of the frame is unusable and cannot be sent to the decoding queue; otherwise, it may cause image display distortion or decoder crashes.

[0084] In some cases, erroneous data frames can be removed from the internal buffer or retrieved again from the server. For example, the client can retrieve the data frame again later through other independent compensation channels. This can be configured according to requirements to ensure that even if data corruption occurs, the client's parsing pipeline will not be blocked, and the user can still obtain the correct content through the compensation mechanism. The specific method is not specifically limited here.

[0085] Continue to refer to Figure 2After extracting the complete data frame, the corresponding decoder is called based on the format identifier in the extracted data frame to decode the image block data in the data frame, thus obtaining the decoded image block. Specifically, the client maintains multiple decoders locally. For each extracted data, the format identifier in the frame header is read, and the corresponding decoder is called according to the format identifier. For example, when the format identifier indicates lossless compression encoding, the lossless decoder is called; when the format identifier indicates lossy compression encoding, the lossy decoder is called. Afterward, the payload data is restored to the original pixel data to obtain the decoded image block.

[0086] For the decoding process, this embodiment constructs a two-layer decoding architecture. The first layer of decoding is general compression and decompression, used to eliminate transport layer redundancy; the second layer of decoding calls the corresponding image decoder according to the format identifier, used to restore the encoded data to pixels, realizing dynamic routing of the decoder. The first layer of decoding and the second layer of decoding are executed in series and independently of each other.

[0087] In other words, during the decoding process, GZIP or Deflate is responsible for removing protocol headers and statistical redundancy, while FORMAT routing is responsible for restoring specific encodings to pixels. The two are connected in series, do not interfere with each other, and complement each other, extracting the data frames that have arrived completely in real time, and immediately decoding and rendering the data frames without waiting for the entire frame data stream to be received.

[0088] Through the above embodiments, the browser's native DecompressionStream API is used to build a decompression pipeline to perform non-blocking, pipelined processing on the received frame data stream. Combined with a block-based independent decoding mechanism, the pipelined processing of receiving, decompressing, parsing, and drawing is realized, avoiding blocking the main drawing thread during the decoding process. Through non-blocking real-time rendering, smooth interaction under large data volumes is ensured.

[0089] In some feasible implementations, after decoding the image block data in the data frame to obtain the decoded image block, the method further includes: using the coordinate information of the image block as the key and the decoded image block data as the value, storing the decoded image block in the cache pool; when the number of image blocks stored in the cache pool reaches a preset capacity limit, removing the image block data that has been accessed the least within a set time from the cache pool.

[0090] The client maintains a cache pool. After obtaining the decoded image block by calling the decoder corresponding to the format identifier, it stores it in a front-end cache pool for subsequent reuse. For example, when the user pans the view and returns to a previously viewed area, there is no need to request and decode it again from the server.

[0091] As an example, this cache pool uses a key-value pair storage structure, storing the coordinate information of image patches as keys and the decoded image patch data as values. The key is a unique identifier for the image patch, composed of its coordinates in the original image and its sharpness level. The value is the decoded image patch data, which can be directly used for rendering without further decoding.

[0092] To control memory usage, fine-grained memory lifecycle management is implemented based on coordinate indexing. As an example, the cache pool has a preset capacity limit, such as caching a maximum of 256 image patches. When the number of image patches in the cache pool reaches this limit, some older data needs to be evicted.

[0093] As an example, the cache pool uses a Least Recently Used (LRU) strategy to manage storage space, which prioritizes evicting image blocks that have not been accessed for the longest time, including image blocks that have neither been read nor updated.

[0094] As an example, when the viewport range of the display area changes, image block data that is no longer visible within the viewport range after the change is preferentially removed from the cache pool.

[0095] This embodiment constructs an LRU cache pool with pixel block coordinates as the key value. Combined with Canvas viewport calculation, it only decodes and retains image blocks near the viewport and in the prediction area. Old blocks are automatically recycled or evicted by LRU, realizing accurate loading, on-demand decoding and real-time recycling of image blocks. It completely decouples the memory usage of large images from the total file size and supports rendering GB-level virtual canvases without crashing.

[0096] The decoded image block carries the original coordinate information. The client calculates its position in the display area based on these coordinates, and then calls the drawing API to draw the decoded image block onto the corresponding area of ​​the canvas.

[0097] In this embodiment, the display area refers to the screen area on the client device used to present image content. In one embodiment, the display area can be an HTML5 Canvas element, which provides a JavaScript drawing API capable of drawing decoded image blocks to specified coordinates. In other embodiments, the display area can also be a WebGL render buffer, an SVG area, or an image view control from a native mobile application. For ease of understanding, this document uses a Canvas as a specific example, but the scope of protection of this application is not limited thereto.

[0098] When drawing image blocks within the current viewport onto the Canvas, the client first searches for the image block with the corresponding coordinates in the cache pool. If the cache is hit, the data is used directly for drawing without re-decoding. However, in the process of streaming chunked transmission based on a custom binary frame structure, block loss or stream loss due to other circumstances may occur. Furthermore, the problem of long-term viewport loss due to waiting for timeout retransmission is also prominent when there is packet loss or network fluctuations.

[0099] To address this issue and achieve millisecond-level missing block detection and completion during network anomalies, in some implementations, rendering the decoded image blocks onto the display area based on coordinate information includes: during rendering, scanning the coordinate set of image blocks within the viewport range of the current display area, checking whether the cache pool contains the decoded image blocks corresponding to the coordinate set; if there are missing coordinates for decoded image blocks not in the cache pool, adding the missing coordinates to a missing queue for missing compensation; if the cache pool contains all the decoded image blocks corresponding to the coordinate set, rendering all the decoded image blocks onto the display area based on the coordinate information.

[0100] Figure 3 This is a flowchart of a block compensation logic in an image rendering method illustrated in this article according to an exemplary embodiment.

[0101] In this embodiment, a missing block compensation mechanism is established that combines viewport-driven detection with dual-channel parallel acquisition. Outside of the main compressed stream, missing specific coordinate blocks are quickly filled in using independent HTTP Range or Index requests.

[0102] The client listens for events related to scrolling, scaling, or window resizing of the display area. Combining this with the current transformation matrix (e.g., the Canvas's translation and scaling ratios), it calculates in real-time the original image area covered by the current viewport and maps this viewport area to a set of coordinates for the image blocks to be displayed. This scanning operation is typically performed in a high-frequency loop; for example, a `requestAnimationFrame` can be used to repeatedly scan the coordinate set within the current viewport to ensure the view remains synchronized with user actions. However, to avoid excessive CPU consumption, the scanning frequency can be throttled, for example, once per frame or every 60ms, or a full scan can be performed only after user interaction has ended.

[0103] For a given set of coordinates, the client iterates through each coordinate, querying the locally maintained LRU cache pool using that coordinate as the key. The cache pool stores decoded image patch data. A cache hit means the image patch data is available and can be used directly for drawing. A cache miss means the image patch has not yet been decoded, or although the data has been received, it has not been decoded, or it has never been requested. In this case, the client records the coordinate as a missing coordinate and adds it to a dedicated missing coordinate queue for compensation. The client retrieves the data from the server through on-demand requests or a compensation channel, decodes it, stores it in the cache pool, and then performs drawing.

[0104] For missing image compensation, in some feasible implementations, the missing image compensation includes: when the image blocks in the missing queue meet the compensation triggering condition, the client initiates a compensation request to the server through an independent network connection, the compensation request being used to obtain the missing image block data; the server responds to the compensation request and returns the corresponding image block data, the client receives the missing image block data returned by the server, stores it in the cache pool, and renders it to the display area.

[0105] As an example, the compensation triggering conditions include: the expected arrival time of the image block corresponding to the missing coordinates in the frame data stream transmitted on the single network connection exceeds a preset time threshold; or the sequence number of the data packet corresponding to the image block corresponding to the missing coordinates in the frame data stream is less than the sequence number of the currently received maximum data packet. For example, if the index in the loss queue is less than the currently requested flag value, it is considered that the data has been lost, and the compensation strategy is triggered; otherwise, it is considered that the data has not yet been requested, and a timer is started. If the data is not received within a preset time threshold (e.g., 0.8s), the compensation strategy is triggered.

[0106] Figure 4 This is another flowchart of the block compensation logic in an image rendering method illustrated in this article according to an exemplary embodiment.

[0107] Once the compensation strategy is triggered, it does not interfere with the reception of the compressed stream over the main single network connection. Instead, it immediately initiates an independent XMLHttpRequest or fetch request, calculates the physical byte offset of the image patch using the server-pre-generated .index file, and constructs a Range:bytes=start-end header for precise fetching. In other words, it initiates an independent compensation request over a separate network connection, so that after the server responds to the compensation request, it returns the image patch data corresponding to the missing coordinates over the independent network connection.

[0108] The following two methods can be used when initiating a compensation request.

[0109] Method 1, the step of initiating a compensation request to the server via an independent network connection, includes: The client uses the Hypertext Transfer Protocol to construct a compensation request that is concurrently executed with the frame data stream transmission on the single network connection. The compensation request includes the missing coordinates. The client then sends the compensation request to obtain the missing image patch data.

[0110] The client constructs a separate HTTP request that bypasses the main data stream connection, enabling the server to locate and return data for the missing image patch. This location can be achieved through tile interface requests based on a Uniform Resource Locator (URL) parameter pointing to a specific resource on the server, or through HTTP range requests based on byte offsets. For example, URL parameters might include the currently viewed image to be rendered, the column and row indices of the missing image patch, and the desired resolution level.

[0111] Method 2, wherein the compensation request is initiated to the server via an independent network connection, includes: the client obtaining a pre-generated index file from the server, the index file recording the byte offset index of each image block in the complete image file; querying the index file according to the missing coordinates, and constructing the compensation request containing the byte range according to the corresponding byte offset index; and sending the compensation request to obtain the missing image block data.

[0112] When preprocessing the image to be rendered, the server writes the encoded data of all image blocks sequentially into a complete binary file. The server maintains an `.index` file, recording the byte offset, length, coordinates, and resolution level of each image block within the larger file. An index record is generated for each image block, and all index records are arranged in coordinate order to form a separate index file. The client first quickly retrieves or caches the index, then directly initiates a `Range:bytes=Start-End` request. Specifically, when the client's missing data queue records one or more image block coordinates that need compensation, the client first queries the cached index file for the corresponding index record. After obtaining this record, the client receives two key values: the starting byte offset and the data length, which determine the byte range of data required for the image block. The client then initiates a compensation request directly over a separate HTTP connection, covering the entire byte range.

[0113] To improve compensation efficiency, in some feasible implementations, constructing the compensation request containing a byte range based on the corresponding byte offset index includes: when there are multiple target image blocks in the missing queue, the target image blocks are image blocks that are spatially adjacent and physically continuously distributed in their storage files, the client merges the byte ranges corresponding to the multiple target image blocks into a continuous byte interval, and constructs a compensation request covering the merged byte interval.

[0114] If there are multiple physically contiguous target image blocks in the missing queue, meaning the starting offset of the next block is exactly equal to the ending offset of the previous block plus 1, the client can merge these target image blocks into a larger byte range and use a compensation request to retrieve the data of multiple image blocks at once, thereby reducing the number of HTTP requests and further reducing the extra overhead of connection establishment and header transmission.

[0115] For example, if there are multiple adjacent target image blocks in the missing queue, such as (10,10), (10,11), (11,10), they can be merged into a single compensation request (if they are contiguous in physical storage), reducing the number of HTTP handshakes.

[0116] For merged compensation requests, the response data received by the client contains multiple contiguous blocks of data, which need to be segmented according to the length of each block recorded in the index. Subsequently, the client decodes the obtained encoded data into pixels and stores it in the LRU cache pool using the coordinate information as the key. Rendering is triggered immediately after storage, allowing the missing areas to be repaired. Because the compensation request runs concurrently with the main data stream, the entire process is transparent to the user.

[0117] To ensure data consistency, in some feasible implementations, the method further includes: after receiving the missing image block data returned after the missing compensation process, the client calculates the hash value of the missing image block data and compares it with the hash check value of the corresponding data frame header in the original frame data stream; if they match, the missing image block data is stored in the cache pool and rendered; if they do not match, a compensation request is re-initiated or the missing coordinates are re-added to the missing queue.

[0118] When missing image block data is received after the missing data compensation process, it is necessary to verify the consistency of the data to prevent errors in the image content. Specifically, after the missing image block data arrives after the missing data compensation process, its hash value is calculated. The hash value of the returned image block data content is compared with the hash value in the HIS protocol frame header. If they are the same, the content is considered correct; otherwise, the current block is put back into the missing data queue, and the missing data compensation operation is triggered again.

[0119] In this embodiment, Hash verification is used to ensure data consistency, ensure the absolute correctness of the repaired data, prevent erroneous images from being displayed, and achieve seamless repair.

[0120] In some feasible implementations, during the process of storing the missing image block data in the cache pool and rendering, the missing image block has the highest rendering priority. That is, when the missing image block data is received, the missing image block is immediately decoded and drawn to the corresponding coordinate position on the display area.

[0121] Since the compensation data has a high priority, it should be rendered immediately after being saved to the cache to prevent users from seeing missing blocks.

[0122] Through the above embodiments, the dual-channel missing packet repair architecture of mainstream connectivity and independent compensation establishes a dual-channel system based on regular transmission of the main compressed stream and on-demand compensation based on HTTP Range / independent interface. Combined with an active viewport missing detection algorithm, it achieves millisecond-level packet loss repair.

[0123] To ensure that compensation data does not conflict with or duplicate data in the main data stream, the client needs to maintain a status flag. For example, for an image patch with specific coordinates, if the client has already received the patch through the main data stream and stored it in the cache, it should be discarded after the response is returned, even if a compensation request has been sent. Therefore, before sending a compensation request, the client marks the coordinates as pending compensation in the missing data queue; when the compensation response returns, it checks again whether the patch already exists in the cache pool to prevent the main data stream from delivering it during this period; if it does not exist, it is stored.

[0124] Traditional rendering processes suffer from response lag and white screen issues during rapid interactions (such as drag / zoom) due to loading high-resolution images. To address this response lag in fast-interaction scenarios and optimize visual continuity during interaction, some feasible implementations include a resolution level field in the frame header of the data frame. This resolution level identifier indicates the data precision level of the image block. Rendering the decoded image block onto the display area based on the coordinate information includes: the client monitoring the user's interaction status; requesting image blocks of different resolution levels based on the interaction status; requesting image blocks carrying a first resolution level identifier from the server when dragging or zooming is detected; and requesting image blocks carrying a second resolution level identifier from the server when the user's operation stops for more than a preset time threshold. The data size of the image block corresponding to the first resolution level is less than the data size of the image block corresponding to the second resolution level.

[0125] Each image patch's data frame header contains a 1-byte sharpness level identifier. The server pre-generates multiple versions of the image patch at the same coordinates with different sharpness / resolution, each version corresponding to a level value. When sending a dataset request, the client can specify the desired sharpness level in the request parameters. The server returns the image patch data with the corresponding precision based on that level.

[0126] Specifically, an interactive state machine, such as Dragging / Idle, is introduced on the client side. The client identifies the interactive operation being performed by the user by listening to DOM events. Monitoring the user's interactive operation status includes: listening to scroll events or mouse wheel events; determining that the user is performing a drag or zoom operation when the event trigger frequency exceeds a frequency threshold within a preset time window; and determining that the user operation has stopped when no event is detected within the preset time threshold.

[0127] To avoid frequent state switching, clients typically employ debouncing or throttling techniques. For example, during dragging, a timer is reset each time an interactive event is triggered, and dragging is only considered to be ongoing if there are no new movement events for a continuous period of time; the state is switched immediately when dragging begins.

[0128] After detecting the user's interaction state, the client requests image tiles of different resolution levels based on the interaction state. For example, when the state changes from static to interactive (drag or zoom operation), the client immediately initiates a request for low-resolution images of all visible image tiles within the current viewport. Of course, tiles already requested in high resolution can be canceled or ignored. As another example, when the state changes from interactive to static (operation stops), the client waits for a preset time threshold (e.g., 300ms) and then initiates a request for high-resolution images of all visible image tiles within the current viewport. Simultaneously, to avoid repeatedly requesting already existing high-resolution tiles, the client checks the local cache and only requests image tiles for which high-resolution data has not yet been cached.

[0129] In this embodiment, the client dynamically requests and renders pixel blocks of different precisions based on the user's interaction status, achieving an optimized experience of fast response during interaction and high-definition rendering when stationary.

[0130] To optimize visual continuity during interaction and ensure operational coherence, in some feasible implementations, after requesting image blocks of different resolution levels according to the interaction operation state, the method further includes: when receiving an image block of the first resolution level, drawing the received image block as a placeholder image at the corresponding coordinate position; when receiving an image block of the second resolution level at the same coordinate position, replacing the rendered image block of the first resolution level with the image block of the second resolution level in the animation frame callback.

[0131] Upon receiving a low-resolution image block, the client uses it as a placeholder, immediately decodes it, and draws it at the corresponding coordinates on the Canvas. When the user stops interacting, the client receives a high-resolution image block at the corresponding coordinates. At this point, the low-resolution placeholder needs to be replaced with the content of the corresponding high-resolution image block. It's important to note that this replacement process is performed atomically at the same coordinates using `requestAnimationFrame`, completed within the same animation frame callback. This allows the user to seamlessly switch from the placeholder image to the final image, ensuring smooth operation and ultimately delivering high-definition image quality.

[0132] This paper provides an image rendering method, device, and storage medium, which have the following beneficial effects: The client obtains the viewport range of the current display area, determines the coordinate set of one or more visible image blocks within the viewport based on the viewport range, and sends a data request containing the coordinate set to the server through the single network connection. In other words, the client only requests the image blocks visible within the current viewport range, without waiting for the entire image to download; decoding and rendering can begin immediately upon the arrival of the first visible block, significantly shortening the first-screen loading time. During decoding, the corresponding decoder is called based on the format identifier to decode only the image block data within the current viewport, thereby improving image rendering efficiency. Furthermore, in the case of on-demand loading, the amount of invalid data transmission approaches zero, significantly reducing network bandwidth consumption. In addition, the server and client utilize a single long-lived connection for frame data stream transmission, greatly reducing protocol overhead, improving first-screen loading speed and transmission stability under weak network conditions, achieving highly efficient transmission with extremely low protocol overhead.

[0133] This article also provides an image rendering method applied to a server, the method comprising: In response to a data request sent by a client, the data request includes the coordinate set, which is the set of coordinates of one or more image blocks visible within the current viewport determined by the client based on the viewport range of the current display area; the data request is sent by the client to the server through the single network connection. The image blocks corresponding to the coordinate set are encapsulated into a frame data stream. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. Send to the client via the single network connection.

[0134] This article also provides an image rendering method applied to a system including a client and a server, the method comprising: The control client is used to obtain the viewport range of the current display area, determine the coordinate set of one or more image blocks visible within the current viewport based on the viewport range, and send a data request containing the coordinate set to the server through a single network connection; The control server is used to encapsulate the image blocks corresponding to the coordinate set into a frame data stream after responding to the data request. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. The client is controlled to receive the frame data stream sent by the server. The client is controlled to extract each data frame sequentially from the frame data stream, decode the image block data in the extracted data frame according to the format identifier, and render the decoded image block onto the display area according to the coordinate information.

[0135] The specific implementation of the image rendering method described above by the server or client is basically the same as that of the image rendering method described above, and the effect is also the same, so it will not be described again here.

[0136] Figure 5 This is a schematic diagram of a computer-readable storage medium 140 provided in this disclosure, on which a computer program is stored, which, when executed by a processor, implements the method of any embodiment of this disclosure.

[0137] This disclosure also provides a computing device, including a memory and a processor; the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the methods of any embodiment of this disclosure when executing the computer instructions.

[0138] Figure 6 This is a schematic diagram of the structure of a computing device provided in this disclosure, such as... Figure 6 As shown, the computing device 15 may include, but is not limited to: a processor 151, a memory 152, and a bus 153 connecting different system components (including the memory 152 and the processor 151).

[0139] The memory 152 stores computer instructions that can be executed by the processor 151, enabling the processor 151 to perform the training method of the aesthetic image generation model according to any embodiment of this disclosure. The memory 152 may include a random access memory unit (RAM) 1521, a cache memory unit (Cache) 1522, and / or a read-only memory unit (ROM) 1523. The memory 152 may also include a program tool 1525 having a set of program modules 1524, including but not limited to: an operating system, one or more application programs, other program modules, and program data. One or more combinations of these program modules may include an implementation of a network environment.

[0140] Bus 153 may include, for example, a data bus, an address bus, and a control bus. The computing device 15 can also communicate with external devices 155 via I / O interface 154, such as a keyboard or a Bluetooth device. The computing device 15 can also communicate with one or more networks via network adapter 156, such as a local area network (LAN), a wide area network (WAN), or a public network. As shown in the figure, the network adapter 156 can also communicate with other modules of the computing device 15 via bus 153.

[0141] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0142] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An image rendering method, characterized in that, Applied to a client, the method includes: Obtain the viewport range of the current display area, determine the coordinate set of one or more image blocks visible within the current viewport based on the viewport range, and send a data request containing the coordinate set to the server through a single network connection; The server receives a frame data stream, which is obtained by the server encapsulating the image blocks corresponding to the coordinate set in response to the data request. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. Each data frame is extracted sequentially from the frame data stream. Based on the format identifier in the extracted data frame, the image block data in the data frame is decoded, and the decoded image block is rendered onto the display area according to the coordinate information.

2. The image rendering method as described in claim 1, characterized in that, The client maintains a cache pool. After decoding the image block data in the data frame to obtain the decoded image blocks, the method further includes: Using the coordinate information of the image block as the key and the decoded image block data as the value, the decoded image block is stored in the cache pool; When the number of image blocks stored in the cache pool reaches the preset capacity limit, the image block data that has been accessed the least within a set time period is removed from the cache pool.

3. The image rendering method as described in claim 2, characterized in that, The method further includes: When the viewport range of the display area changes, the image block data that is no longer visible within the viewport range after the change is removed from the cache pool.

4. The image rendering method as described in claim 2, characterized in that, The step of rendering the decoded image blocks onto the display area based on coordinate information includes: During rendering, the coordinate set of image blocks within the viewport range of the current display area is scanned, and the cache pool is checked to see if the decoded image block corresponding to the coordinate set exists. If there are missing coordinates of decoded image blocks that are not in the cache pool, then the missing coordinates are added to the missing queue for missing compensation. If the cache pool contains all the decoded image blocks corresponding to the coordinate set, then all the decoded image blocks are rendered onto the display area according to the coordinate information.

5. The image rendering method as described in claim 4, characterized in that, The missing compensation includes: When an image block in the missing queue meets the compensation triggering condition, a compensation request is initiated to the server through an independent network connection. The compensation request is used to obtain the missing image block data. After receiving the missing image block data returned by the server, it is stored in the cache pool and rendered to the display area.

6. The image rendering method as described in claim 5, characterized in that, The compensation triggering conditions include: The expected arrival time of the image block corresponding to the missing coordinates in the frame data stream transmitted on the single network connection exceeds a preset time threshold; or The sequence number of the data packet in the frame data stream corresponding to the missing coordinates of the image block is less than the sequence number of the currently received maximum data packet.

7. The image rendering method as described in claim 5, characterized in that, The step of initiating a compensation request to the server via an independent network connection includes: A compensation request is constructed using the Hypertext Transfer Protocol and concurrently executed with the frame data stream transmission on the single network connection, the compensation request containing the missing coordinates; Send the compensation request to obtain the missing image patch data.

8. The image rendering method as described in claim 5, characterized in that, The step of initiating a compensation request to the server via an independent network connection includes: Obtain the index file pre-generated by the server, which records the byte offset index of each image block in the complete image file; The index file is queried based on the missing coordinates, and the compensation request containing the byte range is constructed based on the corresponding byte offset index. Send the compensation request to obtain the missing image patch data.

9. The image rendering method as described in claim 8, characterized in that, Constructing the compensation request containing a byte range based on the corresponding byte offset index includes: When there are multiple target image blocks in the missing queue, the target image blocks are image blocks that are spatially adjacent and physically continuously distributed in their storage files. The byte ranges corresponding to the multiple target image blocks are merged into a continuous byte range. Construct a compensation request that covers the merged byte range.

10. The image rendering method according to any one of claims 4 to 9, characterized in that, The method further includes: After receiving the missing image block data returned after the missing compensation process, the hash value of the missing image block data is calculated and compared with the hash check value of the corresponding data frame header in the original frame data stream; If they match, the missing image patch data is stored in the cache pool and rendered. If there is a discrepancy, a new compensation request will be initiated or the missing coordinates will be added back to the missing coordinate queue.

11. The image rendering method as described in claim 10, characterized in that, During the process of storing the missing image block data into the cache pool and rendering, the missing image block has the highest rendering priority. The missing image block is decoded and drawn to the corresponding coordinate position on the display area.

12. The image rendering method as described in claim 1, characterized in that, The frame header of the data frame also includes a sharpness level field, wherein the sharpness level identifier is used to indicate the data precision level of the image block, and the step of rendering the decoded image block onto the display area according to the coordinate information includes: Monitor the status of user interactions; Request image blocks of different resolution levels based on the interactive operation status; When a user is detected to be performing a drag or zoom operation, an image block carrying a first resolution level identifier is requested from the server; when a user operation stops for more than a preset time threshold, an image block carrying a second resolution level identifier is requested from the server, wherein the image block data size corresponding to the first resolution level is smaller than the image block data size corresponding to the second resolution level.

13. The image rendering method as described in claim 12, characterized in that, The monitoring of user interaction status includes: Listen for scroll events or mouse wheel events; When the frequency of event triggering exceeds the frequency threshold within the preset time window, it is determined that the user is performing a drag or zoom operation. If the event is not detected within the preset time threshold, it is determined that the user operation has stopped.

14. The image rendering method as described in claim 12, characterized in that, After requesting image patches of different resolution levels based on the interactive operation state, the method further includes: When an image block of the first resolution level is received, the received image block is used as a placeholder image and drawn at the corresponding coordinate position; When a second-resolution image block is received at the same coordinate position, the rendered first-resolution image block is replaced with the second-resolution image block in the animation frame callback.

15. An image rendering method, characterized in that, Applied to the server side, the method includes: In response to a data request sent by a client, the data request includes a set of coordinates, which is the set of coordinates of one or more image blocks visible within the current viewport determined by the client based on the viewport range of the current display area; the data request is sent by the client to the server through a single network connection. The image blocks corresponding to the coordinate set are encapsulated into a frame data stream. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. Send to the client via the single network connection.

16. An image rendering method, characterized in that, Applied to a system including a client and a server, the method includes: The control client is used to obtain the viewport range of the current display area, determine the coordinate set of one or more image blocks visible within the current viewport based on the viewport range, and send a data request containing the coordinate set to the server through a single network connection; The control server is used to encapsulate the image blocks corresponding to the coordinate set into a frame data stream after responding to the data request. Each data frame in the frame data stream contains image block data and its corresponding format identifier. The format identifier is used to indicate the compression encoding method used for the image block. The client is controlled to receive the frame data stream sent by the server. The client is controlled to extract each data frame sequentially from the frame data stream, decode the image block data in the extracted data frame according to the format identifier, and render the decoded image block onto the display area according to the coordinate information.

17. A computer device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, when the processor executes the computer program, it performs the steps of the image rendering method according to any one of claims 1 to 14; or, when the processor executes the computer program, it performs the steps of the image rendering method according to claim 15; or, when the processor executes the computer program, it performs the steps of the image rendering method according to claim 16.

18. A computer-readable storage medium, characterized in that, The computer program is stored thereon, and when the program is executed by a processor, it performs the steps of the image rendering method according to any one of claims 1 to 14; or, when the processor executes the computer program, it performs the steps of the image rendering method according to claim 15; or, when the processor executes the computer program, it performs the steps of the image rendering method according to claim 16.

Images