Image processing method and device, electronic equipment and computer program product

Abstract

The invention discloses an image processing method and device, electronic equipment and a computer program product. The method comprises the following steps: deploying a to-be-processed image frame acquired from a video stream on a first canvas; according to the frame rate of the video stream, the first canvas is divided into a plurality of first image blocks of a preset size, and the preset size is in negative correlation with the frame rate; according to the difference between each first image block and a second image block at a corresponding position on a second canvas, a dirty area is determined, a previous image frame of the to-be-processed image frame is deployed on the second canvas in advance, and the second canvas is divided into a plurality of second image blocks with preset sizes; the dirty area is determined at least according to the first image block whose difference with the second image block is greater than a preset difference threshold; redrawing the dirty area to obtain a redrawn image; and generating a target image frame according to the redrawing image and the image frame on the second canvas. The technical problem that a traditional canvas rendering technology is high in resource occupation is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to an image processing method, apparatus, electronic device, and computer program product. Background Technology

[0002] Traditional Canvas rendering requires clearing and redrawing the entire canvas every frame. At 4K / 8K resolution, GPU memory bandwidth usage exceeds 80%, resulting in a frame rate of less than 30 FPS, which cannot meet the needs of real-time interaction and suffers from low efficiency in full-scale rendering.

[0003] Existing Canvas rendering solutions rely on manual marking or simple rectangle detection, which has an error rate of up to 35% for irregular shapes (such as curves and scatter plots) and severe edge jaggedness (SSIM<0.87), resulting in insufficient accuracy in local updates.

[0004] The difference detection algorithm based on pixel traversal has a complexity of O(n²) and a large computational cost; traditional dirty area merging technology is prone to screen tearing and cannot adapt to dynamic scenes.

[0005] Large data dashboards need to support smooth rendering at 60FPS at 8K resolution; mobile devices require a reduction in memory usage of more than 50% to extend battery life; online collaboration tools need millisecond-level response latency to improve user experience. However, current canvas rendering technology cannot meet the core needs of the industry.

[0006] There is currently no effective solution to the problem of high resource consumption in traditional canvas rendering techniques. Summary of the Invention

[0007] This invention provides an image processing method, apparatus, electronic device, and computer program product to at least solve the technical problem of high resource consumption in traditional canvas rendering technology.

[0008] According to one aspect of the present invention, an image processing method is provided, comprising: deploying image frames to be processed obtained from a video stream on a first canvas, wherein the video stream includes a plurality of image frames; dividing the first canvas into a plurality of first image blocks of a preset size according to the frame rate of the video stream, wherein the preset size is negatively correlated with the frame rate; determining dirty regions based on the difference between each first image block and a second image block at a corresponding position on a second canvas, wherein the second canvas pre-deploys a previous image frame of the image frames to be processed, the second canvas is divided into a plurality of second image blocks of the preset size, and the dirty regions are determined at least based on first image blocks whose difference with the second image blocks is greater than a preset difference threshold; redrawing the dirty regions to obtain a redrawn image; and generating a target image frame based on the redrawn image and the image frames on the second canvas.

[0009] Optionally, determining the dirty region based on the difference between each first image block and the corresponding second image block on the second canvas includes: calculating the difference between each first image block and the corresponding second image block on the second canvas; determining the first image block whose difference with the second image block is greater than the preset difference threshold as a dirty block; performing convex hull calculation on the multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks, wherein each convex polygon covers at least one dirty block; and determining the area covered by each convex polygon on the first canvas as the dirty region.

[0010] Optionally, calculating the difference between each first image block and the corresponding second image block on the second canvas includes: determining the position information of each first image block on the first canvas; determining the second image block corresponding to each first image block on the second canvas based on the position information; and calculating the difference between the first image block and the corresponding second image block.

[0011] Optionally, calculating the difference between the first image block and the corresponding second image block includes: converting a third image block represented using the RGB color space into a fourth image block represented using the YUV color space, wherein the third image block includes the first image block and the second image block, and the fourth image block includes a fifth image block converted from the first image block and a sixth image block converted from the second image block; calculating the difference value between the fifth image block and the corresponding sixth image block, wherein the difference value is based on the Euclidean distance in the YUV color space.

[0012] Optionally, performing convex hull calculation on multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks includes: arranging the center points of all the dirty blocks in ascending order of the X-axis coordinate on the first canvas to obtain a dirty block point set, wherein the dirty block point set includes the center point of each dirty block, and the center point located at the edge of the dirty block point set is a vertex; traversing each vertex from left to right on the lower edge of the dirty block point set near the origin of the Y-axis of the first canvas, and removing vertices that cause right rotation to obtain lower convex hull vertices; traversing each vertex from right to left on the upper edge of the dirty block point set away from the origin of the Y-axis of the first canvas, and removing vertices that cause right rotation to obtain upper convex hull vertices; merging the upper convex hull vertices and the lower convex hull vertices to obtain the convex polygon.

[0013] Optionally, after determining the area covered by the convex polygon on the first canvas as the dirty region, the method further includes: detecting whether the boundary overlap range of any two dirty regions exceeds a preset ratio threshold; merging the two dirty regions whose boundary overlap range exceeds the preset ratio threshold into a minimum bounding rectangle, wherein the area covered by the minimum bounding rectangle on the first canvas is the dirty region.

[0014] Optionally, generating a target image frame based on the redrawn image and the image frames on the second canvas includes: extending the redrawn image by a predetermined number of pixels along the normal direction of the edge of the redrawn image to obtain an extended region; filling the extended region with a transparency formula, wherein the transparency formula is used to set the transparency of the extended region to gradually decrease along the normal direction of the edge of the redrawn image; and merging the redrawn image, the extended region, and the image frames on the second canvas to obtain the target image frame.

[0015] According to another aspect of the present invention, an image processing apparatus is also provided, comprising: a deployment module for deploying image frames to be processed acquired from a video stream on a first canvas, wherein the video stream includes a plurality of image frames; a partitioning module for partitioning the first canvas into a plurality of first image blocks of a preset size according to the frame rate of the video stream, wherein the preset size is negatively correlated with the frame rate; a determination module for determining dirty regions based on the difference between each first image block and a second image block at a corresponding position on a second canvas, wherein the second canvas has a previous image frame of the image frames to be processed pre-deployed, the second canvas is partitioned into a plurality of second image blocks of the preset size, and the dirty regions are determined at least based on first image blocks whose difference with the second image blocks is greater than a preset difference threshold; a redrawing module for redrawing the dirty regions to obtain a redrawn image; and a generation module for generating a target image frame based on the redrawn image and the image frames on the second canvas.

[0016] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the above-described image processing method through the computer program.

[0017] According to another aspect of the present invention, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0018] The embodiments described above in this application achieve efficient local updates of image frames in a video stream through a double buffering mechanism and block difference detection technology. The frame rate of the video stream is used as the basis for dynamically adjusting the detection granularity, ensuring that dirty areas can be accurately captured under different load conditions. Dirty areas can be accurately identified and only these areas are redrawn, which greatly reduces unnecessary GPU computation and video memory bandwidth usage and improves overall rendering efficiency. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of a block detection process according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of a convex hull calculation and region merging process according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of an anti-aliasing compensation process according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of a dynamic load balancing process according to an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of an exception handling process according to an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of a Canvas local rendering optimization method based on dynamic detection of dirty regions according to an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention;

[0028] Figure 9 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0032] Dirty Region: This refers to the area in the Canvas where the content has changed. It needs to be identified and locally redrawn using an intelligent detection algorithm to avoid the performance loss of full rendering.

[0033] Convex Hull: A computational geometry algorithm used to wrap a scattered set of varying pixels into a minimum convex polygon, accurately locating the boundary of dirty regions.

[0034] Anti-aliasing Compensation: Eliminates jagged edges caused by local updates by extending the boundaries of the redrawn region and applying a transparency gradient.

[0035] WebAssembly (WASM): A low-level assembly-like language used for high-performance computing tasks, used in this scheme to accelerate the convex hull algorithm.

[0036] SSIM (Structural Similarity Index): An objective evaluation index that measures the smoothness of image edges. The value ranges from 0 to 1, and a higher value indicates better visual fidelity.

[0037] YUV color space: a color coding system that separates luminance (Y) and chromaticity (U, V), which is more in line with the characteristics of human eye perception and reduces misjudgment caused by luminance fluctuations.

[0038] According to an embodiment of the present invention, an image processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0040] Step S102: Deploy the image frames to be processed obtained from the video stream on the first canvas, wherein the video stream includes: multiple image frames;

[0041] Step S104: Based on the frame rate of the video stream, the first canvas is divided into multiple first image blocks of a preset size, wherein the preset size is negatively correlated with the frame rate;

[0042] Step S106: Determine the dirty region based on the difference between each first image block and the corresponding second image block on the second canvas. The second canvas is pre-deployed with the previous image frame of the image frame to be processed, and the second canvas is divided into multiple second image blocks of a preset size. The dirty region is determined at least based on the first image block whose difference with the second image block is greater than a preset difference threshold.

[0043] Step S108: Redraw the dirty area to obtain the redrawn image;

[0044] Step S110: Generate a target image frame based on the redrawn image and the image frame on the second canvas.

[0045] The embodiments described above in this application achieve efficient local updates of image frames in a video stream through a double buffering mechanism and block difference detection technology. The frame rate of the video stream is used as the basis for dynamically adjusting the detection granularity, ensuring that dirty areas can be accurately captured under different load conditions. Dirty areas can be accurately identified and only these areas are redrawn, which greatly reduces unnecessary GPU computation and video memory bandwidth usage and improves overall rendering efficiency.

[0046] In step S102 above, the size of the first canvas is set to 3840×2160 pixels, the WebGL 2.0 context is enabled, and the antialiasing parameter (antialias: true) is configured.

[0047] It should be noted that the first canvas can be the main canvas, and the second canvas can be an off-screen buffer canvas. The second canvas has the same size as the first canvas and is only used to store the previous image frame after image processing. In other words, the second canvas is used to store the previous image frame after rendering. The first canvas and the second canvas constitute a double buffering mechanism.

[0048] In step S104 above, when the frame rate is less than 30 FPS, the system enters a low-load mode and sets the preset size of the image patch to 32×32, which can reduce the computational load of dirty area detection; when the frame rate is less than or equal to 620 FPS and greater than or equal to 30 FPS, the system enters a normal-load mode and sets the preset size of the image patch to 16×16, which can achieve a balance between the computational load and accuracy of dirty area detection; when the frame rate is greater than 30 FPS, the system enters a high-load mode and sets the preset size of the image patch to 8×8, which can improve the accuracy of dirty area detection.

[0049] In step S104 above, the performance under extreme load conditions can be further optimized by using a strategy of adaptively adjusting the size of the detection block.

[0050] In step S106 above, the first image block on the first canvas and the second image block on the second canvas can be represented using the RGB color space. When the difference between the first image block and the second image block is determined, a color space conversion can be performed, that is, the first image block and the second image block represented by the RGB color space can be converted to be represented using the YUV color space. The YUV color space can separate brightness and chromaticity, which is more in line with the characteristics of human eye perception and reduces misjudgment caused by brightness fluctuations.

[0051] Optionally, the conversion formula from RGB color space to YUV color space is:

[0052] ;

[0053] ;

[0054] .

[0055] Optionally, the difference between the first image patch and the second image patch can be determined based on the Euclidean distance in the YUV color space, where the difference is represented by ΔE and calculated using the following formula: If the ΔE of more than 20% of the pixels between the first image block and the second image block is greater than 5, it is marked as a dirty block. Dirty regions can be determined based on dirty blocks.

[0056] In step S110 above, since the redrawn image is obtained by redrawing the dirty area, and the dirty area is the difference between the image frames on the first canvas and the second canvas, the target image frame after processing the image frame to be processed can be obtained based on the redrawn image and the image frames stored on the second canvas.

[0057] The embodiments described above significantly improve image processing efficiency through intelligent dirty area detection and local redrawing technology, especially in high-resolution environments such as 4K or 8K, achieving a significant reduction in GPU memory usage (approximately 60%) and a significant increase in rendering frame rate (from 22FPS to 65FPS). Through anti-aliasing compensation technology, the smoothness of edge (SSIM) is improved from 0.87 to 0.96, greatly improving the visual experience and eliminating visible jagged edges. After the implementation of this technology, it not only provides a smoother interactive experience in scenarios such as data dashboards and online collaboration tools, but also reduces the memory usage of mobile devices and extends battery life.

[0058] For developers, the solution's non-intrusive integration greatly simplifies the development process and improves development efficiency, while ensuring consistency and stability across various hardware environments. It is a powerful tool for enhancing software performance and user experience.

[0059] The embodiments described above in this application also have dynamic load balancing and anomaly recovery mechanisms, which can monitor system load in real time, automatically adjust resource allocation strategies, and ensure stable performance under different device and network conditions.

[0060] Optionally, in the event of context loss, the system can automatically save the state and attempt to rebuild it, greatly enhancing the reliability and robustness of the system.

[0061] As an optional embodiment, determining the dirty region based on the difference between each first image block and the corresponding second image block on the second canvas includes: calculating the difference between each first image block and the corresponding second image block on the second canvas; determining the first image block whose difference with the second image block is greater than a preset difference threshold as a dirty block; performing convex hull calculation on multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks, wherein each convex polygon covers at least one dirty block; and determining the area covered by each convex polygon on the first canvas as a dirty region.

[0062] In the above embodiments of this application, the dirty region refers to the area in the Canvas where the content changes. It is necessary to identify the content of the dirty region and perform local redrawing. Therefore, in the process of identifying the dirty region, the first image block belonging to the dirty region can be determined as the dirty block based on the difference between the first image block and the second image block. Then, the discrete dirty block coordinates are converted into continuous convex polygons through the convex hull algorithm, which effectively locates the boundary of the dirty region, realizes the accurate identification of the dirty region and the merging of the dirty blocks, and reduces the computational burden during local updates.

[0063] Optionally, when the number of dirty blocks is too large, the QuickHull algorithm can be introduced as an alternative to balance computational efficiency and accuracy.

[0064] As an optional embodiment, calculating the difference between each first image block and a corresponding second image block on the second canvas includes: determining the position information of each first image block on the first canvas; determining the second image block corresponding to each first image block on the second canvas based on the position information; and calculating the difference between the first image block and the corresponding second image block.

[0065] The above embodiments of this application employ a location information matching strategy to ensure the accuracy of dirty area detection. Through precise coordinate mapping, the difference calculation of each image block between two image frames can be more accurate, avoiding misjudgment caused by coordinate offset, improving the robustness of dirty area detection, and maintaining a low error rate even at high resolution.

[0066] Alternatively, when calculating the difference between the first image block and the corresponding second image block, you can further improve the sensitivity of difference detection by introducing more advanced color space conversion techniques, such as HSV or HSL.

[0067] As an optional embodiment, calculating the difference between the first image block and the corresponding second image block includes: converting a third image block represented using the RGB color space into a fourth image block represented using the YUV color space, wherein the third image block includes: the first image block and the second image block, and the fourth image block includes: a fifth image block converted from the first image block and a sixth image block converted from the second image block; calculating the difference value between the fifth image block and the corresponding sixth image block, wherein the difference value is the Euclidean distance based on the YUV color space.

[0068] The embodiments described above in this application achieve difference detection that better conforms to the perceptual characteristics of the human eye through color space conversion. Furthermore, converting image blocks represented using the RGB color space to the YUV color space separates brightness and chromaticity, allowing difference calculation to focus more on color changes rather than brightness fluctuations. This improves the accuracy of dirty area detection, effectively reduces misjudgments caused by brightness changes, and enhances the precision of dirty area detection.

[0069] Optionally, by introducing machine learning models to predict dirty areas, the detection speed and accuracy can be further improved.

[0070] As an optional embodiment, performing convex hull calculation on multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks includes: arranging the center points of all dirty blocks in ascending order of the X-axis coordinate on the first canvas to obtain a dirty block point set, wherein the dirty block point set includes the center point of each dirty block, and the center point located at the edge of the dirty block point set is a vertex; at the lower edge of the dirty block point set near the origin of the Y-axis of the first canvas, traversing each vertex in left-to-right order and removing vertices that cause right rotation to obtain lower convex hull vertices; at the upper edge of the dirty block point set away from the origin of the Y-axis of the first canvas, traversing each vertex in right-to-left order and removing vertices that cause right rotation to obtain upper convex hull vertices; merging the upper convex hull vertices and lower convex hull vertices to obtain convex polygons.

[0071] In the embodiments described above, the convex hull calculation adopts an improved Andrew algorithm, which achieves fast and accurate convex hull calculation. The improved Andrew algorithm ensures that the generation of the convex hull is both fast and accurate through coordinate sorting and vertex culling strategies, so that the generated convex polygon can tightly wrap the dirty area, reduce the invalid area during redrawing, and improve rendering efficiency.

[0072] Alternatively, the speed of convex hull computation can be further accelerated by introducing parallel computing techniques, such as Web Workers or GPU compute shaders.

[0073] As an optional embodiment, after determining the area covered by the convex polygon on the first canvas as the dirty area, the method further includes: detecting whether the boundary overlap range of any two dirty areas exceeds a preset ratio threshold; merging the two dirty areas whose boundary overlap range exceeds the preset ratio threshold into a minimum bounding rectangle, wherein the area covered by the minimum bounding rectangle on the first canvas is the dirty area.

[0074] The above embodiments of this application can achieve efficient management of dirty areas through a region merging strategy. By setting an overlap ratio threshold, adjacent or partially overlapping dirty areas are merged into a single redraw area, reducing the number of redraw operations. This not only improves rendering efficiency but also avoids screen tearing caused by frequent local updates.

[0075] Optionally, by introducing a deep learning-based dirty region prediction model, possible dirty region change trends can be predicted in advance, and the rendering strategy can be further optimized.

[0076] As an optional embodiment, generating a target image frame based on the redrawn image and the image frame on the second canvas includes: extending a predetermined number of pixels along the normal direction of the edge of the redrawn image to obtain an extended region; filling the extended region with a transparency formula, wherein the transparency formula is used to set the transparency of the extended region to gradually decrease along the normal direction of the edge of the redrawn image; and merging the redrawn image, the extended region, and the image frame on the second canvas to obtain the target image frame.

[0077] The embodiments described above in this application achieve smooth local updates by using anti-aliasing compensation technology. The anti-aliasing compensation technology solves the problem of jagged edges during local updates by expanding the boundary and varying the transparency, thereby improving visual fidelity and significantly improving edge smoothness (SSIM), making the image quality after local updates close to the level of full rendering.

[0078] Optionally, more advanced filtering techniques, such as Gaussian blur or bilateral filtering, can be introduced to further improve edge smoothness and enhance visual effects.

[0079] The specific working process of the above embodiments of this application is as follows: First, the main Canvas (i.e., the first canvas) and the off-screen Buffer Canvas (i.e., the second canvas) are initialized to establish a double buffering mechanism; then, real-time data streams are received via WebSocket to trigger the execution of the rendering pipeline. In the rendering loop, the content of the previous image frame is copied from the main Canvas to the off-screen Buffer Canvas. Then, based on the YUV color space, block difference detection is performed on the two canvases to identify dirty areas; subsequently, the improved Andrew algorithm is applied to the dirty areas to perform convex hull calculation to determine the boundaries of the dirty areas. Next, anti-aliasing compensation is performed on the convex hull region, and boundary expansion and transparency gradient processing are performed to improve edge smoothness; finally, the detection granularity is dynamically adjusted according to the real-time frame rate to achieve performance adaptation, and the rendering state is automatically saved and restored in abnormal situations to ensure the stability and reliability of the system.

[0080] Throughout the entire process, this application achieves full hardware environment compatibility from low-end mobile devices to high-end workstations through intelligent dirty area detection and local redrawing technology, covering more than 90% of terminal devices, significantly improving rendering efficiency and visual experience, while reducing the maintenance costs for developers.

[0081] This invention also provides an optional embodiment, which offers a Canvas local rendering optimization method based on dynamic detection of dirty regions. The core objective of this method is to address three major problems in traditional Canvas rendering: high overhead of invalid drawing, low accuracy of local updates, and severe edge jaggedness. Through intelligent dirty region detection and anti-aliasing compensation technology, it can optimize performance, increasing the frame rate from 22 FPS to 65 FPS at 4K resolution while reducing GPU memory usage by 60%. It also improves accuracy, reducing the dirty region detection error rate from 35% to 3%, supporting high-precision local updates for irregular graphics. Furthermore, it enhances visual fidelity by improving edge smoothness (SSIM) from 0.87 to 0.96 through anti-aliasing compensation technology, eliminating visible jaggedness. It expands compatibility, adapting to all hardware environments from low-end mobile devices (1GB VRAM) to high-end workstations, covering over 90% of terminal devices. Finally, it improves development efficiency by providing a zero-intrusive integration solution, allowing developers to enable intelligent rendering without modifying business code, reducing maintenance costs by 70%.

[0082] As an optional implementation, the Canvas local rendering optimization method based on dirty region dynamic detection employs a double-buffering mechanism. The main canvas is bound to the HTML document, with a size of 3840×2160 pixels. A WebGL 2.0 context is enabled, and anti-aliasing parameters (antialias: true) are configured. An off-screen buffer canvas is created, which is an off-screen canvas of the same size used only to store the complete pixel data of the previous frame and does not participate in DOM rendering. Static background elements (such as grids and coordinate axes) are drawn to both canvases to ensure initial consistency. The `requestAnimationFrame` is called to start the rendering loop, aligning with the browser refresh rate (60Hz). At the beginning of each frame, `drawImage` is used to copy the main canvas content to the off-screen buffer canvas. The `requestAnimationFrame` callback ensures that double-buffering swapping only occurs within the vertical synchronization cycle, eliminating screen tearing.

[0083] Figure 2 This is a schematic diagram of a block detection process according to an embodiment of the present invention, such as... Figure 2 As shown, the canvas is divided into fixed-size blocks (default 16×16 pixels), and dirty areas are identified based on color space differences; then, a color space conversion is performed, specifically RGB to YUV conversion, using the following formula:

[0084] ;

[0085] ;

[0086] .

[0087] It should be noted that the YUV color space can separate luminance and chrominance, which is more in line with the characteristics of human eye perception and reduces misjudgment caused by luminance fluctuations.

[0088] Optionally, when performing difference calculations, the Euclidean distance ΔE based on the YUV space can be calculated using the following formula: If the ΔE of more than 20% of the pixels between the first image block and the second image block is greater than 5, it is marked as a dirty block. Dirty regions can be determined based on dirty blocks.

[0089] Optionally, the preset size of the segmented image patch can be dynamically adjusted with granularity according to the frame rate of the video stream, as shown below:

[0090] Low load mode (frame rate <30FPS): Block size is increased to 32×32, reducing the amount of detection computation;

[0091] High load mode (frame rate > 60FPS): The block size is reduced to 8×8, improving detection accuracy.

[0092] Figure 3 This is a schematic diagram of a convex hull calculation and region merging process according to an embodiment of the present invention, as shown below. Figure 3 As shown, the improved Andrew algorithm is used to convert discrete dirty block coordinates into continuous convex polygons, accurately locating the boundaries of the dirty region.

[0093] Optionally, improvements to the Andrew algorithm process include:

[0094] Step S1: Sort by coordinates, arranging the center points of all dirty blocks in ascending order of X-axis coordinates.

[0095] Step S2: Construct the lower convex hull by traversing the point set from left to right and removing vertices that cause a right turn (cross product ≤ 0).

[0096] Step S3: Construct the upper convex hull by traversing the point set from right to left and removing vertices that cause a right turn (cross product ≤ 0).

[0097] Step S4: Merge the results, merge the vertices of the upper and lower convex hulls, remove duplicate points, and generate the final convex polygon.

[0098] Step S5, performance acceleration: Using WebAssembly acceleration technology, the core algorithm is implemented in Rust and compiled into a WASM module, resulting in a 3x performance improvement.

[0099] Step S6: Approximate algorithm switching. When the number of dirty blocks > 1000, switch to the QuickHull algorithm (time complexity O(n log h)).

[0100] Figure 4This is a schematic diagram of an anti-aliasing compensation process according to an embodiment of the present invention, as shown below. Figure 4 As shown, visual fidelity is improved by eliminating jagged edges caused by local updates.

[0101] Optionally, anti-aliasing compensation can be achieved through boundary expansion, which can extend outward by 2 pixels along the normal direction of each side of the convex polygon.

[0102] Optionally, the mathematical implementation of boundary expansion for anti-aliasing compensation includes: , After normalization, the expansion is performed.

[0103] Optionally, anti-aliasing compensation can achieve a gradual change in transparency using a transparency formula, which is: .

[0104] It should be noted that anti-aliasing compensation can achieve a smooth transition from the original boundary to the extended edge, with the transparency decreasing linearly from 1 to 0.

[0105] Optionally, bilinear filtering technology can be used to perform interpolation calculations on the extended region to eliminate pixelated jagged edges.

[0106] Optionally, bilinear filtering is implemented by enabling Canvas context configuration, specifically as follows:

[0107] ctx.imageSmoothingEnabled = true;

[0108] ctx.imageSmoothingQuality = 'high'.

[0109] Figure 5 This is a schematic diagram of a dynamic load balancing process according to an embodiment of the present invention, such as... Figure 5 As shown, the system load is monitored in real time, and the resource allocation strategy is dynamically adjusted.

[0110] Optionally, dynamic load balancing is achieved through frame rate monitoring, using performance.now() to calculate the frame interval time, with a target frame rate ≥ 60 FPS (each frame taking ≤ 16.6 ms).

[0111] Optionally, dynamic load balancing supports adaptive strategies to adjust the granularity of dirty region detection, specifically:

[0112] Frame rate <30FPS → Adjust block size to 32×32 to reduce computational load;

[0113] Frame rate > 60 FPS → Block size adjusted to 8×8 to improve detection accuracy.

[0114] Optionally, dynamic load balancing supports handling memory pressure: when memory usage is >80%, LOD (Level of Detail) optimization is automatically enabled.

[0115] Optionally, dynamic load balancing can simplify graphic elements that are far from the viewpoint (such as changing circular markers to squares).

[0116] Optionally, dynamic load balancing supports texture compression, which can switch the format from RGBA32F to ASTC 4x4, reducing the size by 70%.

[0117] Figure 6 This is a schematic diagram of an exception handling process according to an embodiment of the present invention, such as... Figure 6 As shown, it supports context loss detection by listening to the webglcontextlost event; it can save the state by serializing and storing vertex buffers and texture data to IndexedDB; and it supports a reconstruction mechanism that attempts to rebuild the Canvas context after a 1-second delay to restore the rendering state from persistent data.

[0118] As an optional embodiment, the block difference detection process includes:

[0119] Data preparation: Convert the pixel data of the main Canvas and the off-screen Buffer Canvas into YUV space;

[0120] Block traversal: Traverse the canvas in 16×16 pixel blocks and calculate the ΔE value for each block;

[0121] Dirty block marking: If more than 20% of the pixels in a block satisfy ΔE > 5, it is marked as a dirty block and its coordinates are recorded;

[0122] Dynamic adjustment: The block size (8×8 to 32×32) is dynamically adjusted according to the real-time frame rate.

[0123] As an optional embodiment, the convex hull calculation and merging process includes:

[0124] Data sampling: Randomly sample 5% of the center point of the dirty block to reduce the amount of computation;

[0125] Convex hull generation: Execute the improved Andrew algorithm to generate a set of vertices for convex polygons;

[0126] Region merging: Overlapping merging: If the boundaries of two dirty regions overlap by more than 50%, they are merged into the smallest enclosing rectangle;

[0127] Small-area clustering: Independent dirty areas with an area of ​​less than 5% of the canvas are merged into neighboring areas;

[0128] Output: Generates a final list of dirty region coordinates for local redrawing.

[0129] As an optional embodiment, the anti-aliasing compensation process includes:

[0130] Boundary expansion: Expand outward by 2 pixels along the normal direction of each side of the convex polygon to generate an expanded region;

[0131] Gradient processing: Fill the extended area according to the transparency formula to achieve a smooth transition;

[0132] Filtering optimization: Enable bilinear filtering to eliminate pixel jaggedness after magnification.

[0133] As an optional implementation, the dynamic adaptive strategy includes: frame rate-driven real-time adjustment and anomaly recovery process.

[0134] Optionally, frame rate-driven real-time modulation includes:

[0135] Frame rate calculation: Calculate the average frame rate every 10 frames: Average frame rate = \frac{\sum_{i=1}^{10} t_i}{10}];

[0136] Policy Execution: Low-load mode: Detection granularity is increased to 32×32, reducing the number of GPU instruction submissions;

[0137] High load mode: The detection granularity is reduced to 8×8 to avoid missing small areas of change.

[0138] Optionally, the exception recovery process includes:

[0139] Context loss detection: Listen for the browser's webglcontextlost event and capture exception signals;

[0140] State persistence: Serialize vertex buffers and texture data into ArrayBuffers and store them in IndexedDB;

[0141] Delayed Reconstruction: Attempt to rebuild the WebGL context after waiting 1 second to avoid frequent crash loops;

[0142] Data recovery: Load persistent data from IndexedDB and rebind it to the newly created context;

[0143] Degradation strategy: If reconstruction fails, switch to full rendering mode and trigger an alarm notification.

[0144] Figure 7 This is a schematic diagram of a Canvas local rendering optimization method based on dynamic detection of dirty regions according to an embodiment of the present invention, as shown below. Figure 7 As shown, this method can be applied to real-time rendering of 4K data dashboards, and the specific steps are as follows:

[0145] Step S701: Initialize the double-buffered Canvas;

[0146] Step S702, data binding and event triggering;

[0147] Step S703, block difference detection;

[0148] Step S704: Convex hull calculation and region merging;

[0149] Step S705, anti-aliasing compensation processing;

[0150] Step S706: Local redrawing and performance optimization.

[0151] In step S701 above, the double-buffered Canvas is initialized, including the following steps:

[0152] Step S7011, Main Canvas Creation: Create a Canvas with a resolution of 3840×2160 and bind it to the DOM display node. Enable the WebGL 2.0 context and configure the antialiasing parameter (antialias: true).

[0153] Step S7012, Off-screen Buffer Canvas Configuration: Create an off-screen canvas of the same resolution to store the complete rendering result of the previous frame. Double buffering is implemented using requestAnimationFrame to avoid screen tearing.

[0154] Step S7013, Synchronization Mechanism: After each frame is rendered, the content of the main Canvas is copied to the off-screen BufferCanvas as the reference data for the next frame.

[0155] In step S702 above, data binding and event triggering include the following steps:

[0156] Step S7021, Data source access: Receive real-time data streams (transmission frequency 30Hz) using the WebSocket protocol, with the data format being timestamp-value pairs.

[0157] Optionally, an example of a packet structure is as follows:

[0158] "Timestamp (milliseconds) | Value"

[0159] 1620000000000 | 42.5

[0160] 1620000000033 | 43.1.

[0161] Step S7022, Event-driven rendering: Every time WebSocket receives a new data packet, it triggers the execution flow of the rendering pipeline.

[0162] Step S7023, Data preprocessing: Convert the raw data into screen coordinates (normalized to canvas size).

[0163] In step S703 above, the block difference detection includes the following steps:

[0164] Step S7031, Canvas Blocking Strategy: Divide the 3840×2160 canvas into 16×16 pixel blocks, totaling 240 (horizontal) × 135 (vertical) = 32,400 blocks. Calculate feature values ​​for each block independently, reducing the computational overhead of global traversal.

[0165] Step S7032, Color space conversion: Convert RGB pixel data to YUV space and separate the luminance (Y) and chrominance (U, V) components.

[0166] Alternatively, the conversion formula is:

[0167] Luminance: Y = 0.299R + 0.587G + 0.114B;

[0168] Chromaticity: U = 0.492 (B - Y), V = 0.877 (R - Y).

[0169] Step S7033, Difference Determination Logic: Calculate the ΔE value (based on Euclidean distance in YUV space) between the corresponding block of the current frame and the previous frame: .

[0170] Step S7034, Judgment condition: If more than 20% of the pixels in a block have ΔE > 5, it is marked as a dirty block. Dynamic sensitivity adjustment: When the frame rate is lower than 30 FPS, the judgment threshold is automatically relaxed (ΔE > 7).

[0171] In step S704 above, the convex hull calculation and region merging include the following steps:

[0172] Step S7041, Convex Hull Algorithm Optimization, implemented by improving the Andrew algorithm, specifically includes:

[0173] Step S70411, Coordinate sorting: Arrange all dirty block center points in ascending order of X-axis coordinate.

[0174] Step S70412, Lower convex hull construction: Traverse from left to right and remove points that cause a right turn.

[0175] Step S70413, Constructing the upper convex hull: Traverse from right to left, removing points that cause a right turn.

[0176] Step S70414, merge convex hulls: the set of vertices of the upper and lower convex hulls constitutes the final convex polygon.

[0177] Step S7042, performance acceleration: WebAssembly (Rust implementation) is used instead of JavaScript, reducing computation time from 15ms to 5ms. When the number of dirty blocks > 1000, the QuickHull approximation algorithm is switched to (time complexity O(n log h)).

[0178] Step S7043, region merging rules, including: overlapping merging: if the boundaries of two dirty regions overlap by more than 50%, they are merged into the smallest enclosing rectangle; small area clustering: independent dirty regions with an area less than 5% of the canvas are merged into the nearest neighbor region.

[0179] In step S705 above, the anti-aliasing compensation process includes the following steps:

[0180] Step S7051, Edge expansion strategy: Cover the jagged area by expanding outward by 2 pixels along the normal direction of the convex polygon to cover the jagged area (coverage rate reaches 99.2%); Expanded area calculation: Based on the vertex coordinates of the convex polygon, translate along the normal direction to generate a new boundary.

[0181] Step S7052, transparency gradient: using a gradient formula from the boundary to the edge of the extended area, the transparency linearly decreases from 1 to 0 to achieve a smooth transition.

[0182] Step S7053: Configure bilinear filtering, enable imageSmoothingEnabled and imageSmoothingQuality: 'high' in the Canvas context to eliminate pixel jaggedness after magnification.

[0183] In step S706 above, the local redrawing and performance optimization include the following steps:

[0184] Step S7061, Optimize drawing calls: Perform the drawImage operation only on the merged dirty regions, reducing the number of drawing calls from a total of 240 to an average of 35. Employ instantiation drawing techniques (such as WebGL's gl.drawArraysInstanced) to batch process similar graphic elements.

[0185] Step S7062, Double buffer swapping mechanism: After each frame is rendered, the content of the main Canvas is copied to the off-screen Buffer Canvas.

[0186] Step S7063: Use requestAnimationFrame to ensure that the refresh rate is synchronized with the monitor to avoid screen tearing.

[0187] In step S707 above, dynamic load balancing and anomaly handling include the following steps:

[0188] Step S7071, Frame Rate Monitoring and Adaptation: Real-time calculation of frame interval (target 16.6ms / frame), dynamically adjusting the granularity of dirty area detection: Low load mode (frame rate <30FPS), the detection granularity is expanded to 32×32 pixel blocks to reduce the amount of computation; High load mode (frame rate >60FPS), the detection granularity is reduced to 8×8 pixel blocks to improve accuracy.

[0189] Step S7072, dealing with video memory pressure: When video memory usage exceeds 80%, automatically enable LOD (Level of Detail) optimization: replace elements far from the viewpoint with simplified models (e.g., change circular data points to 2px squares); switch the texture compression format from RGBA32F (16MB / texture) to ASTC 4x4 (4.8MB / texture).

[0190] Step S7073, the exception recovery mechanism, includes: Context loss handling: Listen for the `webglcontextlost` event and pause the rendering pipeline. Save the current rendering state (vertex buffer, texture data) to IndexedDB. After a 1-second delay, attempt to rebuild the WebGL context and resume rendering from the persistent data.

[0191] Table 1 is a schematic table comparing the effects before and after optimization according to an embodiment of the present invention. As shown in Table 1, there are significant improvements in many performance indicators.

[0192] Table 1

[0193]

[0194] According to an embodiment of the present invention, an image processing apparatus embodiment is also provided. It should be noted that the image processing apparatus can be used to execute the image processing method in the embodiment of the present invention, and the image processing method in the embodiment of the present invention can be executed in the image processing apparatus.

[0195] Figure 8 This is a schematic diagram of an image processing apparatus according to an embodiment of the present invention, such as... Figure 8As shown, the device may include: a deployment module 81, used to deploy image frames to be processed obtained from a video stream on a first canvas, wherein the video stream includes multiple image frames; a partitioning module 83, used to divide the first canvas into multiple first image blocks of a preset size according to the frame rate of the video stream, wherein the preset size is negatively correlated with the frame rate; a determination module 85, used to determine dirty regions based on the difference between each first image block and a second image block at a corresponding position on a second canvas, wherein the previous image frame of the image frame to be processed is pre-deployed on the second canvas, the second canvas is divided into multiple second image blocks of a preset size, and the dirty regions are determined at least based on first image blocks whose difference from the second image blocks is greater than a preset difference threshold; a redrawing module 87, used to redraw the dirty regions to obtain a redrawn image; and a generation module 89, used to generate a target image frame based on the redrawn image and the image frames on the second canvas.

[0196] It should be noted that the image processing in this embodiment can be used to execute step S102 in this application embodiment, the division module 83 in this embodiment can be used to execute step S104 in this application embodiment, the determination module 85 in this embodiment can be used to execute step S106 in this application embodiment, the redrawing module 87 in this embodiment can be used to execute step S108 in this application embodiment, and the generation module 89 in this embodiment can be used to execute step S110 in this application embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0197] The embodiments described above in this application achieve efficient local updates of image frames in a video stream through a double buffering mechanism and block difference detection technology. The frame rate of the video stream is used as the basis for dynamically adjusting the detection granularity, ensuring that dirty areas can be accurately captured under different load conditions. Dirty areas can be accurately identified and only these areas are redrawn, which greatly reduces unnecessary GPU computation and video memory bandwidth usage and improves overall rendering efficiency.

[0198] As an optional embodiment, the determining module includes: a first calculation unit, used to calculate the difference between each first image block and a second image block at a corresponding position on the second canvas; a first determining unit, used to determine the first image block whose difference with the second image block is greater than a preset difference threshold as a dirty block; a second calculation unit, used to perform convex hull calculation on multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks, wherein each convex polygon covers at least one dirty block; and a second determining unit, used to determine the area covered by each convex polygon on the first canvas as a dirty region.

[0199] As an optional embodiment, the first calculation unit includes: a first determining subunit, configured to determine the position information of each first image block on a first canvas; a second determining subunit, configured to determine a second image block corresponding to each first image block on a second canvas based on the position information; and a first calculation subunit, configured to calculate the difference between the first image block and the corresponding second image block.

[0200] As an optional embodiment, the first calculation subunit includes: a conversion subunit for converting a third image block represented using RGB color space into a fourth image block represented using YUV color space, wherein the third image block includes: a first image block and a second image block, and the fourth image block includes: a fifth image block converted from the first image block and a sixth image block converted from the second image block; and a second calculation subunit for calculating the difference value between the fifth image block and the corresponding sixth image block, wherein the difference value is based on the Euclidean distance in the YUV color space.

[0201] As an optional embodiment, the second calculation unit includes: a sorting subunit, used to arrange the center points of all dirty blocks in ascending order of the X-axis coordinate on the first canvas to obtain a dirty block point set, wherein the dirty block point set includes the center point of each dirty block, and the center point located at the edge of the dirty block point set is a vertex; a first traversal subunit, used to traverse each vertex in left-to-right order at the lower edge of the dirty block point set near the origin of the Y-axis of the first canvas, and remove vertices that cause right rotation to obtain lower convex hull vertices; a second traversal subunit, used to traverse each vertex in right-to-left order at the upper edge of the dirty block point set away from the origin of the Y-axis of the first canvas, and remove vertices that cause right rotation to obtain upper convex hull vertices; and a merging subunit, used to merge upper convex hull vertices and lower convex hull vertices to obtain a convex polygon.

[0202] As an optional embodiment, the device further includes: a detection unit, configured to detect whether the boundary overlap range of any two dirty regions exceeds a preset ratio threshold after the area covered by the convex polygon on the first canvas is determined to be a dirty region; and a first merging unit, configured to merge two dirty regions whose boundary overlap range exceeds the preset ratio threshold into a minimum bounding rectangle, wherein the area covered by the minimum bounding rectangle on the first canvas is the dirty region.

[0203] As an optional embodiment, the generation module includes: an expansion unit for expanding a preset number of pixels along the normal direction of the edge of the redrawn image to obtain an expanded area; a filling unit for filling the expanded area using a transparency formula, wherein the transparency formula is used to set the transparency of the expanded area to gradually decrease along the normal direction of the edge of the redrawn image; and a second merging unit for merging the redrawn image, the expanded area, and image frames on the second canvas to obtain a target image frame.

[0204] Embodiments of the present invention can provide an electronic device, which can be a computer terminal, and the computer terminal can be any one of a group of computer terminal devices. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0205] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0206] In this embodiment, the computer terminal described above can execute program code for the following steps in the image processing method: deploying image frames to be processed obtained from a video stream on a first canvas, wherein the video stream includes: multiple image frames; dividing the first canvas into multiple first image blocks of a preset size according to the frame rate of the video stream, wherein the preset size is negatively correlated with the frame rate; determining dirty regions based on the difference between each first image block and a corresponding second image block on a second canvas, wherein the previous image frame of the image frame to be processed is pre-deployed on the second canvas, the second canvas is divided into multiple second image blocks of a preset size, and the dirty regions are determined at least based on first image blocks whose difference from the second image blocks is greater than a preset difference threshold; redrawing the dirty regions to obtain a redrawn image; generating a target image frame based on the redrawn image and the image frames on the second canvas.

[0207] Figure 9 This is a structural block diagram of a computer terminal according to an embodiment of the present invention, such as... Figure 9 As shown, the computer terminal 90 may include one or more (only one is shown in the figure) processors 92 and memory 94.

[0208] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal 90 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0209] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: deploying image frames to be processed obtained from a video stream on a first canvas, wherein the video stream includes multiple image frames; dividing the first canvas into multiple first image blocks of a preset size according to the frame rate of the video stream, wherein the preset size is negatively correlated with the frame rate; determining dirty regions based on the difference between each first image block and a corresponding second image block on a second canvas, wherein the previous image frame of the image frame to be processed is pre-deployed on the second canvas, the second canvas is divided into multiple second image blocks of a preset size, and the dirty regions are determined at least based on first image blocks whose difference from the second image blocks is greater than a preset difference threshold; redrawing the dirty regions to obtain a redrawn image; generating a target image frame based on the redrawn image and the image frames on the second canvas.

[0210] Optionally, the processor may also execute program code for the following steps: calculating the difference between each first image block and the corresponding second image block on the second canvas; identifying the first image block whose difference with the second image block is greater than a preset difference threshold as a dirty block; performing convex hull calculation on multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks, wherein each convex polygon covers at least one dirty block; and identifying the area covered by each convex polygon on the first canvas as a dirty region.

[0211] Optionally, the processor may also execute program code that performs the following steps: determining the position information of each first image block on the first canvas; determining the second image block corresponding to each first image block on the second canvas based on the position information; and calculating the difference between the first image block and the corresponding second image block.

[0212] Optionally, the processor may also execute program code that performs the following steps: converting a third image block represented using the RGB color space into a fourth image block represented using the YUV color space, wherein the third image block includes a first image block and a second image block, and the fourth image block includes a fifth image block converted from the first image block and a sixth image block converted from the second image block; calculating the difference value between the fifth image block and the corresponding sixth image block, wherein the difference value is the Euclidean distance based on the YUV color space.

[0213] Optionally, the processor may also execute program code with the following steps: arranging the center points of all dirty blocks in ascending order of the X-axis coordinate on the first canvas to obtain a set of dirty block points, wherein the set of dirty block points includes the center point of each dirty block, and the center point located at the edge of the set of dirty block points is a vertex; traversing each vertex from left to right at the lower edge of the set of dirty block points near the origin of the Y-axis of the first canvas, and removing vertices that cause right rotation, to obtain lower convex hull vertices; traversing each vertex from right to left at the upper edge of the set of dirty block points away from the origin of the Y-axis of the first canvas, and removing vertices that cause right rotation, to obtain upper convex hull vertices; merging the upper convex hull vertices and lower convex hull vertices to obtain a convex polygon.

[0214] Optionally, the processor may also execute program code that performs the following steps: detects whether the boundary overlap range of any two dirty regions exceeds a preset ratio threshold; merges the two dirty regions whose boundary overlap range exceeds the preset ratio threshold into a minimum bounding rectangle, wherein the area covered by the minimum bounding rectangle on the first canvas is the dirty region.

[0215] Optionally, the processor may also execute program code for the following steps: extending a preset number of pixels along the normal direction of the edge of the redrawn image to obtain an extended region; filling the extended region with a transparency formula, wherein the transparency formula is used to set the transparency of the extended region to gradually decrease along the normal direction of the edge of the redrawn image; merging the redrawn image, the extended region, and the image frames on the second canvas to obtain the target image frame.

[0216] Those skilled in the art will understand that the structure shown in the figure is for illustrative purposes only, and the computer terminal may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 9 This does not limit the structure of the aforementioned electronic device. For example, the computer terminal 90 may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 9 The different configurations shown.

[0217] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing the hardware related to the terminal device. The computer program can be stored in a non-volatile medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0218] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the image processing method provided in the above embodiments.

[0219] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0220] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: deploying image frames to be processed obtained from a video stream on a first canvas, wherein the video stream includes: multiple image frames; dividing the first canvas into multiple first image blocks of a preset size according to the frame rate of the video stream, wherein the preset size is negatively correlated with the frame rate; determining dirty regions based on the difference between each first image block and a corresponding second image block on a second canvas, wherein the previous image frame of the image frame to be processed is pre-deployed on the second canvas, the second canvas is divided into multiple second image blocks of a preset size, and the dirty regions are determined at least based on first image blocks whose difference from the second image blocks is greater than a preset difference threshold; redrawing the dirty regions to obtain a redrawn image; generating a target image frame based on the redrawn image and the image frames on the second canvas.

[0221] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: calculating the difference between each first image block and a corresponding second image block on the second canvas; identifying first image blocks whose difference with the second image block is greater than a preset difference threshold as dirty blocks; performing convex hull calculation on multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks, wherein each convex polygon covers at least one dirty block; and identifying the area covered by each convex polygon on the first canvas as a dirty region.

[0222] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining the position information of each first image block on a first canvas; determining a second image block corresponding to each first image block on a second canvas based on the position information; and calculating the difference between the first image block and the corresponding second image block.

[0223] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: converting a third image block represented using the RGB color space into a fourth image block represented using the YUV color space, wherein the third image block includes a first image block and a second image block, and the fourth image block includes a fifth image block converted from the first image block and a sixth image block converted from the second image block; calculating the difference value between the fifth image block and the corresponding sixth image block, wherein the difference value is based on the Euclidean distance in the YUV color space.

[0224] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: arranging the center points of all dirty blocks in ascending order of the X-axis coordinate on the first canvas to obtain a dirty block point set, wherein the dirty block point set includes the center point of each dirty block, and the center point located at the edge of the dirty block point set is a vertex; at the lower edge of the dirty block point set near the origin of the Y-axis of the first canvas, traversing each vertex in order from left to right, and removing vertices that cause a right turn, to obtain lower convex hull vertices; at the upper edge of the dirty block point set away from the origin of the Y-axis of the first canvas, traversing each vertex in order from right to left, and removing vertices that cause a right turn, to obtain upper convex hull vertices; merging the upper convex hull vertices and lower convex hull vertices to obtain a convex polygon.

[0225] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: detecting whether the boundary overlap range of any two dirty regions exceeds a preset ratio threshold; merging the two dirty regions whose boundary overlap range exceeds the preset ratio threshold into a minimum bounding rectangle, wherein the area covered by the minimum bounding rectangle on the first canvas is the dirty region.

[0226] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: extending a preset number of pixels along the normal direction of the edge of the redrawn image to obtain an extended region; filling the extended region with a transparency formula, wherein the transparency formula is used to set the transparency of the extended region to gradually decrease along the normal direction of the edge of the redrawn image; merging the redrawn image, the extended region, and the image frames on the second canvas to obtain a target image frame.

[0227] Embodiments of the present invention also provide a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the steps of the image processing method provided in the above embodiments.

[0228] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0229] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0230] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0231] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0232] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0233] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned non-volatile storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0234] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: Deploy the image frames to be processed obtained from the video stream on the first canvas, wherein the video stream includes: a plurality of image frames; Based on the frame rate of the video stream, the first canvas is divided into multiple first image blocks of a preset size, wherein the preset size is negatively correlated with the frame rate; Dirty regions are determined based on the difference between each first image block and a corresponding second image block on the second canvas. The second canvas is pre-deployed with the previous image frame of the image frame to be processed, and the second canvas is divided into a plurality of second image blocks of the preset size. The dirty regions are determined at least based on first image blocks whose difference with the second image blocks is greater than a preset difference threshold. The dirty area is redrawn to obtain a redrawn image; A target image frame is generated based on the redrawn image and the image frame on the second canvas.

2. The method according to claim 1, characterized in that, Based on the difference between each of the first image blocks and the corresponding second image blocks on the second canvas, the dirty regions are determined as follows: Calculate the difference between each of the first image blocks and the corresponding second image blocks on the second canvas; The first image block whose difference from the second image block is greater than the preset difference threshold is identified as a dirty block; Perform convex hull calculation on the multiple dirty blocks to obtain multiple convex polygons covering the dirty blocks, wherein each convex polygon covers at least one of the dirty blocks; The area covered by each of the convex polygons on the first canvas is defined as the dirty region.

3. The method according to claim 2, characterized in that, Calculating the difference between each of the first image patches and the corresponding second image patches on the second canvas includes: Determine the position information of each first image block on the first canvas; Based on the location information, a second image block corresponding to each of the first image blocks is determined on the second canvas; Calculate the difference between the first image block and the corresponding second graphic block.

4. The method according to claim 3, characterized in that, Calculating the difference between the first image patch and the corresponding second image patch includes: A third image block represented using the RGB color space is converted into a fourth image block represented using the YUV color space. The third image block includes the first image block and the second image block, and the fourth image block includes the fifth image block converted from the first image block and the sixth image block converted from the second image block. Calculate the difference value between the fifth image block and the corresponding sixth image block, wherein the difference value is the Euclidean distance based on the YUV color space.

5. The method according to claim 2, characterized in that, Performing convex hull calculations on the multiple dirty blocks yields multiple convex polygons covering the dirty blocks, including: Arrange the center points of all the dirty blocks in ascending order of X-axis coordinates on the first canvas to obtain a dirty block point set, wherein the dirty block point set includes the center point of each dirty block, and the center point located at the edge of the dirty block point set is a vertex; At the lower edge of the dirty block point set near the origin of the first canvas Y-axis, each vertex is traversed in order from left to right, and the vertex that causes a right turn is removed to obtain the lower convex shell vertex; At the upper edge of the dirty point set away from the origin of the first canvas Y-axis, each vertex is traversed in order from right to left, and the vertex that causes a right turn is removed to obtain the upper convex shell vertex; The convex polygon is obtained by merging the vertices of the upper convex shell and the lower convex shell.

6. The method according to claim 2, characterized in that, After determining the area covered by the convex polygon on the first canvas as the dirty region, the method further includes: Detect whether the overlap range of any two dirty areas exceeds a preset ratio threshold; Two dirty regions whose boundary overlap exceeds the preset ratio threshold are merged into a minimum bounding rectangle, wherein the area covered by the minimum bounding rectangle on the first canvas is the dirty region.

7. The method according to claim 1, characterized in that, Generating a target image frame based on the redrawn image and the image frame on the second canvas includes: Extend the area by a predetermined number of pixels along the normal direction of the edge of the redrawn image to obtain the extended region; The extended region is filled using a transparency formula, wherein the transparency formula is used to set the transparency of the extended region to gradually decrease along the normal direction of the edge of the redrawn image; The target image frame is obtained by merging the redrawn image, the expanded area, and the image frame on the second canvas.

8. An image processing apparatus, characterized in that, include: A deployment module is used to deploy image frames to be processed obtained from a video stream on a first canvas, wherein the video stream includes: multiple image frames; The partitioning module is used to divide the first canvas into multiple first image blocks of a preset size according to the frame rate of the video stream, wherein the preset size is negatively correlated with the frame rate; The determination module is used to determine dirty regions based on the difference between each first image block and a corresponding second image block on the second canvas, wherein the second canvas is pre-deployed with the previous image frame of the image frame to be processed, and the second canvas is divided into a plurality of second image blocks of the preset size, and the dirty regions are determined at least based on first image blocks whose difference with the second image blocks is greater than a preset difference threshold. The redrawing module is used to redraw the dirty area to obtain a redrawn image; The generation module is used to generate a target image frame based on the redrawn image and the image frame on the second canvas.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the image processing method of any one of claims 1 to 7 through the computer program.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the image processing method according to any one of claims 1 to 7.

Images