Image processing method and related product

Abstract

The invention discloses an image processing method and a related product. The method comprises the following steps: acquiring a current to-be-processed target rendering image; if the image type of the target rendered image is a key frame, sending the target rendered image to the target terminal; if the image type of the target rendering image is a middle frame, dividing the target rendering image into a plurality of blocks; encoding the at least one target block to obtain an encoding result corresponding to each target block; and sending the at least one coding result to the target terminal. Wherein the at least one target block is at least one changed block in the plurality of blocks; and the target terminal is used for determining a to-be-displayed rendered image according to the at least one coding result and the currently displayed rendered image. Therefore, under the scenes of a mobile network, vehicle-mounted 4G / 5G switching or weak signals and the like, data transmission is realized with less data volume, the probability of occurrence of problems such as interaction delay, picture tearing or input lagging and the like is reduced, and the user experience is reduced.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and related products. Background Technology

[0002] With the development of vehicle-cloud integration and virtual desktop technologies, vehicles are gradually transforming from traditional independent computing terminals into "mobile computing nodes" with cloud-based collaborative capabilities. Under the vehicle-cloud integrated architecture, the virtual desktop system (V2D) becomes a crucial support for scenarios such as in-vehicle entertainment, office work, navigation, and remote maintenance. The virtual desktop system migrates complex computing, rendering, and storage tasks to cloud servers, where high-performance computing power and graphics rendering are provided. The rendering results are then transmitted in real-time to the vehicle's display terminal via the network, achieving a collaborative experience of "lightweight terminal + cloud computing."

[0003] However, existing cloud desktop systems generally rely on centralized cloud rendering and image transmission. This "all-cloud rendering" mode can achieve good picture quality and smooth interaction in high-bandwidth, low-latency network environments, but in scenarios such as mobile networks, in-vehicle 4G / 5G switching, or weak signals, there are often obvious problems such as interaction delay, screen tearing, or input lag, which reduce the user experience. Summary of the Invention

[0004] In view of the above problems, this application provides an image processing method and related products, which aim to improve user experience.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] In a first aspect, embodiments of this application provide an image processing method, which is applied in the cloud and includes:

[0007] Get the target image to be processed;

[0008] If the target rendered image is a keyframe, then the target rendered image is sent to the target terminal;

[0009] If the target rendered image is an intermediate frame, then the target rendered image is divided into multiple blocks;

[0010] Encode at least one target block to obtain the encoding result corresponding to each target block; wherein, at least one target block is at least one block that has changed among multiple blocks;

[0011] At least one encoding result is sent to the target terminal; wherein the target terminal is used to determine the rendering image to be displayed based on at least one encoding result and the currently displayed rendering image.

[0012] Optionally, sending at least one encoded result to the target terminal includes:

[0013] The degree of change in each target segment is scored to obtain the target score;

[0014] Sort at least one coding result in descending order based on at least one target score;

[0015] Divide at least one sorted encoding result into a first encoding set and a second encoding set;

[0016] The first set of codes is sent to the target terminal, and the second set of codes is sent to the target terminal after a preset time.

[0017] Optionally, sending at least one encoded result to the target terminal includes:

[0018] Obtain the real-time bandwidth between the cloud and the target terminal;

[0019] If the real-time bandwidth is less than the preset bandwidth threshold, at least one encoded result is sent to the target terminal at the preset frame rate, or, at least one encoded result is selectively retransmitted to the target terminal.

[0020] Optionally, the method further includes:

[0021] Determine the target depth map and confidence value of the target rendered image;

[0022] Send the target depth map and confidence value to the target terminal.

[0023] Secondly, embodiments of this application provide an image processing apparatus applied in the cloud, comprising:

[0024] The acquisition unit is used to acquire the target rendering image to be processed.

[0025] The first image processing unit is used to send the target rendered image to the target terminal if the image type of the target rendered image is a keyframe.

[0026] A partitioning unit is used to divide the target rendering image into multiple blocks if the target rendering image type is an intermediate frame.

[0027] The encoding unit is used to encode at least one target block to obtain the encoding result corresponding to each target block; wherein, at least one target block is at least one block that has changed among multiple blocks;

[0028] The second image processing unit is used to send at least one encoding result to the target terminal; wherein the target terminal is used to determine the rendering image to be displayed based on at least one encoding result and the currently displayed rendering image.

[0029] Optionally, the second image processing unit is used for:

[0030] The degree of change in each target segment is scored to obtain the target score;

[0031] Sort at least one coding result in descending order based on at least one target score;

[0032] Divide at least one sorted encoding result into a first encoding set and a second encoding set;

[0033] The first set of codes is sent to the target terminal, and the second set of codes is sent to the target terminal after a preset time.

[0034] Optionally, the second image processing unit is used for:

[0035] Obtain the real-time bandwidth between the cloud and the target terminal;

[0036] If the real-time bandwidth is less than the preset bandwidth threshold, at least one encoded result is sent to the target terminal at the preset frame rate, or, at least one encoded result is selectively retransmitted to the target terminal.

[0037] Optionally, the apparatus further includes: a determining unit, configured to:

[0038] Determine the target depth map and confidence value of the target rendered image;

[0039] Send the target depth map and confidence value to the target terminal.

[0040] Thirdly, embodiments of this application provide a control device, including a processor and a memory, wherein the memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to perform the image processing method as described in the first aspect.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which is loaded by a processor to execute the image processing method as described in the first aspect.

[0042] Beneficial effects:

[0043] The image processing method provided in this application involves: acquiring a target rendered image to be processed; if the target rendered image is a keyframe, sending the target rendered image to a target terminal; if the target rendered image is an intermediate frame, dividing the target rendered image into multiple blocks; encoding at least one target block to obtain encoding results corresponding to each target block; and sending at least one encoding result to the target terminal. The at least one target block is at least one block that has changed among the multiple blocks. The target terminal is used to determine the rendered image to be displayed based on at least one encoding result and the currently displayed rendered image.

[0044] By directly sending keyframes to the target terminal, the system ensures that the terminal obtains complete and accurate basic image information, providing a reliable reference for subsequent updates and synthesis based on intermediate frames. This avoids display errors or incompleteness caused by missing key information. Furthermore, by dividing intermediate frames into multiple blocks and encoding and transmitting only at least one target block that has changed, the amount of data that needs to be transmitted is reduced, thus saving bandwidth resources. In scenarios such as mobile networks, 4G / 5G handover in vehicles, or weak signals, data transmission is achieved with less data, reducing the probability of interaction latency, screen tearing, or input stuttering, thereby improving the user experience. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of an image processing system provided in an embodiment of this application;

[0048] Figure 3 This application provides a schematic diagram of the structure of an image processing device according to an embodiment;

[0049] Figure 4 This is a schematic diagram of the structure of a control device provided in an embodiment of this application. Detailed Implementation

[0050] As described earlier, under the vehicle-cloud integrated architecture, the virtual desktop system plays a crucial supporting role in scenarios such as in-vehicle entertainment, office work, navigation, and remote operation and maintenance. For example, passengers can watch high-definition videos and perform office tasks through the virtual desktop system in the car; drivers can use it for precise navigation and remote vehicle operation and maintenance.

[0051] Most existing vehicle-to-everything (V2X) desktop systems rely on centralized cloud rendering and image transmission, i.e., a "full-cloud rendering" mode. In high-bandwidth, low-latency network environments, this mode can achieve good image quality and smooth interaction, allowing users to watch videos and perform operations smoothly. However, in scenarios such as mobile networks, in-vehicle 4G / 5G switching, or weak signal conditions, the following problems arise:

[0052] Interaction delay: After a user performs an action, the system cannot respond immediately, requiring a considerable wait before the result is displayed. For example, when a user touches the screen to switch pages, the page does not switch instantly but requires several seconds or even longer.

[0053] Screen tearing: A phenomenon where the image becomes discontinuous or split during display, affecting the visual experience. For example, when watching a video, the top and bottom parts of the image may suddenly appear misaligned.

[0054] Input lag: The user's input operation is not smooth, and there are pauses or lags. For example, when the user is typing text, the keyboard is slow to respond, and the text cannot be displayed in time.

[0055] Based on this, this application provides an image processing method and related products. The method includes: acquiring a target rendered image to be processed; if the image type of the target rendered image is a keyframe, sending the target rendered image to a target terminal; if the image type of the target rendered image is an intermediate frame, dividing the target rendered image into multiple blocks; encoding at least one target block to obtain encoding results corresponding to each target block; and sending at least one encoding result to the target terminal. Wherein, at least one target block is at least one block that has changed among the multiple blocks; the target terminal is used to determine the rendered image to be displayed based on at least one encoding result and the currently displayed rendered image.

[0056] By directly sending keyframes to the target terminal, the system ensures that the terminal obtains complete and accurate basic image information, providing a reliable reference for subsequent updates and synthesis based on intermediate frames. This avoids display errors or incompleteness caused by missing key information. Furthermore, by dividing intermediate frames into multiple blocks and encoding and transmitting only at least one target block that has changed, the amount of data that needs to be transmitted is reduced, thus saving bandwidth resources. In scenarios such as mobile networks, 4G / 5G handover in vehicles, or weak signals, data transmission is achieved with less data, reducing the probability of interaction latency, screen tearing, or input stuttering, thereby improving the user experience.

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0058] The collection and processing of relevant data (including but not limited to experimental data, test data, simulation data, user data, etc.) involved in this application shall strictly comply with the requirements of national laws and regulations when applied in the following embodiments, obtain the informed consent or separate consent of the subject obtaining the data information, and carry out data use and processing within the scope of laws and regulations and the authorization of the subject.

[0059] See Figure 1 The figure is a flowchart illustrating an image processing method provided in an embodiment of this application.

[0060] Combination Figure 1 As shown, the image processing method provided in this application embodiment can be applied to the cloud, including:

[0061] S11: Get the target image to be processed.

[0062] The target rendering image refers to a specific image that needs to be processed. It can be an image that is rendered in the cloud and has specific content and format.

[0063] S12: If the target rendered image is a keyframe, then send the target rendered image to the target terminal.

[0064] A keyframe is a frame in an image sequence that contains complete information and can independently represent a complete picture or an important state. Keyframes typically play a crucial supporting role in an image sequence, serving as the foundation for constructing or updating the image display. In this embodiment, keyframes can be acquired at a preset period.

[0065] It should be understood that since keyframes contain complete information, sending them directly can ensure that the target terminal can accurately obtain the key content of the image, providing a basis for subsequent construction or updating of the displayed image based on the keyframes, and avoiding errors or incompleteness in image display due to missing information.

[0066] S13: If the target rendering image is an intermediate frame, then the target rendering image is divided into multiple blocks.

[0067] Intermediate frames refer to frames located between keyframes. They have a certain relationship and change with the preceding and following keyframes, and are usually used to represent the dynamic changes of the image through their differences from the keyframes.

[0068] Segmentation refers to dividing the target rendered image into multiple smaller regions for more refined processing and analysis.

[0069] It should be understood that by segmenting the target rendered image into blocks, each block can be analyzed and processed independently, capturing changes in the image more accurately, improving processing efficiency and targeting, and reducing unnecessary processing and data transmission.

[0070] S14: Encode at least one target block to obtain the encoding result corresponding to each target block.

[0071] Among them, at least one target block is at least one block that has changed among multiple blocks.

[0072] The target block refers to the block that changes in the intermediate frame and is the part that needs to be encoded and processed.

[0073] Encoding refers to the process of converting target chunks into a specific format or code to facilitate data compression, reduce transmission volume, or make storage and processing easier.

[0074] It should be understood that, in the embodiments of this application, by encoding only the changed blocks, encoding the entire image is avoided, which greatly reduces the amount of data and computation required for encoding, improves encoding efficiency, and is also beneficial for subsequent transmission and terminal decoding.

[0075] S15: Send at least one encoded result to the target terminal.

[0076] The target terminal is used to determine the rendered image to be displayed based on at least one encoding result and the currently displayed rendered image. It should be understood that, in this embodiment, the target terminal can efficiently generate a new display image using existing image information and new change information, reducing data transmission while ensuring the continuity and accuracy of the image display.

[0077] In this embodiment, by directly sending keyframes to the target terminal, it is ensured that the terminal can obtain complete and accurate basic screen information, providing a reliable reference for subsequent updates and synthesis based on intermediate frames. This avoids screen display errors or incompleteness caused by missing key information. By dividing the intermediate frames into multiple blocks, only at least one target block that has changed is encoded and transmitted, reducing the amount of data that needs to be transmitted and thus saving bandwidth resources. Therefore, in scenarios such as mobile networks, 4G / 5G switching in vehicles, or weak signals, data transmission is achieved with less data, reducing the probability of interaction latency, screen tearing, or input stuttering, thus improving the user experience.

[0078] Based on the image processing method provided in the above embodiments, in one possible implementation, step S15 may include:

[0079] A1: Score the degree of change in each target block to obtain the target score.

[0080] It should be understood that, in the embodiments of this application, a preset algorithm or rule can be used to quantitatively evaluate the degree of change of each changed target block in order to obtain the corresponding score.

[0081] A2: Sort at least one coding result in descending order based on at least one target score.

[0082] It should be understood that the corresponding encoding results can be sorted according to the size of the target score; among them, the target blocks with higher scores represent a greater degree of change.

[0083] A3: Divide at least one of the sorted encoding results into a first encoding set and a second encoding set.

[0084] It should be understood that the sorted encoding results can be divided into two parts according to certain rules (such as proportion, quantity, etc.), forming the first encoding set and the second encoding set respectively.

[0085] A4: Send the first encoding set to the target terminal, and send the second encoding set to the target terminal after a preset time.

[0086] It should be understood that, in the embodiments of this application, the encoded results with a greater degree of change (the first encoded set) may be sent first, and after a period of time, the encoded results with a smaller degree of change (the second encoded set) may be sent.

[0087] In this embodiment of the application, by sending the encoding results in batches, the different degrees of image change are taken into account. The encoding results with greater changes are sent first, which can enable the target terminal to obtain the main change information of the image more quickly, update the displayed image in a timely manner, and improve the user experience. At the same time, sending in batches also helps to balance the network transmission pressure and avoid network congestion caused by transmitting too much data at once.

[0088] Based on the image processing method provided in the above embodiments, in one possible implementation, step S15 may include: obtaining the real-time bandwidth between the cloud and the target terminal; if the real-time bandwidth is less than a preset bandwidth threshold, sending at least one encoding result to the target terminal according to a preset frame rate, or selectively retransmitting a portion of the encoding results in the at least one encoding result to the target terminal.

[0089] Real-time broadband refers to the actual transmission bandwidth of the network between the cloud and the target terminal, reflecting the network's data transmission capability at the current moment.

[0090] If the real-time bandwidth is less than the preset bandwidth threshold, it indicates that the real-time bandwidth is insufficient. In this case, the encoded results can be sent at a preset lower frame rate to reduce the amount of data transmitted per unit time; or, some encoded results can be selectively retransmitted to ensure that important data can accurately reach the target terminal.

[0091] In one possible implementation of the image processing method provided by the above embodiments, it further includes: determining the target depth map and confidence value of the target rendered image; and sending the target depth map and confidence value to the target terminal.

[0092] A target depth map is an image that represents the distance information of each pixel in the image to the camera, reflecting the depth hierarchy of the image.

[0093] The confidence score is an evaluation metric for the accuracy of a target depth map, used to measure the reliability of depth map data.

[0094] It should be understood that, in the embodiments of this application, the target depth map can provide the target terminal with depth information of the image, which helps the terminal to perform more realistic image rendering and display, such as achieving 3D effects and virtual-real fusion; the confidence value can allow the terminal to understand the reliability of the depth map, and make reasonable decisions based on the confidence value in subsequent processing, thereby improving the quality of image processing and display.

[0095] Based on the image processing method provided in the above embodiments, in one possible implementation, it is combined with... Figure 2As shown in the embodiments of this application, an image processing system is also provided, which consists of four main functional modules: cloud main rendering and differential output module (module A), transmission and jitter buffer module (module B), frame synthesis and output module (module C), and timing synchronization and adaptive control module (module D).

[0096] The Cloud Render & ROI Delta module is responsible for main scene rendering, ROI detection, keyframe output, and auxiliary layer generation. It is the "source of the image" in the image processing system, providing core visual content and depth information.

[0097] The Transport & Jitter Buffer module is responsible for network layer stability management, including rate adaptation, block-level FEC redundancy check, retransmission control, and buffer adjustment. The Transport & Jitter Buffer module can ensure a constant end-to-end latency budget in complex vehicular network environments (such as high-speed driving and signal switching).

[0098] The Frame Composition and Output Module (Depth & Time Aware Compositing) is responsible for performing frame fusion and display output on the vehicle side. It uses depth information and time delay information to smoothly combine cloud frames and locally cached frames. The Frame Composition and Output Module can ensure the continuity of the picture and visual consistency, which is a key link in the user's actual perception of smoothness.

[0099] The Latency Budget & Vehicle-Aware Control module is responsible for end-to-end latency budgeting and policy scheduling. It adjusts system parameters in real time based on factors such as vehicle status (driving / parking), network bandwidth, RTT, and packet loss rate. The Latency Budget & Vehicle-Aware Control module achieves a dynamic balance between performance, image quality, and energy consumption.

[0100] The engineering parameters of the image processing system provided in this application embodiment can be:

[0101] TILE_SZ:128 (weak network) / 256 (normal);

[0102] ROI threshold: ΔL1 > 3.0 or SSIM < 0.95;

[0103] Top-K is automatically calculated by B_hat;

[0104] GOP: 0.5~2s (shorter for driving, longer for parking);

[0105] T_buf: 60ms (baseline), range 50~120ms;

[0106] FEC: k / N = 1 / 10 (normal), 2 / 10 (sudden packet loss);

[0107] Bitrate tiers: 2.5 / 4 / 6 / 10Mbps (matches resolution);

[0108] B-frame count: Driving 0~1; Parking 1~2;

[0109] Late drop-off line: T_buf+15ms (Tile level).

[0110] As an example, the execution flow of an image processing system can be: cloud rendering → differential transmission → vehicle-side fusion display → timing feedback adjustment → cloud reconfiguration.

[0111] It should be noted that the image processing system provided in this application embodiment can implement the image processing method provided in this application embodiment through its four modules. The specific details of each module will be introduced one by one below.

[0112] Module A (Cloud Render & ROI Delta) uses a tile + change detection (ROI) approach, encoding and sending only the tiles that have changed. It can periodically send keyframes (IDR / K frames) and send differential frames (P / B or Delta Tiles) in between.

[0113] It can also output an auxiliary layer simultaneously: each frame or every N frames outputs "depth map D (downsampled) + confidence C + semantic layer S (UI / text / icon mask)".

[0114] As an example, the possible execution steps of module A may include:

[0115] Segmentation and Change Detection: The image is divided into tiles of a fixed size (e.g., 128×128 or 256×256). For each tile, pixel difference, SSIM, or optical flow threshold are used to determine if a change has occurred. If the mean difference of pixel L1 ΔL1 > 3.0 or SSIM < 0.95, a change is determined.

[0116] ROI Selection and Prioritization: Each tile is scored based on the degree of change, with higher scores indicating more significant changes. The top-K tiles are selected for priority transmission based on their scores, with the value of K determined by the bandwidth window. The remaining tiles are either delayed or merged for transmission.

[0117] Keyframe strategy: Keyframes (IDR / K frames) are sent periodically, interspersed with differential frames (P / B or DeltaTiles). Keyframes can be output as full frames or a hybrid output method of "keyframes + a small number of static region reuses" can be used to reduce peak bitrate. At the same time, "depth map D (downsampleable) + confidence C + semantic layer S (UI / text / icon mask)" is output as an auxiliary layer every frame or every N frames.

[0118] Linked to vehicle / network conditions: Adjust parameters based on network conditions and vehicle driving status. When the network is poor (RTT>120ms / throughput<6Mbps), reduce TILE_SZ to 128, increase ROI granularity, and shorten GOP to 0.5-1s; when driving at high speed, disable excessive B-frames, prioritize P-frames, and reduce encoding latency.

[0119] It should be understood that Module A uses a Tile+ROI layered transmission approach, encoding and sending only the changed parts. Combined with depth / confidence / semantic auxiliary layers, this forms a "content-aware" backbone output, significantly reducing bandwidth dependence on the entire frame and minimizing bandwidth pressure and latency associated with full-frame encoded transmission in weak network conditions. Furthermore, by processing only "small-scale changes" caused by localized interactions, it avoids the inefficiency of full-frame encoding, thus improving transmission efficiency.

[0120] Module B (Transport, Jitter Buffer, FEC / ARQ) uses a QUIC / UDP channel with custom congestion / rate adaptation; dynamically adjusts the target jitter buffer T_buf; and simultaneously implements block-level FEC (RS or XOR) and selective retransmission (ARQ).

[0121] As an example, possible execution steps for module B may include:

[0122] Real-time bandwidth estimation: The sliding window throughput B_hat=EWMA(∑bytes / Δt) is used, with a window size of 500~1000ms; at the same time, the packet loss rate p_loss, one-way latency OWD, and RTT jitter are estimated together to comprehensively evaluate the channel health H.

[0123] The target buffer time T_buf is adaptive: initially 60ms; if jitter increases or packet loss rate p_loss increases, it is increased proportionally to 80~120ms; if it stabilizes for 5s and jitter decreases, it is slowly reduced back to 60ms; each adjustment does not exceed ±10ms.

[0124] Adaptive Bitrate (Ladder): Presets different resolution and bitrate tiers, such as {mailto:720p@2.5Mbps,900p@4Mbps,1080p@6Mbps,2K@10Mbps}. The appropriate tier is selected based on real-time bandwidth estimation and channel conditions. If B_hat < 0.8·bitrate_cur or OWD > T_buf, the tier is downgraded; if it remains stable for 10 seconds and B_hat > ​​1.2·bitrate_next, the tier is upgraded. Reduces the number of GOPs and B-frames in weak network conditions.

[0125] FEC+ Selective Retransmission (ARQ): Packets are grouped by tile, and k redundant packets are generated for every N tiles (k=1, N=8-12 is recommended). If a specific tile is lost and the loss is within T_repair≤2·T_buf, a conditional retransmission is performed. The vehicle sends a NACK (rate limiting to avoid amplifying congestion).

[0126] Out-of-order / duplicate handling: Deduplicativity is achieved using frame_id and tile_id, and tiles are sorted by t_send and sequence number before being added to the buffer. Lazy tiles that have expired in the buffer (>T_buf+Δ) are discarded to avoid the undesirable effect of "trading stutter for clarity".

[0127] Linked to vehicle / network conditions: When driving at high speed, increase the T_buf limit to 120ms and disable B-frames; when parked and with good network conditions, tighten the T_buf to 50-60ms, allowing for higher bitrates and longer GOPs.

[0128] It should be understood that Module B, through the adaptive combination of block-level FEC / ARQ and target jitter buffer, can effectively cope with the out-of-order and fluctuations caused by mobile network jitter and packet loss without making predictions, stabilize end-to-end latency and frame continuity, and take into account steady-state bit rate and latency.

[0129] Module C (Depth & Time Aware Compositing) is used to implement depth-aware fusion, time-aware smoothing, and semantic priority.

[0130] Deep perception fusion refers to using the depth map D and confidence level C delivered from the cloud to guide each region to adopt either cloud map or cached map.

[0131] Time-aware smoothing refers to determining transition weights based on "frame arrival hysteresis" and "buffer occupancy".

[0132] Semantic priority means prioritizing clarity and stability in the semantic layer, such as UI / text.

[0133] As an example, module C may include the following steps that could be performed:

[0134] Pixel / block-level weight generation:

[0135] Spatial weights W_s are determined by the depth map D and confidence level C sent from the cloud. For areas with shallow foreground and high confidence, cloud frames are used, while for areas with deep background or low confidence, cache (the previous complete frame or a neighboring tile) is used. It is recommended to use partition-level (tile / sub-tile) weights to avoid expensive pixel-level computations.

[0136] The time weight W_t is determined based on the mapping between "frame arrival delay" t_delay and "buf_occ" (buf occupancy). When the delay is large or the buffer is sparse, the weight of the buffered frames is increased to reduce visual jitter.

[0137] The overall weight is represented as W=clamp(a·W_s+(1-a)·W_t), where a defaults to 0.6.

[0138] Synthesis process: If the current frame tiles are incomplete, first fill them with the corresponding area of ​​the nearest complete frame (or use super-resolution / bilinear interpolation to fill the holes), and then fuse them according to the comprehensive weight W. The UI / text semantic layer S is covered on the top layer, and cloud frames are used first (if missing, cached S is used).

[0139] Edge fidelity and deduplication: Guided filtering or bilinear transition is applied to the tile boundaries to avoid gaps; small-scale motion compensation is applied to fast-moving areas (based on cloud-based mv_hint or vehicle-side decoder MV) before fusion.

[0140] Late frame handling: Tiles that are late due to timeout (> T_buf + Δ) are discarded directly; if the delay is not significant and the quality of the current area is significantly improved, "local hot replacement" (area-level update) is allowed, but the frequency is limited to avoid flickering.

[0141] Linking with vehicle status / network: When driving at high speed, the weight of time is increased (a decreases to 0.4) to prioritize stability; when parked, the weight of space is increased (a increases to 0.7) to pursue clearer cloud details.

[0142] It should be understood that Module C, through spatial (depth / confidence) × temporal (hysteresis / buffering) dual-weight fusion, and the combination of partition-level hot replacement and semantic priority layer, can ensure the continuity and visual consistency of the screen even when cloud frames are sometimes late or incomplete, and can also provide a usable and beautiful cloud desktop experience under weak network conditions.

[0143] Module D (Latency Budget & Vehicle-Aware Control) can set the end-to-end latency budget L_e2e (e.g., 120ms for driving and 160ms for parking) and dynamically allocate it to each stage; it uses RTT, bandwidth, packet loss, jitter, vehicle speed, mode (driving / parking), and power consumption budget as inputs to drive parameter table switching.

[0144] As an example, module D may perform the following possible steps:

[0145] The latency budget allocation sets the end-to-end latency budget to approximately L_enc + L_tx + L_dec + L_comp (e.g., 120ms for driving, 160ms for parking), and dynamically allocates it to encoding (L_enc), transmission (L_tx), decoding (L_dec), and synthesis (L_comp). The initial allocation (driving scenario) is L_enc 25ms, L_tx 60ms, L_dec 15ms, and L_comp 20ms. If L_tx consistently exceeds the limit (measured OWD + buffer usage > 60ms), measures such as reducing resolution / bitrate, shortening GOPs, and reducing B-frames are implemented, while keeping L_comp within 20ms to avoid excessive post-processing on the end-side.

[0146] Vehicle state-driven "Profiles": Different profiles are set based on factors such as vehicle speed, mode (driving / parking), and network conditions (RTT, bandwidth, packet loss, jitter), such as P1 driving (weak network), P2 driving (medium network), and P3 parking (good network). Profile switching conditions combine parameters such as B_hat, RTT, p_loss, and v, and incorporate hysteresis and minimum dwell time (≥5s).

[0147] P1 driving on weak networks: 720p~900p, 2.5~4Mbps, GOP 0.5~1s, B=0~1, T_buf 80~120ms;

[0148] P2 driving network: 900p~1080p, 4~6Mbps, GOP1s, B=1, T_buf70~90ms;

[0149] P3 parking network: 1080p~2K, 6~10Mbps, GOP 1.5~2s, B=2, T_buf 50~70ms.

[0150] Cut-off conditions: Combine B_hat, RTT, p_loss, and v, and add hysteresis and minimum dwell time (≥5s).

[0151] Power consumption and temperature control guardrail: Read the vehicle's power consumption / temperature. If it exceeds the budget, limit the decoding parallelism and turn off edge optimization to prioritize L_e2e and image continuity rather than ultimate clarity.

[0152] Abnormal strategies: When sudden packet loss p_loss>8%, briefly enable FEC ratio k / N to increase to 2 / 10 and downgrade; when RTT jitter>40ms lasts for 3s, increase T_buf by 20ms and shorten GOP.

[0153] Module D provided in this application embodiment uses "delay budget" as its core for closed-loop control, integrating video encoding, transmission, synthesis, and vehicle operating conditions. It can dynamically adjust the parameters of each link according to the actual situation, ensuring that end-to-end latency is controlled. At the same time, through profile switching and hysteresis mechanisms, it realizes automatic strategy migration from "driving → stable, parking → clearing", adapting to different vehicle states and network environments, and providing a stable and high-quality interactive experience.

[0154] Based on the image processing system provided in the above embodiments, the image processing system has at least one of the following advantages:

[0155] Significantly reduce end-to-end latency: Through the collaborative mechanism of cloud-based “ROI differential output” and vehicle-side “layered fusion display”, end-to-end latency can be stably controlled within 120-160ms under fluctuating 4G / 5G networks, which is about 40%-60% lower than the traditional full-frame cloud rendering solution, significantly improving real-time interactive performance.

[0156] Effectively reduces transmission bandwidth usage: Tile segmentation and change detection are used to transmit only the changed areas; combined with depth maps and confidence auxiliary layers, data transmission volume can be reduced by about 50% to 70% in weak network scenarios, significantly reducing network burden while ensuring image quality.

[0157] Enhanced image stability and smoothness: A dual-weighted fusion algorithm of spatial (depth / confidence) and temporal (hysteresis / buffering) is used to achieve frame-level smooth transitions and eliminate tearing and flickering issues; automatic compensation and hot replacement are performed when frames are missing or delayed to ensure image continuity.

[0158] Improve availability in weak network environments: Module B introduces an adaptive jitter buffer and a block-level FEC retransmission mechanism, which can maintain stable playback even in mobile scenarios with a packet loss rate of up to 8%, effectively avoiding the phenomenon of "stuttering for clarity".

[0159] Intelligent adaptation to different vehicle conditions and network status: Module D automatically switches encoding profiles and parameters (such as GOP, T_buf, FEC ratio) through dynamic analysis of vehicle driving status, bandwidth, RTT and other information. It prioritizes stability when driving and prioritizes improving image quality when parked.

[0160] Based on the image processing method provided in the above embodiments, see [link to relevant documentation]. Figure 3 This application also provides a schematic diagram of the structure of an image processing device.

[0161] Combination Figure 3 As shown, the image processing apparatus 30 provided in this application embodiment is applied in the cloud and includes:

[0162] Acquisition unit 31 is used to acquire the target rendering image to be processed;

[0163] The first image processing unit 32 is used to send the target rendered image to the target terminal if the image type of the target rendered image is a keyframe.

[0164] The partitioning unit 33 is used to divide the target rendering image into multiple blocks if the image type of the target rendering image is an intermediate frame;

[0165] Encoding unit 34 is used to encode at least one target block to obtain the encoding result corresponding to each target block; wherein, at least one target block is at least one block that has changed among multiple blocks.

[0166] The second image processing unit 35 is used to send at least one encoding result to the target terminal; wherein the target terminal is used to determine the rendering image to be displayed based on at least one encoding result and the currently displayed rendering image.

[0167] In one possible implementation, the second image processing unit 35 is used for:

[0168] The degree of change in each target segment is scored to obtain the target score;

[0169] Sort at least one coding result in descending order based on at least one target score;

[0170] Divide at least one sorted encoding result into a first encoding set and a second encoding set;

[0171] The first set of codes is sent to the target terminal, and the second set of codes is sent to the target terminal after a preset time.

[0172] In one possible implementation, the second image processing unit 35 is used for:

[0173] Obtain the real-time bandwidth between the cloud and the target terminal;

[0174] If the real-time bandwidth is less than the preset bandwidth threshold, at least one encoded result is sent to the target terminal at the preset frame rate, or, at least one encoded result is selectively retransmitted to the target terminal.

[0175] In one possible implementation, it also includes: a determining unit, used for:

[0176] Determine the target depth map and confidence value of the target rendered image;

[0177] Send the target depth map and confidence value to the target terminal.

[0178] It should be noted that the image processing apparatus provided in this application embodiment has the same beneficial effects as the image processing method provided in the above embodiments, and therefore will not be described again.

[0179] In one possible implementation, see Figure 4 The figure is a schematic diagram of a control device provided in an embodiment of this application.

[0180] The control device may include a memory 411 and a processor 412. For example... Figure 4 As shown, the memory can be random access memory (RAM), flash memory, read-only memory (ROM), EPROM, non-volatile read-only memory (Electronic Programmable ROM), registers, hard disks, removable disks, etc.

[0181] The memory 411 can store computer instructions. When the computer instructions stored in the memory 411 are executed by the processor 412, the processor 412 can be used to execute image processing methods. The memory 411 can also store data.

[0182] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape) or a semiconductor medium (e.g., solid-state disk (SSD)).

[0183] This application also provides a readable storage medium for storing the methods provided in the above embodiments. Examples include random access memory (RAM), flash memory, read-only memory (ROM), EPROM, non-volatile read-only memory (EPROM), registers, hard disks, removable disks, or any other form of storage medium in the art.

[0184] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0185] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Regarding the methods disclosed in the embodiments, since they correspond to the product embodiments disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the description of the product embodiments.

[0186] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, The method is applied in the cloud and includes: Get the target image to be processed; If the target rendered image is a keyframe, then the target rendered image is sent to the target terminal; If the target rendered image is an intermediate frame, then the target rendered image is divided into multiple blocks; Encode at least one target block to obtain the encoding result corresponding to each target block; wherein, at least one target block is at least one block that has changed among the multiple blocks; At least one of the encoding results is sent to the target terminal; wherein the target terminal is used to determine the rendering image to be displayed based on at least one of the encoding results and the currently displayed rendering image.

2. The image processing method according to claim 1, characterized in that, Sending at least one of the encoded results to the target terminal includes: The degree of change in each of the target blocks is scored to obtain the target score; Sort at least one of the encoded results in descending order based on at least one of the target scores; Divide at least one of the sorted encoding results into a first encoding set and a second encoding set; The first encoding set is sent to the target terminal, and the second encoding set is sent to the target terminal after a preset time.

3. The image processing method according to claim 1, characterized in that, Sending at least one of the encoded results to the target terminal includes: Obtain the real-time bandwidth between the cloud and the target terminal; If the real-time bandwidth is less than a preset bandwidth threshold, then at least one of the encoded results is sent to the target terminal at a preset frame rate, or, a portion of the encoded results from at least one of the encoded results is selectively retransmitted and sent to the target terminal.

4. The image processing method according to claim 1, characterized in that, The method further includes: Determine the target depth map and confidence value of the target rendered image; The target depth map and the confidence value are sent to the target terminal.

5. An image processing apparatus, characterized in that, The device is applied in the cloud and includes: The acquisition unit is used to acquire the target rendering image to be processed. The first image processing unit is configured to send the target rendered image to the target terminal if the image type of the target rendered image is a keyframe. A partitioning unit is used to divide the target rendered image into multiple blocks if the image type of the target rendered image is an intermediate frame; An encoding unit is used to encode at least one target block to obtain an encoding result corresponding to each target block; wherein, at least one target block is at least one block that has changed among the plurality of blocks; The second image processing unit is configured to send at least one of the encoding results to the target terminal; wherein the target terminal is configured to determine the rendering image to be displayed based on at least one of the encoding results and the currently displayed rendering image.

6. The image processing apparatus according to claim 5, characterized in that, The second image processing unit is used for: The degree of change in each of the target blocks is scored to obtain the target score; Sort at least one of the encoded results in descending order based on at least one of the target scores; Divide at least one of the sorted encoding results into a first encoding set and a second encoding set; The first encoding set is sent to the target terminal, and the second encoding set is sent to the target terminal after a preset time.

7. The image processing apparatus according to claim 5, characterized in that, The second image processing unit is used for: Obtain the real-time bandwidth between the cloud and the target terminal; If the real-time bandwidth is less than a preset bandwidth threshold, then at least one of the encoded results is sent to the target terminal at a preset frame rate, or, a portion of the encoded results from at least one of the encoded results is selectively retransmitted and sent to the target terminal.

8. The image processing apparatus according to claim 5, characterized in that, The device further includes: a determining unit, configured to: Determine the target depth map and confidence value of the target rendered image; The target depth map and the confidence value are sent to the target terminal.

9. A control device, characterized in that, It includes a processor and a memory, the memory being used to store programs, instructions, or code, and the processor being used to execute the programs, instructions, or code in the memory to perform the image processing method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that is loaded by a processor to execute the image processing method as described in any one of claims 1-4.

Images