A vehicle scenario prediction method

Abstract

The present application relates to the technical field of scene prediction, and more particularly to a vehicle scene prediction method, the present application proposes the following scheme, the vehicle terminal collects the vehicle data and the vehicle control data, the cloud end receives and generates the prediction scene video stream, the cabin end receives and carries out color adjustment and image quality evaluation, adjusts the display queue according to the evaluation result and generates the vehicle control data. In the color adjustment process, motion compensation, color distortion processing and secondary local compensation are carried out based on the vehicle control data to improve the color consistency and spatiotemporal coherence of the prediction frame. The image quality evaluation is based on multi-color space feature extraction and nonlinear regression network, and the quality score is output in real time to guide remote control decision. The cloud end uses optical flow prediction combined with self-motion compensation and occlusion repair to generate continuous and high-quality prediction scene video stream, realizing low-delay and high-stability remote vehicle control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of scene prediction, in particular to a vehicle scene prediction method. BACKGROUND

[0002] In the existing remote vehicle control technology, real-time video data is usually collected at the vehicle end, processed through the cloud, and then transmitted to the cabin end for remote driving control. However, due to the inference error of the scene prediction algorithm, especially in complex driving conditions such as acceleration, braking, and steering, local color distortion, brightness drift, and occlusion holes may easily occur in the predicted video stream, resulting in a decrease in the accuracy of the vehicle control data generated by the cabin based on the video stream, affecting the safety and reliability of remote control. The existing technology mainly uses traditional image quality evaluation methods, mainly focusing on structural definition or resolution indicators, which are difficult to sensitively capture the unique color distortion characteristics in the predicted scene, and lack the ability to adaptively compensate for color and motion trends based on vehicle motion characteristics, making it difficult to meet the stringent requirements of the unmanned cabin for continuous and high-quality visual input. Therefore, there is an urgent need for a technical solution that can perform fine color correction on predicted video streams, reliable image quality evaluation, and optimization of prediction generation based on vehicle motion information, to improve the overall stability and safety of the remote control system.

[0003] For example, the Chinese patent application with the application publication number CN118094456A discloses an unmanned driving network performance influence factor prediction model in a city scene, which is an LSTM-based unmanned driving network performance influence factor prediction model, including feature fusion, unmanned driving network performance influence factor establishment, neural network structure building, loss function construction, neural network training, and simulation experiment verification steps. The steps include: step 1 feature fusion step, as the input of the neural network; step 2 unmanned driving network performance influence factor establishment step, as the output of the neural network; step 3 neural network structure building step; step 4 loss function construction step; step 5 neural network training step; step 6 simulation experiment verification step.

[0004] The above existing technologies all have the problem raised in the background: local color distortion, brightness drift, and occlusion holes may easily occur in the predicted video stream, resulting in a decrease in the accuracy of the vehicle control data generated by the cabin based on the video stream. To solve the above problems, the present application designs a vehicle scene prediction method. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a vehicle scene prediction method to solve the problems of the prior art.

[0006] To achieve the above object, the present application provides the following technical scheme:

[0007] A vehicle scene prediction method applied to a cabin end of a remote video transmission platform, the remote video transmission platform further comprising a vehicle end and a cloud end, the vehicle end being configured to transmit vehicle end data, and the cloud end being configured to receive the vehicle end data and generate a predicted scene video stream, the vehicle scene prediction comprising:

[0008] receiving the predicted scene video stream and performing color adjustment on the predicted scene video stream to generate an enhanced video stream, wherein the color adjustment comprises:

[0009] identifying target regions and background regions through semantic segmentation, and generating processed frames and performing color distortion processing in combination with vehicle control data;

[0010] processing the enhanced video stream through a pre-trained image quality evaluation model to output real-time quality scores, wherein the image quality evaluation model is constructed based on multi-color space feature extraction and a nonlinear regression network;

[0011] comparing the real-time quality scores with preset quality thresholds, and generating compensation control signals according to the comparison results to adjust a display queue and generate a continuous display video stream;

[0012] generating vehicle control data according to the display video stream.

[0013] performing color adjustment on the predicted scene video stream to generate an enhanced video stream, comprising:

[0014] performing semantic segmentation on the predicted scene video stream to identify target regions and background regions;

[0015] obtaining vehicle control data of a current frame and historical frames, and adjusting the target regions and the background regions according to the vehicle control data of the current frame and the historical frames to generate first processed frames and second processed frames;

[0016] extracting a color space of the first processed frame, performing color distortion processing on the color space according to vehicle control data of the current frame to generate a corrected frame;

[0017] fusing the corrected frame with the second processed frame according to timestamps to generate an enhanced video stream.

[0018] adjusting the target region and the background region according to vehicle control data of the current frame and historical frames, comprising:

[0019] predicting a pixel displacement vector of the current frame in the historical frames according to vehicle posture parameters in the vehicle control data of the historical frames to obtain a first displacement matrix, wherein the vehicle posture parameters comprise a steering wheel turning angle, a longitudinal speed, and a brake opening degree;

[0020] performing motion estimation on the first displacement matrix according to vehicle control data of the current frame to obtain a second displacement matrix;

[0021] fusing the first displacement matrix and the second displacement matrix to obtain a motion trend matrix;

[0022] performing pixel-level motion compensation on the target region according to the motion trend matrix to generate the first processed frame;

[0023] performing global affine motion compensation on the background region according to the motion trend matrix to generate the second processed frame.

[0024] the color distortion processing on the color space according to vehicle control data of the current frame, comprising:

[0025] determining a motion phase of the current frame according to vehicle control data, wherein the motion phase comprises an acceleration phase, a turning phase, and a braking phase;

[0026] extracting historical target region pixels with color entropy greater than a preset color threshold in a rolling buffer corresponding to the motion phase to generate a phase reference palette, wherein the rolling buffer is set based on a predicted scene video stream corresponding to the historical frames;

[0027] comparing the color space with the phase reference palette channel by channel to generate a color offset measure;

[0028] performing hierarchical compensation on the color space of the first processed frame according to the color offset measure to generate the corrected frame, wherein the hierarchical compensation comprises offset cancellation on full-frame channels and chroma remapping on pixel blocks.

[0029] the fusing the corrected frame with the second processed frame according to timestamps, comprising:

[0030] Spatiotemporally synchronize the correction frame and the second processing frame according to timestamps;

[0031] For each target pixel in the correction frame, construct a neighborhood set at a corresponding position of the second processing frame, wherein a search radius of the neighborhood set is adjusted according to acceleration in the current frame control data;

[0032] In the neighborhood set, jointly sort according to color similarity and spatial distance, select a preset number of neighborhood pixels and extract color offset vectors thereof;

[0033] Calculate a local color compensation coefficient according to the color offset vectors to generate a secondary compensation matrix, and compensate the target pixel according to the secondary compensation matrix until each target pixel is processed.

[0034] The processing of the enhanced video stream by the pre-trained image quality evaluation model comprises:

[0035] Sequentially acquire a to-be-evaluated frame from the enhanced video stream at a preset sampling frequency;

[0036] Perform multi-color space conversion on each to-be-evaluated frame in parallel to obtain RGB, YCbCr, Lab and HSV representations of the to-be-evaluated frame;

[0037] Extract luminance, chrominance component mean, chrominance component variance, color contrast, edge gradient amplitude and local texture energy under each color space, and splice to form a frame-level feature vector;

[0038] Input the frame-level feature vector into an offline-trained nonlinear regression network to obtain a corresponding frame-level quality score;

[0039] Perform temporal exponential smoothing on the frame-level quality score to generate a real-time quality score.

[0040] The generation of a compensation control signal according to the comparison result comprises:

[0041] If the real-time quality score is less than the preset quality threshold, a first compensation control signal is generated, wherein the first compensation control signal marks the current frame as an invalid frame and deletes it from the display queue, and performs motion compensation interpolation between adjacent valid frames to generate a replacement frame;

[0042] If the real-time quality score is greater than or equal to the preset quality threshold, a second compensation control signal is generated, wherein the second compensation control signal inserts the current frame into the display queue as a valid frame;

[0043] Write the first compensation control signal or the second compensation control signal into a display buffer control register at the cabin end to update the display queue.

[0044] A vehicle scene prediction method is applied to the cloud of a remote video transmission platform. The remote video transmission platform further includes an in-vehicle terminal and a cabin terminal. The in-vehicle terminal is used to transmit in-vehicle data. The cloud terminal is used to receive the in-vehicle data from the in-vehicle terminal and generate a predicted scene video stream. The cabin terminal is used to receive the predicted scene video stream from the cloud terminal and perform quality assessment, generating vehicle control data based on the quality assessment results. The vehicle scene prediction includes:

[0045] Receive the vehicle-side data and the vehicle control data of the current frame, preprocess the two consecutive frames of video data in the vehicle-side data, and generate the first preprocessed frame and the second preprocessed frame;

[0046] The preprocessed vehicle-side data and vehicle control data are input into the preset scene prediction model. Based on the temporal characteristics of adjacent video frames and the vehicle motion state represented by the vehicle control information, the pixel-level motion vector is calculated through the optical flow prediction algorithm to generate a predicted scene video stream of multiple frames.

[0047] The predicted scene video stream and the original video stream are simultaneously timestamped, encoded and encrypted, and then sent to the cabin terminal.

[0048] The step of generating a predicted scene video stream of multiple frames by calculating pixel-level motion vectors using an optical flow prediction algorithm based on the temporal characteristics of adjacent video frames and the vehicle motion state represented by vehicle control information includes:

[0049] A multi-scale pyramid is constructed in the first preprocessing frame and the second preprocessing frame. The multi-scale pyramid is traversed by an optical flow algorithm to generate an initial pixel-level motion vector field.

[0050] Based on the vehicle control data of the current frame, calculate the vehicle self-motion transformation matrix, and compensate the initial pixel-level motion vector field based on the vehicle self-motion transformation matrix to generate a motion vector field;

[0051] With constant brightness and constant gradient constraints as objectives, the motion vector field is iteratively optimized by combining total variational regularization to generate a refined vector field.

[0052] The second preprocessed frame is reverse-distorted and resampled according to the refined vector field to generate the first prediction frame;

[0053] Using the first predicted frame as the current frame, optical flow estimation, self-motion compensation, vector optimization, and resampling are performed cyclically until the target number of predicted frames are generated, forming a predicted frame sequence. The predicted frame sequence is then subjected to forward and backward consistency detection frame by frame, occlusion areas are marked, and edge-preserving interpolation and block matching are used to repair and fill occlusion holes, and the predicted scene video stream is output.

[0054] A vehicle scene prediction method is applied to an in-vehicle terminal of a remote video transmission platform. The in-vehicle terminal includes multiple cameras, and the remote video transmission platform further includes a cloud and a cabin terminal. The cloud is used to receive in-vehicle data from the in-vehicle terminal and generate a predicted scene video stream. The cabin terminal is used to receive the predicted scene video stream from the cloud and perform quality assessment, generating vehicle control data based on the quality assessment results. The vehicle scene prediction method includes:

[0055] Send the vehicle-side data collected by the camera and the vehicle control data of the current frame to the cloud;

[0056] Receive vehicle control data from the cabin.

[0057] Compared with the prior art, the beneficial effects of the present invention are:

[0058] This application introduces an image quality assessment method based on vehicle control data, including motion trend compensation, local color distortion processing, and multi-color space feature extraction. This method effectively addresses the image quality degradation issues caused by color shift, occlusion holes, and motion errors in existing remote prediction video streams. Furthermore, it employs acceleration-based adaptive neighborhood search and local secondary color compensation techniques to improve the consistency and spatial naturalness of the predicted frame colors. Attached Figure Description

[0059] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0060] Figure 1 This is a schematic diagram illustrating an exemplary application scenario of an embodiment of the present invention;

[0061] Figure 2 This is a flowchart illustrating a vehicle scene prediction method according to an embodiment of the present invention;

[0062] Figure 3 This is a flowchart illustrating a predictive video enhancement method according to an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of an image quality scoring process according to an embodiment of the present invention;

[0064] Figure 5 This is a flowchart illustrating another vehicle scene prediction method according to an embodiment of the present invention;

[0065] Figure 6 This is a flowchart illustrating another vehicle scene prediction method according to an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0067] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0068] This application provides a vehicle scene prediction method that enables remote control of autonomous driving based on 5G communication. Specifically, the autonomous vehicle needs to transmit high-definition images and videos in real time so that the remote control center or other vehicles can analyze and make decisions.

[0069] Please see Figure 1 This figure is a schematic diagram of an exemplary application scenario provided by an embodiment of this application.

[0070] like Figure 1 As shown, the application scenario consists of the vehicle end, the cloud end, and the cabin end, and the three interact with each other through a transmission network.

[0071] Figure 1 The diagram shows that the vehicle-mounted sensors include image sensors and dynamic sensors, used to acquire raw video streams (i.e., vehicle-mounted data) and vehicle motion state parameters (i.e., vehicle control data, including longitudinal speed, steering angle, brake opening, etc.) of the vehicle driving scene in real time. The vehicle-mounted data and vehicle control data are uploaded to the cloud via a transmission network, serving as the basic input for the cloud to generate predicted scene video streams.

[0072] In one example, the vehicle can also collect vehicle data through other sensors, including but not limited to multi-camera arrays, steering wheel angle sensors, accelerometers, and brake opening sensors.

[0073] Figure 1 The cloud-based workflow is illustrated. First, data preprocessing is performed, including format conversion, resolution adjustment, and format conversion of vehicle control information. Then, the data is analyzed by a scene prediction model deployed in the cloud to predict subsequent scene changes, generate a predicted scene video stream, and upload the predicted scene video stream to the cabin via a transmission network.

[0074] Figure 1The workflow at the cabin end is illustrated. First, video data sent from the cloud is received and decoded. To address the issue of insensitivity to color distortion in the predicted video, which prevents accurate selection of high-quality predicted frames and affects the security of remote control, the predicted video undergoes color adjustment. Then, it is evaluated using an image quality assessment model, the display queue is adjusted, and a continuous display video stream is generated. Finally, vehicle control data is generated based on the display video stream and uploaded to the vehicle end via a transmission network.

[0075] In one example Figure 1 The transmission network shown can be configured using a 5G network to reduce transmission latency and achieve real-time control of autonomous vehicles.

[0076] In one example Figure 1 The shown end-user module also includes a human-assisted image evaluation module, which introduces a human intervention mechanism on top of automatic image quality evaluation. When the real-time quality score output by the image quality evaluation model is lower than a second preset human review threshold, or when multiple consecutive frames of the predicted scene video stream exhibit abnormalities such as color anomalies, image blurring, or motion vector anomalies, the human review process is automatically triggered, prompting the end-user operator to manually confirm. The operator quickly reviews the current frame using a visual interface of the predicted scene video stream, manually confirming whether to accept, discard, or replace the current frame. This further improves the accuracy of image selection in abnormal situations and the security of remote control, building upon automatic processing. The review results can be synchronously recorded to the backend log system for subsequent data training and system optimization.

[0077] Next, with reference to the accompanying drawings, the vehicle scene prediction method provided in the embodiments of this application will be described.

[0078] Please see Figure 2 The figure is a flowchart illustrating a vehicle scene prediction method provided in an embodiment of this application. Figure 2 The method shown can be applied to the cabin side of a remote video transmission platform, which also includes an in-vehicle terminal and a cloud terminal. The in-vehicle terminal is used to transmit in-vehicle data, and the cloud terminal is used to receive the in-vehicle data and generate a predicted scene video stream. Figure 2 The method shown includes the following steps S1-S4, and the specific steps are as follows:

[0079] S1: Receive the predicted scene video stream, perform color adjustment on the predicted scene video stream, and generate an enhanced video stream;

[0080] In this embodiment, after receiving the scene video stream generated by cloud prediction, the cabin first identifies the target and background regions in the video frames using semantic segmentation. Based on vehicle control data from the current and historical frames, motion trend compensation and color distortion correction are performed on each region to generate a continuous and color-enhanced video stream. By combining motion state to derive the color distortion adjustment strategy, the problem of insensitivity to target region color distortion in traditional predictive videos under scene changes can be effectively solved, improving the color accuracy of the predicted frames and the reliability of subsequent decisions.

[0081] S2: Process the enhanced video stream using a pre-trained image quality assessment model and output a real-time quality score;

[0082] In this embodiment, a pre-trained image quality assessment model is used for real-time quality scoring of the enhanced video stream. This model is based on feature extraction in three-dimensional multi-color spaces (RGB, YCbCr, Lab, and HSV) and combines it with a nonlinear regression network to perform frame-level quality prediction. By introducing complex feature dimensions such as brightness, chromaticity, and edge gradients from multiple color spaces, the model significantly improves the ability to perceive predicted video distortion, particularly the detection accuracy of color shifts and local distortions, thus providing a high-confidence reference for subsequent compensation strategies.

[0083] S3: Compare the real-time quality score with the preset quality threshold, generate a compensation control signal based on the comparison result, and generate a continuous display video stream;

[0084] In this embodiment, the real-time quality score is compared with a set quality threshold. When the current frame score is lower than the quality threshold, the frame is marked as invalid and removed from the display queue. A replacement frame is generated based on adjacent valid frames using motion compensation interpolation. If the score is higher than or equal to the quality threshold, the current frame is directly added to the display queue. By dynamically updating the display queue, the final output video stream effectively ensures high stability in terms of color continuity and motion coherence, avoiding visual breaks in remote control caused by single-frame distortion and improving the security of remote control decisions.

[0085] In one example, a compensation control signal is generated based on the comparison results, including:

[0086] If the real-time quality score is less than the preset quality threshold, a first compensation control signal is generated, wherein the first compensation control signal marks the current frame as an invalid frame and deletes it from the display queue, and performs motion compensation interpolation between adjacent valid frames to generate a replacement frame;

[0087] If the real-time quality score is greater than or equal to the preset quality threshold, a second compensation control signal is generated, wherein the second compensation control signal inserts the current frame as a valid frame into the display queue.

[0088] The first compensation control signal or the second compensation control signal is written into the display buffer control register at the cabin end to update the display queue.

[0089] S4: Generate vehicle control data based on the displayed video stream;

[0090] In this embodiment, based on a continuously output high-quality display video stream, key target objects and environmental state changes are extracted, and combined with the time series relationships in the display queue to generate vehicle control data for remote operation. The generated vehicle control data covers commands such as steering, acceleration, and deceleration, and can be transmitted back to the vehicle terminal for execution in real time via the transmission network. By ensuring that the video frames used to generate vehicle control commands have high color consistency and motion coherence, safety hazards caused by image misjudgment during remote control can be further reduced, and the accuracy and robustness of overall driving decisions can be improved.

[0091] While existing technologies have proposed several solutions for optimizing the quality of remotely predicted video streams, such as image enhancement processing based on convolutional neural networks or applying image quality correction algorithms directly at the generation stage, these methods often fail to adequately consider the local color distortion issues caused by rapid scene changes and vehicle movement in dynamic driving environments during predicted scene video streams. Especially in autonomous driving scenarios, color deviations or local distortions in predicted images can easily interfere with the judgment criteria of image quality assessment models, leading to misclassification of low-quality frames as high-quality frames, or vice versa, thus severely impacting the remote control system's selection and use of safety decision frames. Existing single color correction methods generally focus on global adjustments, lacking precise control over motion trends, color drift, and localized distortion, and are unable to dynamically and finely compensate for color distortion and restore realism in changing scenarios.

[0092] This application establishes an adaptive color adjustment process by introducing motion phase determination, rolling buffer extraction of reference samples with high color entropy, dynamic generation of reference color palettes, and a pixel-level hierarchical compensation mechanism. Specifically, by combining vehicle attitude parameters such as steering wheel angle, longitudinal speed, and brake opening, the motion phase of the current prediction frame is first inferred, thereby selecting historical reference color data with optimal spatiotemporal consistency. This avoids directly using the distortion characteristics of the current frame as a benchmark, reducing the system's sensitivity to abnormal frames. Then, a rolling buffer is used to dynamically extract historical target region pixels with high color entropy, constructing a spatiotemporally stable reference color palette, further ensuring the color compensation strategy's adaptability and noise resistance. Finally, in the compensation stage, a hierarchical strategy of global bias cancellation and local pixel block chromaticity remapping accurately corrects color distortion in the predicted image, enabling the subsequent image quality assessment network based on multi-color space feature extraction to obtain more realistic and reliable input features, significantly improving the accuracy of high-quality frame selection. Compared to existing solutions that rely solely on single-space enhancement or static color adjustment, this embodiment not only considers the dynamic correlation between the predicted frame and the vehicle's motion state, but also integrates spatiotemporal stability and pixel-level flexible correction during local compensation. This deeply couples color distortion suppression with the image quality assessment process, improving the overall system's adaptability to complex driving scenarios. For example, under conditions of high-speed scene switching, continuous steering, or strong lighting changes, this embodiment can effectively stabilize the color characteristics of the predicted image, increasing the success rate of the cabin's selection of continuous high-quality predicted frames and significantly enhancing the decision-making accuracy and driving safety of the remote control system.

[0093] Please see Figure 3 The figure is a flowchart illustrating a predictive video enhancement method provided in an embodiment of this application. Figure 3 The method shown can be applied to step S1 of the aforementioned method, and the specific steps are as follows:

[0094] S1.1: Perform semantic segmentation on the predicted scene video stream to identify the target region and background region;

[0095] Specifically, after scene generation, the predicted scene video stream exhibits significant differences in color distortion sensitivity across different regions due to variations in vehicle movement, camera perspective, and ambient lighting conditions. This is particularly true for target areas such as the road ahead and obstacles, which play a crucial role in subsequent remote control decisions. Therefore, semantic segmentation of the predicted scene video stream is necessary to distinguish key target regions from background areas. This provides accurate region segmentation for subsequent color adjustment and local processing, ensuring targeted color compensation strategies and avoiding ineffective or over-compensated processing.

[0096] In this embodiment, a real-time semantic segmentation model based on a deep convolutional neural network is employed to perform frame-by-frame processing on each received frame of the predicted scene video stream. The semantic segmentation model uses an encoder-decoder structure. The encoder extracts multi-scale semantic features, while the decoder combines a skip connection mechanism for high-precision spatial feature reconstruction, thereby outputting a pixel-level category segmentation map. Based on the segmentation map, the image is divided into target regions (such as roads, pedestrians, vehicles, and traffic signs) and background regions (such as the sky, buildings, and vegetation). Each pixel is classified according to its predicted category, forming target and background masks, which serve as the basis for subsequent adjustment processing.

[0097] S1.2: Obtain vehicle control data of the current frame and historical frames, adjust the target area and background area according to the vehicle control data of the current frame and historical frames, and generate a first processing frame and a second processing frame;

[0098] Specifically, during actual vehicle operation, acceleration, braking, and steering cause rapid changes in the position of objects in the image. Directly processing color based on the current frame can easily lead to inaccurate color matching due to spatial misalignment. Therefore, it is necessary to incorporate historical frame vehicle control data to compensate for motion trends between the target and background regions, correcting the displacement effects caused by vehicle movement, and generating aligned frames to ensure the consistency and accuracy of subsequent color feature extraction.

[0099] In this embodiment, the expected pixel displacement of the current frame in the historical frames is first calculated based on the vehicle posture data of the historical frames, constructing a preliminary first displacement matrix. Then, based on the actual vehicle control data of the current frame, motion estimation correction is performed on the first displacement matrix to generate a second displacement matrix that reflects the actual motion trend. By fusing the first and second displacement matrices, the final motion trend matrix is ​​obtained. Based on this motion trend matrix, pixel-level motion compensation is performed on the target area to correct the target displacement deviation caused by motion, generating a first processed frame; simultaneously, a global affine transformation is applied to the background area for overall motion compensation, generating a second processed frame to ensure the consistency of the overall scene.

[0100] Preferably, to avoid the amplification of compensation errors caused by violent driving actions such as strong acceleration and sharp turns during motion compensation, this embodiment introduces a motion trend correction mechanism based on acceleration thresholds. Specifically, before performing motion trend compensation, the longitudinal and lateral acceleration indices in the current frame of vehicle control data are first extracted and compared with preset dynamic threshold values. When any acceleration index exceeds the set threshold value, the system determines that the current vehicle is in a state of violent motion, such as rapid acceleration, rapid deceleration, or sharp turning, thereby triggering the motion trend correction logic.

[0101] In one example, the correction process includes the following steps: First, determine the main direction of influence based on the type of acceleration exceeding the threshold (longitudinal or lateral); when longitudinal acceleration exceeds the limit, prioritize correcting the displacement vector components in the front-to-back direction; when lateral acceleration exceeds the limit, prioritize correcting the displacement vector components in the left-to-right direction. Then, based on the absolute value of the acceleration, dynamically adjust the vector amplitude in the motion trend matrix in the corresponding direction according to a preset scaling factor. The scaling factor is adaptively set based on the deviation of acceleration; the greater the deviation, the higher the adjustment amplitude, thereby effectively suppressing displacement estimation anomalies caused by violent vehicle movements.

[0102] In one example, to ensure the continuity of motion compensation, this embodiment employs a gradual vector adjustment strategy during the correction process. This involves smoothly stretching or compressing the vectors of the motion trend matrix over several consecutive frames, rather than abruptly changing them, thus avoiding the introduction of new spatial distortions due to excessive compensation changes. Furthermore, an upper limit protection mechanism is set to restrict the vector adjustment amplitude to no more than the maximum allowable threshold, thereby ensuring that even under extreme driving conditions, the compensated first and second processing frames maintain a reasonable spatial alignment.

[0103] In one example, the specific steps for adjusting the target area and the background area are as follows:

[0104] S1.2.1: Based on the vehicle attitude parameters in the historical frame vehicle control data, predict the pixel displacement vector of the current frame in the historical frame to obtain the first displacement matrix, wherein the vehicle attitude parameters include steering wheel angle, longitudinal speed and brake opening.

[0105] In this embodiment, vehicle control data at the corresponding time point of the historical frames is first extracted, including steering wheel angle, longitudinal speed, and brake opening. The steering wheel angle is used to estimate vehicle yaw, the longitudinal speed is used to estimate forward movement distance, and the brake opening reflects the trend of longitudinal speed change. These vehicle attitude parameters are input into a simulation model, which is essentially a vehicle dynamics model. The model calculates the translation and rotation trends of the current frame relative to historical frames, and further maps these trends onto the image plane to obtain the estimated displacement vector of each pixel between the historical frame and the current frame, thus forming the first displacement matrix.

[0106] Furthermore, in the displacement calculation process, to account for the differences in vehicle behavior under different road conditions and driving modes, this embodiment also adjusts the calculation frequency according to the driving mode. When historical frame vehicle control data indicates that the vehicle is in a state of rapid acceleration or sharp turning, it automatically switches to a high dynamic response model and uses a higher-order nonlinear motion calculation to improve the accuracy of displacement prediction and avoid the accumulation of prediction errors caused by using a single linear model.

[0107] S1.2.2: Perform motion estimation on the first displacement matrix based on the current frame vehicle control data to obtain the second displacement matrix;

[0108] Specifically, since historical vehicle control data can only reflect the past driving status of vehicles, and vehicle motion has real-time changing characteristics, relying solely on historical data for prediction will result in a lag, leading to deviations in displacement estimation. To correct this deviation, it is necessary to combine the vehicle control data of the current frame to perform real-time motion estimation and updates on the initially generated first displacement matrix, thereby improving the accuracy of displacement prediction.

[0109] In this embodiment, real-time vehicle control data at the corresponding time point of the current frame is extracted, including steering wheel angle, longitudinal speed, and brake opening. By comparing the current vehicle control data with historical vehicle control data, the changes in vehicle motion trends, such as changes in yaw rate and acceleration / deceleration trends, are calculated. Based on these changes, the displacement vector of each pixel in the first displacement matrix is ​​adjusted, refining its offset direction and magnitude, thereby generating a second displacement matrix that better reflects the actual state of the current vehicle motion and the impact of real-time vehicle motion on image displacement.

[0110] S1.2.3: Merge the first displacement matrix and the second displacement matrix to obtain the motion trend matrix;

[0111] Specifically, the first displacement matrix calculated from historical frames and the second displacement matrix corrected based on current vehicle control data represent the predicted trend and the real-time correction trend, respectively. If used independently, they are prone to under-prediction or over-correction. To balance historical continuity and real-time responsiveness, they need to be fused together to integrate prediction and real-time changes, forming a unified motion trend matrix that reflects the actual movement trend of the vehicle.

[0112] In this embodiment, by setting fusion weights, the proportions of the first and second displacement matrices are dynamically adjusted according to the vehicle's motion state. For example, when the vehicle is moving smoothly, the weight of the historical prediction component is increased to ensure the continuity of image displacement compensation; while during violent motion, the weight of the real-time correction component is increased to enhance the real-time performance of the compensation. During the fusion process, a pixel-level weighted average is used to synthesize the corresponding displacement vectors of the two matrices to ensure that the motion trend matrix has physical continuity both locally and globally.

[0113] Preferably, in the fusion stage, to prevent motion artifacts from being introduced after fusion, this embodiment also includes a local consistency detection mechanism, that is, to verify the consistency of displacement direction before and after fusion in each local region. When the local inconsistency exceeds the threshold, the region weight redistribution is automatically triggered to enhance local coherence and improve the overall spatial coordination of the motion trend matrix.

[0114] S1.2.4: Perform pixel-level motion compensation on the target region according to the motion trend matrix to generate the first processing frame;

[0115] Specifically, the target area typically includes important elements such as the road ahead, vehicles, and pedestrians, and its positional accuracy directly affects subsequent color adjustments and the generation of remote control commands. To ensure the spatial consistency of the target area in the predicted video stream, pixel-level motion compensation needs to be performed on the target area based on the motion trend matrix to correctly map the target area from historical frames to the current frame.

[0116] In this embodiment, for each pixel within the target region mask, pixel-level reverse mapping is performed based on the displacement vector in the motion trend matrix. This involves migrating the color value of the corresponding pixel in the historical frame to the target pixel position in the current frame using the compensation vector, generating the aligned first processed frame. Bilinear interpolation is employed to mitigate image tearing and jagged edges caused by discrete displacement, improving the smoothness and naturalness of the compensated image.

[0117] S1.2.5: Perform global affine motion compensation on the background region based on the motion trend matrix to generate a second processing frame;

[0118] Specifically, the background region is less sensitive to displacement accuracy compared to the target region and typically exhibits large-area continuous features. To reduce computational complexity while ensuring reasonable compensation, a holistic affine motion compensation method can be used for the background region, avoiding the high overhead of pixel-by-pixel calculations while improving background consistency and stability.

[0119] In this embodiment, a global affine transformation matrix is ​​estimated based on the overall displacement distribution characteristics of background region pixels in the motion trend matrix. This affine transformation matrix is ​​calculated using a least-squares fitting method based on the relationship between the initial and expected positions of the background pixels, effectively approximating the large-scale translation, rotation, and scale changes of the background region. Subsequently, a global mapping is performed on the background region based on the affine transformation matrix, and resampling is performed to obtain the second processing frame.

[0120] S1.3: Extract the color space of the first processing frame, and perform color distortion processing on the color space according to the vehicle control data of the current frame to generate a correction frame;

[0121] Specifically, the motion phase (acceleration, steering, braking) of the current frame is first determined by comprehensively considering steering wheel angle, longitudinal speed, and braking opening to identify the dominant trend of color drift. Then, historical target region pixel samples with color entropy exceeding a set color threshold are extracted from the rolling buffer corresponding to the motion phase to construct a reference color palette with strong spatiotemporal correlation. Multi-channel deviation comparison is performed on the extracted color space of the first processed frame, calculating the offset of the current color component from the reference color palette channel by channel to generate a color shift metric. Based on the color shift metric, a hierarchical compensation strategy is executed: first, offset cancellation is performed across all frames to correct global color drift; then, local chromaticity remapping is performed on a pixel-by-pixel basis to restore local color detail and realism, resulting in a precisely corrected frame.

[0122] In one example, the specific steps for color distortion processing of a color space are as follows:

[0123] S1.3.1: Determine the motion phase based on the vehicle control data of the current frame, wherein the motion phase includes the acceleration phase, the steering phase and the braking phase;

[0124] Specifically, the color distortion in the predicted scene video stream is closely related to the vehicle's current motion state. Under different motion states, the distribution patterns of image color shift and brightness changes differ significantly. Therefore, to improve the targeting of subsequent color distortion processing, it is necessary to first determine the motion phase corresponding to the current frame and formulate an appropriate color processing strategy based on the motion phase differences, thereby more accurately controlling the distortion compensation process.

[0125] In this embodiment, based on the vehicle control data attached to the current frame, including the longitudinal speed change rate, steering wheel angle change rate, and brake opening change rate, which correspond to the characteristic quantities of acceleration, steering, and braking states respectively, the main motion characteristics of the current frame are determined by setting acceleration thresholds, steering angle change thresholds, and brake opening thresholds. When the longitudinal speed change rate exceeds the acceleration threshold and the steering wheel angle change rate is less than the steering angle change threshold, it is determined to be an acceleration phase; when the steering wheel angle change rate exceeds the steering angle change threshold, it is determined to be a steering phase; and when the brake opening change rate exceeds the brake opening threshold, it is determined to be a braking phase.

[0126] As a preferred embodiment, to avoid the problem of unstable motion phase determination under continuous and rapid changing conditions, this embodiment uses a short-time smoothing mechanism to determine the motion phase within three consecutive frames using the principle of dominant motion feature dominance. That is, the motion feature that occupies the most frames is used as the current motion phase, which improves the continuity of motion determination and prevents the color processing process from causing processing oscillations due to frequent phase switching.

[0127] S1.3.2: In the rolling buffer corresponding to the motion phase, extract historical target region pixels with color entropy greater than a preset color threshold to generate a phase reference color palette, wherein the rolling buffer is set based on the predicted scene video stream corresponding to the historical frame;

[0128] Specifically, relying solely on the current frame for color compensation presents significant uncertainties, especially in autonomous driving scenarios. Due to prediction errors, localized occlusion caused by vehicle movement, and environmental changes, the color characteristics of the current frame may shift significantly, affecting the accuracy and stability of subsequent color correction. Therefore, to obtain a more reliable color reference, it is necessary to go back in time, extract stable color features from historical data, and establish a spatiotemporally continuous color benchmark. Since the vehicle only receives the predicted scene video stream transmitted from the cloud, and to minimize transmission latency, the cloud does not send the original video data from the vehicle, this embodiment preferably uses historical predicted scene video streams as the data source. These historical predicted scene video streams are generated based on the vehicle's actual motion state and environmental changes. They are highly correlated with the current frame in terms of temporal continuity and motion consistency, effectively supporting the spatiotemporal consistency requirements during color distortion correction, thus reasonably replacing the original frame as the color reference source.

[0129] In this embodiment, independent rolling buffer modules are established for different vehicle motion phases (acceleration, steering, braking). Each buffer module is responsible for storing pixel information of the target region in several prediction frames received within a limited time window under the corresponding motion phase. To select representative and stable pixel samples, this embodiment performs color entropy calculation on the target region image blocks in the rolling buffer. By evaluating the color richness and texture complexity of the local region, the color expression capability of the pixel block is determined. A color threshold is set, and pixel blocks with entropy values ​​higher than the color threshold are marked as valid samples. Their main color components are extracted to construct a phase reference color palette. Pixel blocks with high color entropy are often located in areas with rich color changes and obvious lighting details. Their color characteristics maintain good consistency during motion, and therefore can be used as a reliable basis for compensation.

[0130] Furthermore, to prevent pixel samples in the cache from losing their representativeness due to excessive accumulation over time, this embodiment introduces a dynamic time window management mechanism in the rolling cache, retaining only high color entropy valid pixels within the most recent set time period (e.g., 1 second or 2 seconds). Expired samples are automatically discarded after the time window expires, ensuring the color palette can reflect the true color distribution characteristics under the current motion environment in real time. Simultaneously, by combining motion phase classification, color reference confusion caused by changes in vehicle motion state is avoided, further improving the targeting and robustness of color distortion processing.

[0131] S1.3.3: Compare the color space with the phase reference palette channel by channel to generate a color offset metric;

[0132] Specifically, in order to quantify the degree of color distortion in the target area of ​​the current frame, it is necessary to compare the color characteristics of the current frame with the historical reference colors channel by channel. By calculating the deviation, the variation trend of different color components (such as R, G, B, or L, a, b in Lab space) in spatial distribution can be accurately described, providing a numerical basis for subsequent compensation.

[0133] In this embodiment, the first processing frame is mapped to a unified color space (preferably Lab space to improve the decoupling between luminance and color channels). Then, statistical analysis is performed on the mean, variance, and extreme value differences between the L, a, and b channels and the corresponding channels in the phase reference palette. Based on the statistical results, the offset metric for each channel is calculated, mainly characterized by the difference between the mean of the current frame channel and the mean of the reference channel, as well as the rate of change of variance.

[0134] S1.3.4: Perform hierarchical compensation on the color space of the first processed frame according to the color offset metric to generate a corrected frame, wherein the hierarchical compensation includes offset cancellation on the entire frame channel and chroma remapping on the pixel block.

[0135] Specifically, color distortion is usually manifested as a combination of overall drift and local color distortion. Therefore, a hierarchical compensation strategy is needed. First, the offset of the global channel is corrected to unify the overall color reference of the frame. Then, the local areas are carefully corrected to achieve fine-grained correction and avoid new local distortion problems caused by simple global adjustments.

[0136] In this embodiment, firstly, based on the color offset measurement results, a uniform bias cancellation is performed on each color channel of the first processing frame. That is, the overall brightness or chromaticity of each channel is shifted towards the reference direction according to the offset direction to cancel the overall color drift. After the global bias is completed, the frame is divided into several small pixel blocks (preferably 16×16 or 32×32 in size). Local color statistics are performed again on the pixels in each block and compared with the local samples of the reference color palette. If the local offset exceeds the fine-grained threshold, chromaticity remapping is performed on the pixel block. The remapping operation is completed through a local color space transformation matrix to restore the color distribution in the region.

[0137] S1.4: The corrected frame is fused with the second processed frame according to the timestamp to generate an enhanced video stream;

[0138] It should be noted that both the correction frame and the second processing frame here correspond to single-frame images. The enhanced video stream is generated by summarizing and sorting the single-frame images over time.

[0139] Specifically, the correction frame and the second processing frame represent the target area and the background area after color correction, respectively. Due to the asynchronous processing of motion compensation and processing strategies, direct superposition may lead to visual inconsistencies such as edge misalignment and color abrupt changes. Therefore, strict spatiotemporal synchronization processing based on timestamps is required, and seamless stitching is achieved through a neighborhood fusion strategy to ensure the integrity and consistency of the enhanced video stream.

[0140] In one example, the calibration frame and the second processing frame are first aligned based on their timestamps to ensure consistent spatial reference at the same time. Then, for each target pixel in the calibration frame, a neighborhood set is established at the corresponding position in the second processing frame. The neighborhood radius is dynamically adjusted based on the acceleration magnitude in the vehicle control data to adapt to local differences under varying motion intensities. The neighborhood set is jointly sorted by color similarity and spatial distance, and color offset vectors are extracted from several optimal neighborhood pixels to generate a local compensation coefficient matrix. This local compensation coefficient matrix is ​​applied to the target pixels in the calibration frame to achieve secondary local color compensation and boundary transition processing, ultimately completing the fusion of the calibration frame and the second processing frame, outputting a continuous and natural enhanced video stream. The specific steps for fusing the calibration frame and the second processing frame are as follows:

[0141] S1.4.1: Perform spatiotemporal synchronization between the correction frame and the second processing frame based on the timestamp;

[0142] Specifically, the correction frame and the second processing frame originate from different processing paths and are generated after independent compensation for the target region and background region, respectively. Due to the difference in their processing timing, direct fusion can easily lead to spatial misalignment or temporal asynchrony. To ensure that the pixel positions and timelines of each frame correspond precisely during fusion, time-stamp-based spatiotemporal synchronization processing needs to be performed before fusion.

[0143] In this embodiment, for the correction frame and the second processing frame, their respective generation timestamps are first extracted. The timestamps of the two frames are compared, and based on a set maximum allowable time deviation threshold, the pair of frames that are closest in time are selected as synchronization candidate frames. If a small time offset exists, this embodiment preferably uses a bidirectional optical flow compensation method to perform subtle pixel-level time interpolation correction on the earlier frame, making the two frames as aligned as possible on the time axis. This ensures that after spatial synchronization, the fusion region will not experience jumps due to time misalignment.

[0144] S1.4.2: For each target pixel in the correction frame, a neighborhood set is constructed at the corresponding position in the second processing frame, wherein the search radius of the neighborhood set is adjusted according to the acceleration in the vehicle control data of the current frame;

[0145] Specifically, as the vehicle's acceleration changes drastically during motion, the pixel displacement of objects in the scene also changes accordingly, resulting in varying degrees of spatial offset between the target area and the background area. Using a fixed-radius neighborhood search might fail to cover truly relevant background pixels, affecting the accuracy of subsequent color compensation. Therefore, it is necessary to dynamically adjust the neighborhood search radius based on the vehicle's acceleration in the current frame. This ensures that the neighborhood set can adapt to the large displacements caused by drastic motion while controlling the search range during stable driving, thereby improving processing accuracy and efficiency.

[0146] In this embodiment, for each pixel marked as the target region in the correction frame, its preliminary corresponding position in the second processing frame is first determined. Based on the longitudinal and lateral accelerations in the current frame's vehicle control data, these are compared with preset low-speed, medium-speed, and high-speed acceleration thresholds, respectively. The search radius is then set according to the different speed levels. A larger acceleration results in a larger search radius to capture potential offsets caused by violent motion; conversely, a smaller radius is used to improve local search accuracy. The neighborhood set includes all valid background pixels within the search radius and their color attribute data.

[0147] Preferably, in this embodiment, the motion intensity reference value is compared with a set multi-level acceleration threshold, and a specific neighborhood search radius is set according to different acceleration intervals, as follows:

[0148] When the exercise intensity is less than 1m / s 2 (During low-speed, stable driving) the search radius is set to 3 pixels.

[0149] When the motion intensity is greater than or equal to 1 m / s 2 And less than 3m / s 2 When in between (medium-speed cruising or slight turning), the search radius is set to 5 pixels;

[0150] When the motion intensity is greater than or equal to 3 m / s 2 And less than or equal to 6 m / s 2 During periods of rapid acceleration or moderate-intensity steering, the search radius is set to 7 pixels.

[0151] When the exercise intensity is greater than 6m / s 2 During rapid acceleration, deceleration, or sharp turns, the search radius is set to 10 pixels.

[0152] The motion intensity is calculated based on the longitudinal acceleration and the lateral acceleration.

[0153] In this example, a neighborhood radius protection mechanism is set, which means that no matter how the acceleration changes, the upper limit of the search radius is limited to 10 pixels and the lower limit is limited to 3 pixels. This ensures that the neighborhood set is kept within a reasonable range, which can cover the displacement changes caused by motion without introducing too many irrelevant pixels, thus ensuring the local accuracy of color compensation.

[0154] In addition, when dynamically switching the search radius, in order to avoid the discontinuity of the neighborhood caused by the radius jump, a smooth radius update method can be adopted. That is, when the acceleration change exceeds the preset threshold between adjacent frames, the search radius is gradually adjusted according to the maximum growth or shrinkage rate of 2 pixels per frame, so as to smoothly transition the neighborhood size and ensure the temporal stability and search consistency of the neighborhood set structure.

[0155] S1.4.3: In the neighborhood set, sort the neighborhood pixels jointly based on color similarity and spatial distance, select a preset number of neighborhood pixels and extract their color offset vectors;

[0156] Specifically, when multiple candidate pixels exist within a neighborhood set, directly using all pixels for color compensation can easily introduce noise, especially in edge or dynamic regions, where interference from low-quality pixels can amplify color errors. Therefore, it is necessary to filter and sort the pixels within the neighborhood based on color characteristics and spatial relationships, prioritizing pixels with similar color characteristics and close proximity to ensure that the subsequently calculated color offset vector is more accurate and improves the compensation effect.

[0157] In this embodiment, for each target pixel, the Euclidean distance between it and all pixels in its neighborhood set in the color space (preferably Lab color space) is calculated as a color similarity metric, and the spatial distance is also calculated as a positional similarity metric. The color similarity and spatial distance are weighted and fused using a set weighting coefficient to obtain a comprehensive ranking score. The pixels are then sorted in ascending order according to their comprehensive ranking scores, and a predetermined number of the top-ranked optimal neighborhood pixels are selected as valid reference pixels.

[0158] Preferably, to ensure the stability of color compensation, extremely abnormal pixel values ​​can be filtered out using a color anomaly removal mechanism. This involves removing neighboring pixels whose color distance exceeds twice the standard deviation of the local mean, thus preventing extreme outliers from affecting the local color estimation results. Simultaneously, a color offset vector is calculated for the finally selected neighboring pixels—the difference vector between the target pixel's current color value and the average color value of its neighboring pixels—serving as the basis for subsequent local compensation.

[0159] S1.4.4: Calculate the local color compensation coefficients based on the color offset vector to generate a secondary compensation matrix, and compensate the target pixels based on the secondary compensation matrix until each target pixel has been processed;

[0160] In this embodiment, based on the color offset vector extracted in the previous stage, a local color compensation coefficient is calculated for each target pixel. This coefficient is determined by the color channel offset and a weighted averaging strategy. Specifically, based on the difference between the average color value of neighboring pixels and the current pixel's color value, the color values ​​in each color channel (e.g., L, a, and b channels in Lab color space) are adjusted by a certain scaling factor to generate a local quadratic compensation matrix. This compensation matrix is ​​applied to the current target pixel to achieve fine-grained adjustment and correction of the color value. The specific scaling factor can be determined by those skilled in the art through extensive experimentation.

[0161] Please see Figure 4 This figure illustrates an image quality scoring process provided in an embodiment of this application. Specifically, in remote predictive control scenarios, due to spatiotemporal inference, motion compensation, and rendering errors during the generation of the predicted scene video stream in the cloud, the most common type of image distortion is not traditional problems such as structural blurring or resolution degradation, but rather visual anomalies primarily characterized by color distortion. This color distortion manifests as local tone shifts, inconsistent brightness, abnormal saturation, color gamut compression, or color gamut expansion. Because the generation mechanism of the predicted scene differs from that of a camera directly capturing real images, color rendering is constrained by model calculation errors and compensation errors, resulting in distortion patterns that are localized, asymmetric, and unpredictable. The color shift in the predicted image is particularly pronounced during vehicle acceleration, steering, or complex lighting changes. Traditional quality assessment methods based on texture sharpness, structural similarity (such as SSIM), or resolution measurements cannot sensitively capture such subtle and localized color deviations, thus failing to support the requirements for high-reliability and high-precision predicted frame selection. In this embodiment, based on an in-depth analysis of the distortion mechanism in the predicted scene video stream, an image quality assessment strategy dominated by color characteristics is proposed. Specifically, this method extracts composite features from multiple color spaces, including the mean, variance, contrast, edge gradient, and local texture energy, to fully perceive color changes and distortion patterns, constructing frame-level feature vectors. Quality regression modeling is then performed based on these vectors. By focusing on local anomalies and global shifts in the color dimension, it not only accurately captures subtle distortions in the predicted image but also improves the ability to identify anomalous predicted frames under complex scene changes, ensuring the selection of high-quality predicted frames that truly meet driving safety requirements. This method, with color as its core feature, adapts to typical distortion patterns arising from the prediction generation mechanism, is highly targeted, and provides reliable evaluation results, significantly outperforming traditional evaluation methods that primarily rely on resolution or structural stability. Figure 4 The method shown can be applied to step S2 of the aforementioned method, and the specific steps are as follows:

[0162] S2.1: Sequentially acquire the frames to be evaluated from the enhanced video stream at a preset sampling frequency;

[0163] Specifically, since the enhanced video stream is continuously generated, it is affected by factors such as bandwidth fluctuations, latency, and prediction errors during transmission and processing. Performing quality assessment on every frame would impose an enormous computational load, and real-time performance would be difficult to guarantee. Therefore, it is necessary to selectively extract a subset of frames for evaluation at a reasonable sampling frequency. This ensures overall image quality continuity while balancing processing efficiency and real-time system response speed, meeting the low-latency and high-reliability requirements of remote vehicle control.

[0164] In this embodiment, the sampling frequency is set to a certain multiple of the fixed frame rate, for example, 1 frame is sampled every 5 frames. The specific frequency can be dynamically adjusted according to the transmission link bandwidth and the processing capability of the terminal. The sequentially acquired frames to be evaluated are arranged according to their timestamps to ensure that the samples cover the entire video stream, without concentrated sampling or omissions, forming a set of frames to be evaluated that are representative of the time, ensuring that subsequent quality assessments effectively reflect the overall video quality change trend.

[0165] S2.2: Perform multi-color space conversion in parallel for each frame to be evaluated to obtain the RGB, YCbCr, Lab and HSV representations of the frame to be evaluated;

[0166] Specifically, in vehicle prediction scenarios, a single color space (such as RGB only) is insufficient to fully capture the color distortion features in complex scenes. Since predicted images often exhibit saturation anomalies, brightness shifts, or color shifts in local areas, and different types of distortion show different feature sensitivities in different color spaces, it is necessary to extract multiple color space representations in parallel to enhance the model's ability to perceive multiple types of distortion, thereby improving the comprehensiveness and accuracy of image quality assessment.

[0167] In this embodiment, for each frame to be evaluated, the original RGB representation is first preserved, and then simultaneously converted to the YCbCr color space to separate luminance and chromaticity components, to the Lab color space to achieve perceptual uniformity mapping, and to the HSV color space to enhance the response to changes in hue and saturation. Each color space conversion uses a standard high-precision conversion formula to ensure the consistency and physical interpretability of color components across different spaces. Parallel conversion employs asynchronous multi-threaded processing to maximize the utilization of onboard multi-core computing resources and ensure that no additional latency is introduced.

[0168] S2.3: Extract the brightness, chromaticity component mean, chromaticity component variance, color contrast, edge gradient magnitude, and local texture energy in each color space, and concatenate them to form a frame-level feature vector;

[0169] Specifically, image quality is affected by many factors, and a single luminance or chromaticity index cannot fully describe the distortion phenomena that may occur in the predicted image. Especially in complex driving scenarios, local detail blurring, color shift, and loss of edge information are very likely to occur. Therefore, it is necessary to comprehensively extract luminance, chromaticity, and texture features in multiple color spaces to construct high-dimensional frame-level feature vectors and fully capture global and local quality change information of the image.

[0170] In this embodiment, the following features are extracted for each color space: RGB, YCbCr, Lab, and HSV: the full-frame mean and variance of the luminance and chrominance channels (reflecting overall brightness and color stability), color contrast (measuring the breadth of color gamut distribution), edge gradient magnitude (measuring image sharpness and detail preservation), and local texture energy (reflecting local structural complexity). Feature extraction employs a combination of local window scanning and global statistics to ensure that both global trends and local anomaly detection are taken into account.

[0171] S2.4: Input the frame-level feature vector into the offline-trained nonlinear regression network to obtain the corresponding frame-level quality score;

[0172] Specifically, in vehicle remote control scenarios, the quality changes of the predicted scene video stream are complex and nonlinear. Relying solely on simple linear models is insufficient to accurately model the mapping relationship between multidimensional features and actual visual quality. Therefore, a nonlinear regression network is needed, which, through large-scale offline training, learns the complex mapping relationship between features and human-perceived quality to achieve high-precision frame-level quality prediction.

[0173] In this embodiment, a lightweight multilayer perceptron (MLP) structure is used as the regression network. The input receives the concatenated frame-level feature vectors, and the network contains two or three fully connected hidden layers. The activation function is ReLU, and the output is mapped to a continuous quality score (e.g., the range of 0-1). During the network training phase, a manually annotated video quality score database is used as the supervision signal, and mean squared error loss is employed for optimization to ensure that the trained network can accurately regress the input features.

[0174] Furthermore, to improve the stability and robustness of the model in practical applications, this embodiment introduces data augmentation strategies during training, such as color perturbation, local occlusion simulation, and motion blur processing of training samples, to enhance the model's generalization ability to various types of distortion, thereby ensuring that reliable quality scores can be output in different cabin scenarios.

[0175] S2.5: Perform temporal exponential smoothing on the frame-level quality score to generate a real-time quality score;

[0176] Specifically, single-frame quality scores are significantly affected by local noise and occasional distortion. Directly using the raw scores may lead to excessive quality fluctuations in the continuously predicted video stream, affecting the stability of subsequent display stream control. Therefore, it is necessary to perform temporal smoothing on frame-level quality scores to eliminate short-term noise interference and extract the main trend of quality changes in order to form a stable and reliable real-time quality score.

[0177] In this embodiment, the exponentially weighted moving average (EWMA) method is used to smooth the quality scores of consecutive frames. The quality score of the current frame is a weighted superposition of the smoothed result of the previous frame and the current original score with a fixed weight. The weight factor is preferably set between 0.7 and 0.9 and can be dynamically adjusted according to the actual scenario. This allows the quality score to respond to rapid changes (such as sudden distortion) while effectively suppressing random fluctuations.

[0178] This application also provides another vehicle scene prediction method, which will be described below with reference to the accompanying drawings.

[0179] Please see Figure 5 The figure is a flowchart illustrating another vehicle scene prediction method provided in an embodiment of this application. Figure 5 The method shown can be applied to the cloud of a remote video transmission platform, which also includes a vehicle-mounted terminal and a cabin terminal. The vehicle-mounted terminal is used to transmit vehicle-mounted data. The description of the cabin terminal can be found in the description section above and will not be repeated here. Figure 5 The methods shown include the following A1-A3:

[0180] A1: Receives vehicle-side data and vehicle control data of the current frame, preprocesses two consecutive frames of video data in the vehicle-side data, and generates the first preprocessed frame and the second preprocessed frame.

[0181] In this embodiment, the vehicle-mounted terminal acquires real-time video streams of the environment in front of the vehicle via image sensors, while simultaneously acquiring vehicle attitude information such as longitudinal speed, steering wheel angle, and brake opening. After receiving two consecutive frames of vehicle-mounted video data, the cloud first performs unified resolution standardization processing to ensure consistency in frame data format across different sensors or at different times. Simultaneously, it performs color standard space mapping (e.g., unifying to the sRGB color gamut) to avoid prediction errors caused by differences in color distribution. For vehicle control data, numerical normalization and timestamp alignment are performed to ensure a one-to-one correspondence with video frames. After these processes, the two standardized video frames are used as the first and second preprocessed frames, providing stable and comparable basic input data for subsequent scene prediction based on optical flow calculations.

[0182] A2: Input the preprocessed vehicle-side data and vehicle control data into the preset scene prediction model to generate a multi-frame predicted scene video stream;

[0183] In this embodiment, the standardized first and second preprocessed frames, along with the corresponding vehicle control data, are input into a scene prediction model deployed in the cloud. The scene prediction model, based on optical flow inference, combines the vehicle motion trends (acceleration, steering, braking states) represented by the vehicle control data to perform pixel-level motion vector estimation, constructing a motion correlation field between consecutive frames. Based on the calculated motion vector field, the model performs inverse resampling and pixel interpolation on the input frames, sequentially generating a predicted scene video stream for several subsequent frames. By fusing vehicle control data, the inference bias of traditional optical flow algorithms under rapid movement or complex occlusion conditions can be effectively compensated, significantly improving prediction accuracy and temporal consistency. The predicted video stream generated based on this method can predict vehicle trajectory and changes in the scene ahead in advance in the cloud, providing continuous and stable visual input for remote control at the cabin, significantly reducing the real-time inference burden at the cabin, and improving the overall system response speed and reliability.

[0184] A3: Simultaneously timestamp the predicted scene video stream and the original video stream, encode and encrypt them, and then send them to the cabin terminal;

[0185] In this embodiment, to ensure that the cabin can accurately and synchronously receive and play the predicted scene video stream generated by the cloud, the cloud adds a standardized timestamp tag to each frame after generating the predicted frame sequence, according to a strict timing relationship. This timestamp is synchronously corrected based on the vehicle's original acquisition time and cloud processing delay estimation, ensuring that the playback timing of the predicted frames is highly consistent with the actual movement state of the vehicle. After timestamp tagging, the entire predicted scene video stream undergoes efficient encoding processing (such as H.265 / HEVC encoding format) to reduce transmission bandwidth overhead, and an end-to-end encryption mechanism is adopted to ensure the integrity and security of the data during transmission. Subsequently, the encoded and encrypted predicted scene video stream is sent to the cabin via 5G or other high-speed wireless transmission links. Through synchronous tagging and encrypted transmission, not only is the timing accuracy and continuity of the cabin's video playback guaranteed, but the security of data transmission is also improved, preventing potential security risks caused by data loss or tampering during remote control.

[0186] The specific steps for A2 are as follows:

[0187] A2.1: Construct a multi-scale pyramid in the first preprocessing frame and the second preprocessing frame, and traverse the multi-scale pyramid using an optical flow algorithm to generate an initial pixel-level motion vector field;

[0188] Specifically, when vehicle motion is rapid or scene depth changes significantly, single-scale optical flow estimation is prone to getting stuck in local optima, leading to inaccurate pixel-level motion vectors and affecting the overall spatiotemporal consistency of subsequent prediction frames. Therefore, a multi-scale pyramid structure needs to be introduced to perform motion estimation in a layered and progressive manner, from roughly capturing the overall motion trend at a large scale to meticulously capturing local detail changes at a small scale, thus balancing global consistency with local accuracy.

[0189] In this embodiment, Gaussian pyramids are constructed for the first and second preprocessed frames, respectively, to progressively downsample the original resolution image, forming multiple scale levels. Optical flow estimation starts from the top layer (lowest resolution), using a sparse matching method to initially estimate the overall large-scale motion trend; then, layer by layer downwards, using the optical flow result of the previous layer as initial values, dense matching technology is used to refine the motion vector at higher resolutions. Within each scale level, optical flow estimation comprehensively employs the assumption of constant brightness and local smoothness constraints to reduce noise interference. Finally, after completing all scale level iterations at the lowest layer (original resolution), an initial full-frame pixel-level motion vector field is formed.

[0190] Furthermore, to prevent error accumulation during multi-scale transmission, this embodiment introduces a local anomaly detection mechanism between each scale to locally re-match high residual regions in the optical flow field, ensuring that anomaly regions can be corrected in a timely manner, thereby improving the stability and accuracy of the overall optical flow estimation.

[0191] A2.2: Based on the vehicle control data of the current frame, calculate the vehicle self-motion transformation matrix, and compensate the initial pixel-level motion vector field based on the vehicle self-motion transformation matrix to generate a motion vector field;

[0192] Specifically, the optical flow vector field calculated solely through pixel matching between images is insufficient to accurately reflect the image changes caused by the actual movement of the vehicle, especially when dynamic objects or road surface changes are present, which can easily lead to mismatches. Therefore, it is necessary to introduce vehicle self-motion information (i.e., vehicle control data), calculate the self-motion transformation matrix based on the actual motion state of the vehicle within the time interval, compensate for the initially estimated optical flow field, and eliminate or correct the systematic deviations caused by pure pixel matching.

[0193] In this embodiment, based on data such as the longitudinal velocity, steering wheel angle, and brake opening of the current frame, and combined with the vehicle's physical model, the vehicle's translation and rotation angle within the time interval are calculated, and a projection transformation matrix from three-dimensional space to the image plane is further constructed. Using this transformation matrix, the initial pixel-level motion vector field is inversely compensated, stripping away the components caused by the vehicle's self-motion in the vector field, retaining only the real motion components that change within the scene, and generating a compensated motion vector field.

[0194] Furthermore, to improve the compensation accuracy, this embodiment introduces a motion fuzz modeling mechanism. When calculating the self-motion transformation matrix, the dynamic influence of vehicle acceleration changes on the projection is considered, and the compensation vector amplitude and direction are dynamically adjusted to ensure that the self-motion compensation process still has high accuracy in rapid acceleration and sharp turning scenarios.

[0195] A2.3: With constant brightness constraints and constant gradient constraints as objectives, the motion vector field is iteratively optimized by combining total variational regularization to generate a refined vector field;

[0196] Specifically, even after self-motion compensation, the initial motion vector field may still contain noise, holes, or local errors due to occlusion, illumination changes, and the complexity of local motion. Direct use of this field can affect the accuracy and coherence of subsequent prediction frames. Therefore, the motion vector field needs further optimization, with brightness constancy and gradient constancy as optimization objectives. At the same time, a regularization mechanism is introduced to suppress noise and anomalous jumps, generating a smooth and physically reasonable refined vector field.

[0197] In this embodiment, a constant brightness constraint is set, meaning that the brightness change of the mapped pixel pairs is minimized under the action of the vector field. Simultaneously, a constant gradient constraint is set, requiring that the gradient direction and magnitude of the mapped pixels be as consistent as possible. Total variational regularization is used as a smoothing term, penalizing regions with drastic changes in the vector field during optimization, encouraging a smooth and continuous motion distribution. Overall optimization is achieved through an iterative solver, where each iteration updates the vector components and minimizes the loss objective until convergence, ultimately outputting a refined motion vector field.

[0198] A2.4: Perform reverse warping resampling on the second preprocessed frame based on the refined vector field to generate the first prediction frame;

[0199] Specifically, in order to infer the scene at future moments in time sequence, it is necessary to reverse simulate the pixel distribution of subsequent frames based on the current image and its motion trend. Therefore, it is necessary to perform a reverse distortion resampling operation on the second preprocessed frame to generate the first prediction frame as the starting point of the prediction sequence.

[0200] In this embodiment, based on the refined vector field, the expected future position of each pixel in the second preprocessed frame is calculated, and the corresponding sampling position is determined by a reverse mapping method. To avoid artifacts caused by simple interpolation, this embodiment preferably uses a bilinear interpolation combined with an edge-preserving interpolation strategy to perform weighted fusion on the pixels surrounding the target sampling position, ensuring that the resampling results are continuous and natural in spatial detail. The final generated first prediction frame, while maintaining structural consistency, can reasonably reflect the motion trend, serving as the basis for subsequent iterative predictions.

[0201] A2.5: Take the first predicted frame as the current frame, and perform optical flow estimation, self-motion compensation, vector optimization and resampling in a loop until the target number of predicted frames are generated to form a predicted frame sequence. Then, perform forward and backward consistency detection on each frame of the predicted frame sequence, mark the occlusion area and use edge-preserving interpolation and block matching to repair and fill the occlusion holes, and output the predicted scene video stream.

[0202] Specifically, to generate a continuous video stream of predicted future scenes, a single prediction step is insufficient. It is necessary to use the first predicted frame as the new current frame and repeat the processes of optical flow estimation, self-motion compensation, vector optimization, and inverse warp sampling to form a coherent multi-frame prediction that fully represents scene changes over several future moments. Simultaneously, because occlusion areas and accumulated errors are easily amplified during multi-step calculations, occlusion areas must be continuously detected and repaired during the generation process to ensure the integrity and naturalness of the predicted scene.

[0203] In this embodiment, an iterative loop mechanism is employed, using each newly generated prediction frame as input to repeatedly execute steps A2.1 to A2.4 until a set target number of prediction frames are generated. After each generation, forward and backward consistency checks are performed, verifying the consistency of pixel mapping between two consecutive frames and identifying potential occlusion or mismatched regions. For detected occlusion regions, edge-preserving interpolation is first applied to flexibly fill texture details based on surrounding visible pixel information. When edge interpolation fails to effectively repair the occlusion, a block matching repair strategy is further introduced, retrieving local blocks from temporally adjacent frames for optimal matching and filling to restore the reasonable content of the occluded region. Finally, all prediction frames are integrated to form a complete prediction scene video stream, possessing temporal continuity, spatial consistency, and color stability, meeting the requirements of remote control at the control center for continuous, high-quality visual input.

[0204] This application also provides another vehicle scene prediction method, which will be described below with reference to the accompanying drawings.

[0205] Please see Figure 6 The figure is a flowchart illustrating another vehicle scene prediction method provided in an embodiment of this application. Figure 6 The method shown can be applied to the vehicle-mounted end of a remote video transmission platform, which includes multiple cameras. The remote video transmission platform also includes a cloud terminal and a cabin terminal. The descriptions of the cabin terminal and the cloud terminal are provided above and will not be repeated here. Figure 6 The methods shown include the following Q1-Q2:

[0206] Q1: Send the vehicle-side data collected by the camera and the vehicle control data of the current frame to the cloud;

[0207] Q2: Receive vehicle control data from the cabin.

[0208] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A vehicle scene prediction method, applied to the cabin end of a remote video transmission platform, the remote video transmission platform further comprising an in-vehicle terminal and a cloud terminal, the in-vehicle terminal for transmitting in-vehicle data, and the cloud terminal for receiving the in-vehicle data from the in-vehicle terminal and generating a predicted scene video stream, characterized in that, The vehicle scenario prediction includes: The system receives the predicted scene video stream and performs color adjustment on the predicted scene video stream to generate an enhanced video stream, wherein the color adjustment includes: The target region and background region are identified by semantic segmentation, and the vehicle control data is combined to generate a processing frame and perform color distortion processing. The enhanced video stream is processed by a pre-trained image quality assessment model to output a real-time quality score, wherein the image quality assessment model is constructed based on multi-color space feature extraction and nonlinear regression network. The real-time quality score is compared with a preset quality threshold, and a compensation control signal is generated based on the comparison result to adjust the display queue and generate a continuous display video stream. Vehicle control data is generated based on the displayed video stream.

2. The vehicle scene prediction method according to claim 1, characterized in that, Perform color adjustment on the predicted scene video stream to generate an enhanced video stream, including: Semantic segmentation is performed on the predicted scene video stream to identify the target region and background region; Obtain vehicle control data from the current frame and historical frames, adjust the target area and background area based on the vehicle control data from the current frame and historical frames, and generate a first processing frame and a second processing frame. Extract the color space of the first processing frame, and perform color distortion processing on the color space according to the vehicle control data of the current frame to generate a corrected frame; The corrected frame is fused with the second processed frame based on the timestamp to generate an enhanced video stream.

3. The vehicle scene prediction method according to claim 2, characterized in that, Adjusting the target area and background area based on the vehicle control data of the current frame and historical frames includes: Based on the vehicle attitude parameters in the historical frame vehicle control data, the pixel displacement vector of the current frame in the historical frame is predicted to obtain the first displacement matrix, wherein the vehicle attitude parameters include steering wheel angle, longitudinal speed and brake opening. Based on the current frame vehicle control data, motion estimation is performed on the first displacement matrix to obtain the second displacement matrix; The first displacement matrix and the second displacement matrix are fused to obtain the motion trend matrix; Pixel-level motion compensation is performed on the target region based on the motion trend matrix to generate a first processing frame; Global affine motion compensation is performed on the background region based on the motion trend matrix to generate a second processing frame.

4. The vehicle scene prediction method according to claim 2, characterized in that, The step of performing color distortion processing on the color space based on the vehicle control data of the current frame includes: The motion phase is determined by analyzing the current frame based on vehicle control data. The motion phase includes acceleration phase, steering phase, and braking phase. In the rolling buffer corresponding to the motion phase, historical target region pixels with color entropy greater than a preset color threshold are extracted to generate a phase reference color palette, wherein the rolling buffer is set based on the predicted scene video stream corresponding to the historical frame. The color space is compared with the phase reference color palette channel by channel to generate a color offset metric. The color space of the first processed frame is graded and compensated according to the color offset metric to generate a corrected frame, wherein the graded compensation includes offset cancellation of the entire frame channel and chroma remapping of the pixel block.

5. The vehicle scene prediction method according to claim 2, characterized in that, The step of fusing the corrected frame with the second processed frame based on the timestamp includes: The correction frame and the second processing frame are spatiotemporally synchronized according to the timestamp; For each target pixel in the correction frame, a neighborhood set is constructed at the corresponding position in the second processing frame, wherein the search radius of the neighborhood set is adjusted according to the acceleration in the vehicle control data of the current frame; In the neighborhood set, the pixels are jointly sorted according to color similarity and spatial distance, a preset number of neighborhood pixels are selected and their color offset vectors are extracted. The local color compensation coefficient is calculated based on the color offset vector to generate a secondary compensation matrix. The target pixel is then compensated based on the secondary compensation matrix until each target pixel has been processed.

6. The vehicle scene prediction method according to claim 1, characterized in that, The process of processing the enhanced video stream using a pre-trained image quality assessment model includes: The frames to be evaluated are sequentially acquired from the enhanced video stream at a preset sampling frequency; For each frame to be evaluated, perform multi-color space conversion in parallel to obtain the RGB, YCbCr, Lab, and HSV representations of the frame to be evaluated; In each color space, the brightness, chromaticity component mean, chromaticity component variance, color contrast, edge gradient magnitude, and local texture energy are extracted and concatenated to form a frame-level feature vector. The frame-level feature vector is input into an offline-trained nonlinear regression network to obtain the corresponding frame-level quality score. Temporal exponential smoothing is applied to the frame-level quality score to generate a real-time quality score.

7. The vehicle scene prediction method according to claim 1, characterized in that, The step of generating a compensation control signal based on the comparison result includes: If the real-time quality score is less than the preset quality threshold, a first compensation control signal is generated, wherein the first compensation control signal marks the current frame as an invalid frame and deletes it from the display queue, and performs motion compensation interpolation between adjacent valid frames to generate a replacement frame; If the real-time quality score is greater than or equal to the preset quality threshold, a second compensation control signal is generated, wherein the second compensation control signal inserts the current frame as a valid frame into the display queue. The first compensation control signal or the second compensation control signal is written into the display buffer control register at the cabin end to update the display queue.

8. The vehicle scene prediction method according to claim 1, characterized in that, The vehicle scene prediction method also includes: Send the vehicle-side data collected by the camera and the vehicle control data of the current frame to the cloud; Receive vehicle control data from the cabin.

9. A vehicle scene prediction method applied to the cloud of a remote video transmission platform, wherein the remote video transmission platform further includes an in-vehicle terminal and a cabin terminal, the in-vehicle terminal for transmitting in-vehicle data, the cloud terminal for receiving the in-vehicle data from the in-vehicle terminal and generating a predicted scene video stream, and the cabin terminal for receiving the predicted scene video stream from the cloud terminal and performing quality assessment, and generating vehicle control data based on the quality assessment results, characterized in that... The vehicle scenario prediction includes: Receive the vehicle-side data and the vehicle control data of the current frame, preprocess the two consecutive frames of video data in the vehicle-side data, and generate the first preprocessed frame and the second preprocessed frame; The preprocessed vehicle-side data and vehicle control data are input into the preset scene prediction model. Based on the temporal characteristics of adjacent video frames and the vehicle motion state represented by the vehicle control information, the pixel-level motion vector is calculated through the optical flow prediction algorithm to generate a predicted scene video stream of multiple frames. The predicted scene video stream and the original video stream are simultaneously timestamped, encoded and encrypted, and then sent to the cabin terminal.

10. The vehicle scene prediction method according to claim 9, characterized in that, The step of generating a predicted scene video stream of multiple frames by calculating pixel-level motion vectors using an optical flow prediction algorithm based on the temporal characteristics of adjacent video frames and the vehicle motion state represented by vehicle control information includes: A multi-scale pyramid is constructed in the first preprocessing frame and the second preprocessing frame. The multi-scale pyramid is traversed by an optical flow algorithm to generate an initial pixel-level motion vector field. Based on the vehicle control data of the current frame, calculate the vehicle self-motion transformation matrix, and compensate the initial pixel-level motion vector field based on the vehicle self-motion transformation matrix to generate a motion vector field; With constant brightness and constant gradient constraints as objectives, the motion vector field is iteratively optimized by combining total variational regularization to generate a refined vector field. The second preprocessed frame is reverse-distorted and resampled according to the refined vector field to generate the first prediction frame; Using the first predicted frame as the current frame, optical flow estimation, self-motion compensation, vector optimization, and resampling are performed cyclically until the target number of predicted frames are generated, forming a predicted frame sequence. The predicted frame sequence is then subjected to forward and backward consistency detection frame by frame, occlusion areas are marked, and edge-preserving interpolation and block matching are used to repair and fill occlusion holes, and the predicted scene video stream is output.

Images