A remote control anti-delay video transmission method based on future scene generation network

Abstract

The present invention discloses a remote control anti-delay video transmission method based on a future scene generation network, including: synchronizing the time base of a remote control vehicle and a remote cockpit; collecting original video stream data of the environment in which the remote control vehicle is located; adding camera numbers and timestamp information to the original video stream data; preprocessing the decoded video stream data to generate multiple future scenes; generating multiple future scene video stream data; adding camera numbers and timestamp information to multiple future scene video streams; decrypting, decoding and distributing the data; performing packet loss and delay detection on the original video stream data, and optimizing the visual quality of the future scene video stream data; forming a complete and continuous video stream. The present invention greatly reduces the delay in video transmission, and future scene generation can predict and generate video content at future moments in advance, thereby effectively filling the video gaps that may occur due to transmission delays or data packet loss.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission, and in particular relates to a remote-controlled anti-delay video transmission method based on a future scene generation network. Background Art

[0002] With the development of 5G communication and the development of unmanned vehicles, a stable video transmission system is essential. For unmanned driving based on 5G remote control, vehicles need to transmit high-definition images and videos in real time so that the remote control center or other vehicles can analyze and make decisions. For example:

[0003] 1. In some cases, a remote operator needs to intervene in the control of an unmanned vehicle through the cockpit; through the 5G network, the remote operator can receive the real-time video stream transmitted by the vehicle, understand the environment around the vehicle in real time, and perform remote operations.

[0004] 2. Unmanned vehicles can share images and video information in real time through 5G networks to achieve vehicle collaboration; for example, the vehicle in front can transmit real-time video of road conditions to the vehicle behind to help it make decisions in advance.

[0005] 3. Unmanned vehicles require high-precision maps to assist navigation. Vehicles can capture images of roads and environments in real time and transmit them to map servers through 5G networks to update and build high-precision maps in real time.

[0006] However, the existing 5G network transmission has the following problems:

[0007] Although 5G networks have lower latency, there is still a certain transmission delay in the transmission of video streams when remotely controlling unmanned vehicles, especially when the network is unstable or the distance is long. When unmanned vehicles are equipped with multiple cameras, the load pressure of video streams is further increased. In this case, it may cause the display screen of the remote cockpit to go black, and video frames shared between vehicles or map server video frames to be lost. Previous 5G remote control video transmission technology has no corresponding optimization or compensation method for this problem. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a remote-controlled anti-delay video transmission method based on a future scene generation network, so as to solve the problems of video freeze and black screen caused by video frame loss or delayed arrival of video frames due to network instability when performing remote vehicle control in the prior art.

[0009] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] A remote control anti-delay video transmission method based on a future scene generation network of the present invention is applied to a remote vehicle control scenario, wherein the scenario includes a remote-controlled vehicle, a cloud server, a remote cockpit, and a display screen. The cloud server is respectively connected to the remote-controlled vehicle and the remote cockpit through a network, and the remote cockpit is connected to the display screen through a data line. The method steps are as follows:

[0011] 1) Synchronize the time bases of the remote-controlled vehicle and the remote cockpit;

[0012] 2) Collecting raw video stream data of the remote-controlled vehicle’s environment through multiple surround-view cameras;

[0013] 3) Add the camera number and timestamp information to the original video stream data, encode and encrypt it, and send it to the cloud server;

[0014] 4) The cloud server decrypts and decodes the received video stream data, and pre-processes the decoded video stream data to generate multiple future scenes;

[0015] 5) Generate multiple channels of future scene video stream data;

[0016] 6) Add camera numbers and timestamp information to multiple future scene video streams;

[0017] 7) Encode and encrypt the original video stream data and the future scene video stream data and send them to the remote cockpit;

[0018] 8) The remote cockpit decrypts, decodes and distributes the received data;

[0019] 9) Detect packet loss and delay of original video stream data, and optimize the visual quality of future scene video stream data;

[0020] 10) Form a complete and continuous video stream.

[0021] Furthermore, the step 1) specifically includes:

[0022] 11) The GPS timing equipment in the remote cockpit receives the standard time signal from the navigation satellite and synchronizes the time information to each subsystem of the remote cockpit based on the universal precision time protocol;

[0023] 12) The GPS timing device in the remote-controlled vehicle receives the standard time signal from the navigation satellite and converts it into the time reference signal of the remote-controlled vehicle system to ensure the time consistency between the remote-controlled vehicle and the remote cockpit.

[0024] Furthermore, the step 2) specifically includes:

[0025] 21) Install multiple surround view cameras evenly around the remote control vehicle to ensure that the cameras can cover a 360-degree field of view around the vehicle; calibrate the angle and focal length of each camera according to its installation position to ensure the clarity and distortion-free of the video images; use a laser rangefinder and calibration plate to perform dynamic and static calibration on each camera to ensure that the collected video stream data meets the expected visual perception standards;

[0026] 22) The real-time video data of the remote-controlled vehicle's environment collected by each surround-view camera is formatted according to a set frame rate and resolution; the remote-controlled vehicle's environment includes: roads, traffic conditions, natural environment, obstacles, and special operating scenarios that the remote-controlled vehicle may face.

[0027] Furthermore, the step 3) specifically includes:

[0028] 31) When processing video stream data, embed the corresponding camera number and timestamp information in each frame of video data;

[0029] 32) Use a high-efficiency video coding encoder to compress and encode the original video stream data after embedding the camera number and timestamp information. The original video stream data is divided into several video segments with camera numbers and timestamps during compression coding, which is convenient for subsequent encryption and transmission;

[0030] 33) Encrypt the compressed and encoded video stream data, and use the AES-128 encryption algorithm to encrypt each video segment. After encryption, each video segment becomes a ciphertext stream;

[0031] 34) The encrypted video stream data is packaged and transmitted to the cloud server through the HLS transmission protocol; the HLS transmission protocol divides the video stream data into small file segments and transmits them sequentially through HTTP.

[0032] Furthermore, the step 4) specifically includes:

[0033] 41) The cloud server uses the AES-128 decryption algorithm to decrypt the received video stream data;

[0034] 42) Use an efficient video codec to restore the decrypted video stream data to the original video frame; each decoded frame retains the camera number and timestamp information;

[0035] 43) Divide the decoded n frames of video into n-1 groups, and perform Gaussian pyramid downsampling on each group of images, adjust the image size to 1024x512, normalize the image pixel values, adjust their range to [0,1], and convert the image data type to float32 by dividing the image pixel value by 255 and the corresponding data format conversion function; convert the image color channel order from RGB to the BGR format required for future scene generation, and adjust the channel order to the number of channels, height, and width; pack each group of images into a batch according to the set batch size.

[0036] Furthermore, the step 5) specifically includes:

[0037] 51) Extract multi-scale, multi-level shallow features from each set of images; gradually process the input image through layered convolutional layers to extract fine-grained feature maps; in each convolutional layer, gradually reduce the resolution of the input feature map and continuously increase the number of channels to compress the data volume and retain feature information;

[0038] 52) Further optimize the feature map obtained in step 51), enhance the expressiveness of the feature through multi-scale dilated convolution, use dilated convolution to expand the receptive field without increasing the amount of calculation, and thus integrate information from different spatial scales; by setting different dilation rates to process feature maps from different scales, it is possible to obtain edge, texture, and local shape information of the input image at multiple scales; then use a linear layer to fuse the feature maps to generate cross-scale appearance features, thereby retaining the detailed information of the input features and enhancing the global context information of the feature maps;

[0039] 53) Using the transformer structure and the inter-frame attention mechanism to capture the motion information and appearance information between two input frames; realizing the unified extraction of motion information and appearance information through the inter-frame attention mechanism; the inter-frame attention mechanism generates an attention map for extracting motion features by calculating the similarity between the current frame and the adjacent frames. The attention map can capture the motion information between frames and enhance the expression ability of the appearance features without losing details, so that when generating future video frames, the motion changes between the input video frames can be accurately reflected while retaining the details and texture of the object;

[0040] 54) Convert the acquired feature information into a specific optical flow estimation, and gradually refine the estimation results to finally generate accurate future video frames; specifically: generate a preliminary optical flow estimation through linear expansion and feature fusion, the preliminary optical flow estimation represents the pixel motion between the input video frames, and gradually refine the preliminary optical flow estimation so that the preliminary optical flow estimation can accurately capture the subtle motion changes in the input video frames; use 3 convolutional layers, each convolutional layer continuously updates and corrects the estimation results of the previous layer by calculating the residual, combined with upsampling operations, so that the preliminary optical flow estimation can be gradually restored to the same resolution as the input image; use the generated optical flow and fusion map to synthesize future video frames to ensure the visual consistency and motion coherence of the synthesized frames with the input video frames;

[0041] 55) The future video frames generated in step 54) are refined to improve the visual quality of the final output video frames; specifically, the future video frames are repaired using multi-scale features and context information to ensure the high quality of the output video frames; in the refinement network, the initially generated future video frames are downsampled several times to extract multi-scale features, and the features are processed by convolutional layers to further enhance texture and detail information; then, the features are restored to the same resolution as the input video frames through upsampling operations, and fused with the initial future video frames to generate the final output future video frames;

[0042] 56) Encapsulate the future video frames generated by the refinement process, assemble the video frames in sequence, and obtain a generated video stream V_g{Fg_1,Fg_2,...,Fg_m} that is several frames faster than the original video stream V_o={Fo_1,Fo_2,...,Fo_m} in the time domain, where m is the total number of frames, Fo_i, Fg_i represent the i-th frame of the original video stream and the generated video stream, respectively.

[0043] Furthermore, the step 6) specifically includes:

[0044] Add the same camera number as the camera number of the original video stream to each future scene video stream. At the same time, add timestamp information to the video frame of each future scene video stream generated. The timestamp t_1 of the original video frame and the corresponding generated future scene video frame t_2 have the following corresponding relationship:

[0045] t_2=t_1 + t_p

[0046] Here, t_p indicates that the generated future scene video stream is faster than the original video stream in the time domain.

[0047] Furthermore, the step 7) specifically includes:

[0048] 71) Using high-efficiency video coding to compress and encode the original video stream data and the future scene video stream data according to the set frame rate and resolution;

[0049] 72) The compressed and encoded original video stream data and future scene video stream data are encrypted using the AES-128 symmetric encryption algorithm. Each video segment is converted into a ciphertext stream after encryption. During the encryption process, the camera number and timestamp information are also encrypted.

[0050] 73) The encoded and encrypted original video stream data and future scene video stream data are sent to the remote cockpit via the HLS transmission protocol.

[0051] Furthermore, the step 8) specifically includes:

[0052] 81) Decrypt the received original video stream data and future scene video stream data using the AES-128 decryption algorithm;

[0053] 82) Use an efficient video codec to convert the compressed original video stream data and future scene video stream data into the original video V_o and the generated video V_g. During the decoding process, each frame of video is extracted from the two received video stream data to ensure the clarity and integrity of the video. Each frame of the generated video after decoding retains the camera number and timestamp information.

[0054] Furthermore, the packet loss and delay detection of the original video stream data in step 9) specifically includes:

[0055] The receiving time point and serial number of the video data packet in the original video stream are analyzed to detect whether there is packet loss and obtain a set M of missing video frames that need to be displayed in the next display cycle but have not been received. The set M contains the serial numbers of all missing frames.

[0056] Furthermore, the visual quality optimization of the future scene video stream data in step 9) specifically includes:

[0057] 91) Delay-reduction optimized coding;

[0058] The de-delay encoder is used to extract the multi-scale features of the generated video frame {Fg_1, Fg_2, ..., Fg_m}} layer by layer, and the input image is gradually downsampled from the original high resolution to capture deeper semantic information; the image size is compressed through the downsampling operation of each layer, and a larger receptive field is captured to obtain the global structural information of the image; the encoder is composed of multiple non-linear activation convolutional layers, each of which contains a convolutional layer, a non-linear activation layer, an attention mechanism layer, a jump connection layer, and a downsampling layer; the de-delay encoder encodes the generated video frame to obtain a shallow generated video frame feature Fg_s containing global information;

[0059] 92) Generate deep feature extraction of video frames;

[0060] The shallow generated video frame features Fg_s extracted by the encoder are further processed to perform further feature extraction, enhance important information and reduce redundant information to obtain deep features Fg_d;

[0061] 93) De-delay optimized decoding;

[0062] The de-delay optimization decoder is used for upsampling, which is implemented by the upsampling method PixelShuffle; each pixel value is allocated to a larger space, thereby doubling the image resolution and halving the number of channels; the de-delay optimization decoder gradually restores the features extracted and processed by the encoder and the intermediate layer to the resolution of the original image; through layer-by-layer upsampling, the de-delay optimization decoder reconstructs the spatial structure and detail information of the image; in the process of restoring the image resolution, a jump connection is made with the corresponding layer of the encoder to merge the high-resolution features with the low-resolution features to retain more details; the de-delay optimization decoder decodes the deep features Fg_d of the generated video frame, and outputs the generated video frame {Fhq_1, Fhq_2, ..., Fhq_m} after deblurring and de-delaying, where m is the total number of frames, and Fhq_i represents the i-th high-quality generated video frame; the timestamp of the high-quality generated video frame is aligned with the timestamp of the generated video frame extracted in the step 82) to assemble into a high-quality generated video stream V_hq.

[0063] Furthermore, the step 10) specifically includes:

[0064] The complete video data compensated by the optimized high-quality generated video stream V_hq is transmitted to the driving display screen for real-time display.

[0065] Beneficial effects of the present invention:

[0066] 1. Efficient video transmission and optimization;

[0067] The present invention makes video transmission more efficient and stable in remote control applications by introducing future scene generation and visual quality optimization based on deep learning. In particular, when the network condition is poor or the bandwidth is limited, the video compensation mechanism can be used to predict the scene to be transmitted and generate compensation frames, ensuring the smoothness and clarity of the video. The present invention effectively reduces the computing pressure of the cloud server in the scenarios of one-control-many and multi-channel video transmission; in addition, during the video transmission process, the image distortion caused by motion blur or compression and the delay in generating the image are eliminated in real time, so that the operator can obtain high-quality video echo, further improving the accuracy and reliability of remote operation, greatly reducing the phenomenon of video freeze and quality degradation caused by network instability, and providing technical guarantee for the widespread application of 5G remote control.

[0068] 2. Strong anti-delay capability;

[0069] The present invention significantly reduces the delay in video transmission. Future scene generation can predict and generate video content at future moments in advance, thereby effectively filling the video gaps that may occur due to transmission delays or data packet loss. The future scene generation used in the present invention realizes the unified processing of motion and appearance information through the inter-frame attention mechanism, thereby improving the generation efficiency. It is particularly suitable for application scenarios that require extremely high real-time performance, such as remote driving, industrial robot control, etc., which significantly improves the response speed and real-time performance of the system, and ensures the safety and operating experience of remote operation.

[0070] 3. Intelligent prediction and compensation;

[0071] The present invention can predict the next few frames of video. When the network environment is not ideal or frame loss occurs during video transmission, the lost data can be compensated by generating predicted frames, thereby maintaining the continuity and consistency of the video stream. This not only improves the robustness of video transmission, but also effectively avoids operational errors or information loss caused by frame loss, thereby ensuring the accuracy and safety of remote control operations.

[0072] 4. Strong adaptability;

[0073] The present invention takes into account the changes in different network conditions and transmission delays, and has strong adaptability. It can adjust the video transmission strategy and compensation mechanism in real time according to the real-time detected network conditions. For example, when the network conditions are poor, the proportion of predicted frames can be increased, or the resolution of video transmission can be reduced to reduce delays; while when the network conditions are good, high-resolution and high-frame-rate video transmission can be provided. This flexible and adaptive design ensures that the system can work stably in various complex environments, and guarantees the continuity of remote operation and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 The schematic diagram of the method of the present invention is shown in FIG. DETAILED DESCRIPTION

[0075] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0076] Reference Figure 1 As shown, a remote control anti-delay video transmission method based on a future scene generation network is applied to a remote vehicle control scenario. The scenario includes a remote-controlled vehicle, a cloud server, a remote cockpit, and a display screen. The cloud server is connected to the remote-controlled vehicle and the remote cockpit through a network, and the remote cockpit is connected to the display screen through a data cable. The method steps are as follows:

[0077] 1) Synchronize the time bases of the remote-controlled vehicle and the remote cockpit; specifically:

[0078] 11) The GPS timing equipment in the remote cockpit receives the standard time signal from the navigation satellite and synchronizes the time information to various subsystems in the remote cockpit based on the General Precision Time Protocol (GPTP);

[0079] 12) The GPS timing device in the remote-controlled vehicle receives the standard time signal from the navigation satellite and converts it into the time reference signal of the remote-controlled vehicle system to ensure the time consistency between the remote-controlled vehicle and the remote cockpit.

[0080] 2) Collecting raw video stream data of the remote-controlled vehicle’s environment through multiple surround-view cameras; specifically including:

[0081] 21) Install multiple surround view cameras evenly around the remote control vehicle to ensure that the cameras can cover a 360-degree field of view around the vehicle; calibrate the angle and focal length of each camera according to its installation position to ensure the clarity and distortion-free of the video images; use a laser rangefinder and calibration plate to perform dynamic and static calibration on each camera to ensure that the collected video stream data meets the expected visual perception standards;

[0082] 22) Formatting the real-time video data of the remote-controlled vehicle's environment collected by each surround-view camera according to the set frame rate and resolution; the remote-controlled vehicle's environment includes: roads, traffic conditions (such as vehicles, pedestrians, traffic signs), natural environment (such as weather, lighting, terrain), obstacles (such as buildings, trees, roadblocks) and special operating scenarios that the remote-controlled vehicle may face (such as industrial areas and densely populated urban areas).

[0083] 3) Add the camera number and timestamp information to the original video stream data, encode and encrypt it, and send it to the cloud server; specifically:

[0084] 31) When processing video stream data, embed the corresponding camera number and timestamp information in each frame of video data;

[0085] 32) Use the High Efficiency Video Coding (HEVC) encoder to compress and encode the original video stream data after embedding the camera number and timestamp information. The original video stream data is divided into several video segments with camera numbers and timestamps during compression encoding to facilitate subsequent encryption and transmission;

[0086] 33) Encrypt the compressed and encoded video stream data, and use the AES-128 encryption algorithm to encrypt each video segment. After encryption, each video segment becomes a ciphertext stream;

[0087] 34) The encrypted video stream data is packaged and transmitted to the cloud server through the HLS (HTTP Live Streaming) transmission protocol; the HLS transmission protocol divides the video stream data into small file segments and transmits them sequentially through HTTP.

[0088] 4) The cloud server decrypts and decodes the received video stream data, and pre-processes the decoded video stream data to generate multiple future scenes; specifically, the following steps are included:

[0089] 41) The cloud server uses the AES-128 decryption algorithm to decrypt the received video stream data;

[0090] 42) Use the High Efficiency Video Coding (HEVC) decoder to restore the decrypted video stream data to the original video frame; each decoded frame retains the camera number and timestamp information;

[0091] 43) Divide the decoded n frames of video into n-1 groups (each group consists of two adjacent frames), perform Gaussian pyramid downsampling on each group of images, adjust the image size to 1024x512, normalize the image pixel values, adjust their range to [0,1], and convert the image data type to float32 by dividing the image pixel value by 255 and the corresponding data format conversion function; convert the image color channel order from RGB to the BGR format required for future scene generation, and adjust the channel order to the number of channels, height, and width; pack each group of images into a batch according to the set batch size.

[0092] 5) Generate multiple channels of future scene video stream data; specifically including:

[0093] 51) Extract multi-scale, multi-level shallow features from each set of images; gradually process the input image through layered convolutional layers to extract fine-grained feature maps; in each convolutional layer, gradually reduce the resolution of the input feature map and continuously increase the number of channels to compress the data volume and retain feature information;

[0094] 52) Further optimize the feature map obtained in step 51), enhance the expressiveness of the feature through multi-scale dilated convolution, and use dilated convolution to expand the receptive field without increasing the amount of calculation, thereby integrating information from different spatial scales; by setting different dilation rates (1, 2, 4) to process feature maps from different scales, it is possible to obtain edge, texture and local shape information of the input image at multiple scales; then use a linear layer to fuse the feature maps to generate cross-scale appearance features, thereby retaining the detailed information of the input features and enhancing the global context information of the feature maps;

[0095] 53) Using the transformer structure and the inter-frame attention mechanism to capture the motion information and appearance information between two input frames; realizing the unified extraction of motion information and appearance information through the inter-frame attention mechanism; the inter-frame attention mechanism generates an attention map for extracting motion features by calculating the similarity between the current frame and the adjacent frames. The attention map can capture the motion information between frames and enhance the expression ability of the appearance features without losing details, so that when generating future video frames, the motion changes between the input video frames can be accurately reflected while retaining the details and texture of the object;

[0096] 54) Convert the acquired feature information into a specific optical flow estimation, and gradually refine the estimation results to finally generate accurate future video frames; specifically: generate a preliminary optical flow estimation through linear expansion and feature fusion, the preliminary optical flow estimation represents the pixel motion between the input video frames, and gradually refine the preliminary optical flow estimation so that the preliminary optical flow estimation can accurately capture the subtle motion changes in the input video frames; use 3 convolutional layers, each convolutional layer continuously updates and corrects the estimation results of the previous layer by calculating the residual, combined with upsampling operations, so that the preliminary optical flow estimation can be gradually restored to the same resolution as the input image; use the generated optical flow and fusion map to synthesize future video frames to ensure the visual consistency and motion coherence of the synthesized frames with the input video frames;

[0097] 55) The future video frames generated in step 54) are refined to improve the visual quality of the final output video frames; specifically, the future video frames are repaired using multi-scale features and context information to ensure the high quality of the output video frames; in the refinement network, the initially generated future video frames are downsampled several times to extract multi-scale features, and the features are processed by convolutional layers to further enhance texture and detail information; then, the features are restored to the same resolution as the input video frames through upsampling operations, and fused with the initial future video frames to generate the final output future video frames;

[0098] 56) Encapsulate the future video frames generated by the refinement process, assemble the video frames in sequence, and obtain a generated video stream V_g{Fg_1,Fg_2,...,Fg_m} that is several frames faster than the original video stream V_o={Fo_1,Fo_2,...,Fo_m} in the time domain, where m is the total number of frames, Fo_i, Fg_i represent the i-th frame of the original video stream and the generated video stream, respectively.

[0099] 6) Add camera numbers and timestamp information to multiple future scene video streams; specifically:

[0100] Add the same camera number as the camera number of the original video stream to each future scene video stream. At the same time, add timestamp information to the video frame of each future scene video stream generated. The timestamp t_1 of the original video frame and the corresponding generated future scene video frame t_2 have the following corresponding relationship:

[0101] t_2=t_1 + t_p

[0102] Here, t_p indicates that the generated future scene video stream is faster than the original video stream in the time domain.

[0103] 7) Encode and encrypt the original video stream data and the future scene video stream data and send them to the remote cockpit; specifically including:

[0104] 71) Using High Efficiency Video Coding (HEVC) to compress and encode the original video stream data and the future scene video stream data according to the set frame rate and resolution;

[0105] 72) The compressed and encoded original video stream data and future scene video stream data are encrypted using the AES-128 symmetric encryption algorithm. Each video segment is converted into a ciphertext stream after encryption. During the encryption process, the camera number and timestamp information are also encrypted.

[0106] 73) The encoded and encrypted original video stream data and future scene video stream data are sent to the remote cockpit via the HLS transmission protocol.

[0107] 8) The remote cockpit decrypts, decodes and distributes the received data; specifically:

[0108] 81) Decrypt the received original video stream data and future scene video stream data using the AES-128 decryption algorithm;

[0109] 82) Use the High Efficiency Video Coding (HEVC) decoder to convert the compressed original video stream data and the future scene video stream data into the original video V_o and the generated video V_g. During the decoding process, each frame of video is extracted from the two received video stream data to ensure the clarity and integrity of the video. Each frame of the generated video frame after decoding retains the camera number and timestamp information.

[0110] 9) Detect packet loss and delay of original video stream data, and optimize the visual quality of future scene video stream data;

[0111] The packet loss and delay detection of the original video stream data specifically includes:

[0112] The receiving time point and serial number of the video data packet in the original video stream are analyzed to detect whether there is packet loss and obtain a set M of missing video frames that need to be displayed in the next display cycle but have not been received. The set M contains the serial numbers of all missing frames.

[0113] Among them, the visual quality optimization of future scene video stream data specifically includes:

[0114] 91) Delay-reduction optimized coding;

[0115] The de-delay encoder is used to extract the multi-scale features of the generated video frame {Fg_1, Fg_2, ..., Fg_m}} layer by layer, and the input image is gradually downsampled from the original high resolution to capture deeper semantic information; the image size is compressed through the downsampling operation of each layer, and a larger receptive field is captured to obtain the global structural information of the image; the encoder is composed of multiple non-linear activation convolutional layers, each of which contains a convolutional layer, a non-linear activation layer, an attention mechanism layer, a jump connection layer, and a downsampling layer; the de-delay encoder encodes the generated video frame to obtain a shallow generated video frame feature Fg_s containing global information;

[0116] 92) Generate deep feature extraction of video frames;

[0117] The shallow generated video frame features Fg_s extracted by the encoder are further processed to perform further feature extraction, enhance important information and reduce redundant information to obtain deep features Fg_d;

[0118] 93) De-delay optimized decoding;

[0119] The de-delay optimization decoder is used for upsampling, which is implemented by the upsampling method PixelShuffle; each pixel value is allocated to a larger space, thereby doubling the image resolution and halving the number of channels; the de-delay optimization decoder gradually restores the features extracted and processed by the encoder and the intermediate layer to the resolution of the original image; through layer-by-layer upsampling, the de-delay optimization decoder reconstructs the spatial structure and detail information of the image; in the process of restoring the image resolution, a jump connection is made with the corresponding layer of the encoder to merge the high-resolution features with the low-resolution features to retain more details; the de-delay optimization decoder decodes the deep features Fg_d of the generated video frame, and outputs the generated video frame {Fhq_1, Fhq_2, ..., Fhq_m} after deblurring and de-delaying, where m is the total number of frames, and Fhq_i represents the i-th high-quality generated video frame; the timestamp of the high-quality generated video frame is aligned with the timestamp of the generated video frame extracted in the step 82) to assemble into a high-quality generated video stream V_hq.

[0120] 10) Form a complete and continuous video stream, including:

[0121] The complete video data compensated by the optimized high-quality generated video stream V_hq is transmitted to the driving display screen for real-time display.

[0122] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A remote control delay-resistant video transmission method based on a future scene generation network, characterized in that: Applied to the remote vehicle control scenario, the scenario includes a remote-controlled vehicle, a cloud server, a remote cockpit, and a display screen. The cloud server is connected to the remote-controlled vehicle and the remote cockpit through the network, and the remote cockpit is connected to the display screen through a data cable. The method steps are as follows: 1) Synchronize the time base of the remote-controlled vehicle and the remote cockpit; 2) Collecting original video stream data of the environment where the remote-controlled vehicle is located through multiple surround-view cameras; 3) Add the camera number and timestamp information to the original video stream data, encode and encrypt it, and send it to the cloud server; 4) The cloud server decrypts and decodes the received video stream data, and pre-processes the decoded video stream data to generate multiple future scenes; 5) Generate multiple channels of future scene video stream data; 6) Add camera numbers and timestamp information to multiple future scene video streams; 7) Encode and encrypt the original video stream data and the future scene video stream data and send them to the remote cockpit; 8) The remote cockpit decrypts, decodes and distributes the received data; 9) Detect packet loss and delay of original video stream data, and optimize visual quality of future scene video stream data; 10) Form a complete and continuous video stream; The step 4) specifically includes: 41) The cloud server uses the AES-128 decryption algorithm to decrypt the received video stream data; 42) Use an efficient video codec to restore the decrypted video stream data to the original video frame; each decoded frame retains the camera number and timestamp information; 43) Divide the decoded n frames of video into n-1 groups, and perform Gaussian pyramid downsampling on each group of images, adjust the image size to 1024x512, and then normalize the image pixel values ​​to adjust their range to [0,1], and convert the image data type to float32 by dividing the image pixel value by 255 and the corresponding data format conversion function; convert the color channel order of the image from RGB to the BGR format required for future scene generation, and adjust the channel order to the number of channels, height, and width; pack each group of images into a batch according to the set batch size; The step 5) specifically includes: 51) Extract multi-scale, multi-level shallow features from each set of images; gradually process the input image through layered convolutional layers to extract fine-grained feature maps; in each convolutional layer, gradually reduce the resolution of the input feature map and continuously increase the number of channels to compress the data volume and retain feature information; 52) further optimize the feature map obtained in step 51), enhance the expressiveness of the feature through multi-scale dilated convolution, use dilated convolution to expand the receptive field without increasing the amount of calculation, and thus integrate information from different spatial scales; by setting different dilation rates to process feature maps from different scales, it is possible to obtain edge, texture, and local shape information of the input image at multiple scales; then use a linear layer to fuse the feature maps to generate cross-scale appearance features, thereby retaining the detailed information of the input features and enhancing the global context information of the feature maps; 53) Using the transformer structure and the inter-frame attention mechanism to capture the motion information and appearance information between two input frames; realizing the unified extraction of motion information and appearance information through the inter-frame attention mechanism; the inter-frame attention mechanism generates an attention map for extracting motion features by calculating the similarity between the current frame and the adjacent frames, and the attention map can capture the motion information between frames and enhance the expression ability of the appearance features without losing details, so that when generating future video frames, the motion changes between the input video frames can be accurately reflected while retaining the details and texture of the object; 54) Convert the acquired feature information into a specific optical flow estimation, and gradually refine the estimation result, and finally generate an accurate future video frame; specifically: generate a preliminary optical flow estimation through linear expansion and feature fusion, the preliminary optical flow estimation represents the pixel motion between the input video frames, and gradually refine the preliminary optical flow estimation so that the preliminary optical flow estimation can accurately capture the subtle motion changes in the input video frames; use 3 convolutional layers, each convolutional layer continuously updates and corrects the estimation result of the previous layer by calculating the residual, and combined with the upsampling operation, the preliminary optical flow estimation can be gradually restored to the same resolution as the input image; use the generated optical flow and fusion map to synthesize the future video frame to ensure the visual consistency and motion coherence of the synthesized frame with the input video frame; 55) performing refinement processing on the future video frame generated in step 54) to improve the visual quality of the final output video frame; specifically: using multi-scale features and context information to repair the future video frame to ensure the high quality of the output video frame; in the refinement network, the initially generated future video frame is downsampled several times to extract multi-scale features, and the features are processed by the convolution layer to further enhance the texture and detail information; then through the upsampling operation, the features are restored to the same resolution as the input video frame, and fused with the initial future video frame to generate the final output future video frame; 56) Encapsulating the future video frames generated by the refinement process, assembling the video frames in order, and obtaining a generated video stream V_g{Fg_1, Fg_2, ..., Fg_m} that is faster than the original video stream V_o={Fo_1, Fo_2, ..., Fo_m} by several frames in the time domain, where m is the total number of frames, Fo_i, Fg_i represent the i-th frame of the original video stream and the generated video stream respectively; The step 10) specifically includes: The complete video data compensated by the optimized high-quality generated video stream V_hq is transmitted to the driving display screen for real-time display.

2. The remote control delay-resistant video transmission method based on future scene generation network according to claim 1 is characterized in that: The step 1) specifically includes: 11) The GPS timing equipment in the remote cockpit receives the standard time signal from the navigation satellite and synchronizes the time information to each subsystem of the remote cockpit based on the universal precision time protocol; 12) The GPS timing device in the remote-controlled vehicle receives the standard time signal of the navigation satellite and converts it into the time reference signal of the remote-controlled vehicle system to ensure the time consistency between the remote-controlled vehicle and the remote cockpit.

3. The remote control delay-resistant video transmission method based on future scene generation network according to claim 1 is characterized in that: The step 2) specifically includes: 21) Install multiple surround view cameras evenly around the remote control vehicle to ensure that the cameras can cover a 360-degree field of view around the vehicle; calibrate the angle and focal length of each camera according to the installation position to ensure the clarity and distortion of the video images; use a laser rangefinder and calibration plate to dynamically and statically calibrate each camera to ensure that the collected video stream data meets the expected visual perception standards; 22) The real-time video data of the remote-controlled vehicle's environment collected by each surround-view camera is formatted according to the set frame rate and resolution; the remote-controlled vehicle's environment includes: roads, traffic conditions, natural environment, obstacles, and special operating scenarios that the remote-controlled vehicle may face.

4. The remote control delay-resistant video transmission method based on future scene generation network according to claim 1 is characterized in that: The step 3) specifically includes: 31) When processing the video stream data, embed the corresponding camera number and timestamp information in each frame of video data; 32) using a high-efficiency video coding encoder to compress and encode the original video stream data after the camera number and timestamp information are embedded, and the original video stream data is divided into a plurality of video segments with camera numbers and timestamps during compression and encoding; 33) Encrypting the compressed and encoded video stream data, using the AES-128 encryption algorithm to encrypt each video segment, and each video segment becomes a ciphertext stream after encryption; 34) The encrypted video stream data is packaged and transmitted to the cloud server through the HLS transmission protocol; the HLS transmission protocol divides the video stream data into small file segments and transmits them sequentially through HTTP.

5. The remote control delay-resistant video transmission method based on future scene generation network according to claim 1 is characterized in that: The step 6) specifically includes: Add the same camera number as the camera number of the original video stream to each future scene video stream. At the same time, add timestamp information to the video frame of each future scene video stream generated. The timestamp t_1 of the original video frame and the corresponding generated future scene video frame t_2 have the following corresponding relationship: t_2=t_1+t_p Here, t_p indicates that the generated future scene video stream is faster than the original video stream in the time domain.

6. The remote control delay-resistant video transmission method based on future scene generation network according to claim 1 is characterized in that: The step 7) specifically includes: 71) using high-efficiency video coding to compress and encode the original video stream data and the future scene video stream data according to the set frame rate and resolution; 72) The compressed and encoded original video stream data and the future scene video stream data are encrypted using the AES-128 symmetric encryption algorithm. Each video segment is converted into a ciphertext stream after encryption. During the encryption process, the camera number and timestamp information are also encrypted. 73) The encoded and encrypted original video stream data and future scene video stream data are sent to the remote cockpit via the HLS transmission protocol.

7. The remote control delay-resistant video transmission method based on future scene generation network according to claim 1 is characterized in that: The step 8) specifically includes: 81) Decrypting the received original video stream data and future scene video stream data using the AES-128 decryption algorithm; 82) Use an efficient video codec to convert the compressed original video stream data and future scene video stream data into the original video V_o and the generated video V_g. During the decoding process, each frame of video is extracted from the two received video stream data to ensure the clarity and integrity of the video. Each frame of the generated video frame after decoding retains the camera number and timestamp information.

8. The remote control delay-resistant video transmission method based on future scene generation network according to claim 1 is characterized in that: The visual quality optimization of the future scene video stream data in step 9) specifically includes: 91) Delay-removal and optimized coding; The de-delay encoder is used to extract the multi-scale features of the generated video frame {Fg_1, Fg_2, ..., Fg_m}} layer by layer, and the input image is gradually downsampled from the original high resolution to capture deeper semantic information; the image size is compressed through the downsampling operation of each layer, and a larger receptive field is captured to obtain the global structural information of the image; the encoder is composed of multiple non-linear activation convolutional layers, each of which contains a convolutional layer, a non-linear activation layer, an attention mechanism layer, a jump connection layer, and a downsampling layer; the de-delay encoder encodes the generated video frame to obtain a shallow generated video frame feature Fg_s containing global information; 92) Generate deep feature extraction of video frames; The shallow generated video frame features Fg_s extracted by the encoder are further processed to perform further feature extraction, enhance important information and reduce redundant information to obtain deep features Fg_d; 93) De-delay optimized decoding; The de-delay optimization decoder is used for upsampling, which is implemented by the upsampling method PixelShuffle; each pixel value is allocated to a larger space, thereby doubling the image resolution and halving the number of channels; the de-delay optimization decoder gradually restores the features extracted and processed by the encoder and the intermediate layer to the resolution of the original image; through layer-by-layer upsampling, the de-delay optimization decoder reconstructs the spatial structure and detail information of the image; in the process of restoring the image resolution, a jump connection is made with the corresponding layer of the encoder to merge the high-resolution features with the low-resolution features to retain more details; the de-delay optimization decoder decodes the deep feature Fg_d of the generated video frame, and outputs the generated video frame {Fhq_1, Fhq_2, ..., Fhq_m} after deblurring and delaying, where m is the total number of frames, and Fhq_i represents the i-th high-quality generated video frame; the timestamp of the high-quality generated video frame is aligned with the timestamp of the generated video frame extracted in the step 82) to assemble into a high-quality generated video stream V_hq.

Images