Vehicle-ground cooperative video compression, restoration and transmission method and system

Abstract

The invention provides a vehicle-ground cooperative video compression and restoration transmission method and system, and relates to the technical field of computers, and the method comprises the steps: carrying out the feature extraction of each video frame in a to-be-transmitted video collected by a vehicle, carrying out the clustering based on the extracted features, and determining a key frame in the to-be-transmitted video; determining a transmission ratio of a key frame to a common frame in the to-be-transmitted video based on a channel state parameter between the vehicle and the ground, and determining a transmission frame in the to-be-transmitted video based on the transmission ratio; generating an intermediate frame between the adjacent transmission frames based on the optical flow between the adjacent transmission frames and the visibility map; and generating an image sequence based on the transmission frame and the intermediate frame, and performing super-resolution restoration on the image sequence to obtain a restored video corresponding to the to-be-transmitted video on the ground station. According to the method and the device provided by the invention, efficient and stable video transmission between the vehicle and the ground is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a vehicle-ground collaborative video compression, restoration and transmission method and system. Background Art

[0002] Vehicle-ground collaborative video transmission refers to the transmission of high-definition video inside and outside the vehicle captured in real time by the on-board camera of a high-speed vehicle to a ground receiving station. Due to the huge amount of video data to be transmitted, limited bandwidth resources, and the high-speed movement of vehicles will cause large fluctuations in network status, making it impossible to efficiently and stably transmit videos between the vehicle and the ground.

[0003] Therefore, how to achieve efficient and stable video transmission between vehicles and the ground has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The present application provides a vehicle-ground collaborative video compression and restoration transmission method and system, which are used to solve the technical problem of how to perform efficient and stable video transmission between the vehicle and the ground.

[0005] The present application provides a vehicle-ground collaborative video compression, restoration and transmission method, comprising: Extracting features from each video frame in the video to be transmitted collected by the vehicle, clustering based on the extracted features, and determining key frames in the video to be transmitted; Determine a transmission ratio of key frames to common frames in the video to be transmitted based on a channel state parameter between the vehicle and the ground, and determine a transmission frame in the video to be transmitted based on the transmission ratio; Based on the optical flow and visibility map between adjacent transmission frames, generate an intermediate frame between the adjacent transmission frames; An image sequence is generated based on the transmission frame and the intermediate frame, and super-resolution restoration is performed on the image sequence to obtain a restored video corresponding to the video to be transmitted at the ground station.

[0006] In some embodiments, extracting features from each video frame in the video to be transmitted collected by the vehicle, clustering based on the extracted features, and determining key frames in the video to be transmitted include: Extracting features from each video frame in the video to be transmitted based on a feature extraction network to obtain image features corresponding to each video frame; Cluster the image features corresponding to each video frame to obtain multiple classes; The video frame corresponding to the image feature closest to the cluster center in each class is used as the key frame; Wherein, the feature extraction network is trained based on the sample video to be transmitted and the restored video corresponding to the sample video to be transmitted; the feature extraction network includes at least one attention feature extraction layer.

[0007] In some embodiments, determining the transmission ratio of key frames to ordinary frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determining the transmission frames in the video to be transmitted based on the transmission ratio include: Obtaining real-time values ​​of multiple channel status parameters between the vehicle and the ground; Determine the estimated value of the channel state between the vehicle and the ground based on the calculation weights corresponding to the various channel state parameters and the real-time values ​​of the various channel state parameters; Determining a transmission ratio of the key frames and a transmission ratio of the common frames based on the channel state estimation value; The transmission ratio of the key frames is inversely proportional to the channel state estimation value; and the transmission ratio of the common frames is directly proportional to the channel state estimation value.

[0008] In some embodiments, generating an intermediate frame between adjacent transmission frames based on an optical flow and a visibility map between adjacent transmission frames comprises: Performing optical flow calculation on adjacent first transmission frames and second transmission frames based on an optical flow calculation network to obtain a bidirectional optical flow between the first transmission frame and the second transmission frame; Based on the bidirectional optical flow, determining a first optical flow from the intermediate frame to the first transmission frame and a second optical flow from the intermediate frame to the second transmission frame; Processing the first optical flow and the second optical flow based on an optical flow interpolation network to determine a first visibility map of the intermediate frame relative to the first transmission frame and a second visibility map of the intermediate frame relative to the second transmission frame; Based on the first visibility map and the second visibility map, interpolation is performed between the first transmission frame and the second transmission frame to generate the intermediate frame.

[0009] In some embodiments, the generating of an image sequence based on the transmission frame and the intermediate frame, performing super-resolution restoration on the image sequence, and obtaining a restored video corresponding to the video to be transmitted at the ground station includes: generating an initial image sequence based on the transmission frame and the intermediate frame; Preprocessing the initial image sequence based on an image preprocessing network to obtain the image sequence; Processing the image sequence based on a super-resolution restoration network to obtain a restored frame corresponding to each frame in the image sequence; Based on the restored frames corresponding to the frames, generating the restored video corresponding to the video to be transmitted at the ground station; Wherein, the image preprocessing network includes multiple residual blocks; the preprocessing includes noise reduction and artifact removal.

[0010] In some embodiments, the preprocessing the initial image sequence based on the image preprocessing network to obtain the image sequence includes: Determining a current frame in the initial image sequence; Preprocessing the current frame based on the image preprocessing network to obtain a current iterative processing result of the current frame; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is greater than a preset difference, continuing to preprocess the current iterative processing result of the current frame based on the image preprocessing network; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is less than a preset difference, the current iterative processing result is used as the preprocessing result of the current frame, and the next frame in the initial image sequence is determined as the current frame; The image sequence is determined based on the preprocessing results of each frame in the initial image sequence.

[0011] In some embodiments, the method further comprises: During the training process of the super-resolution restoration network, random degradation processing is performed on the training samples.

[0012] The present application provides a vehicle-ground collaborative video compression and restoration transmission system, comprising: A key frame extraction module is used to extract features from each video frame in the video to be transmitted collected by the vehicle, cluster the extracted features, and determine the key frames in the video to be transmitted; A video transmission module, used to determine the transmission ratio of key frames to common frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determine the transmission frames in the video to be transmitted based on the transmission ratio; An intermediate frame insertion module, used for generating an intermediate frame between adjacent transmission frames based on the optical flow and visibility graph between the adjacent transmission frames; The video restoration module is used to generate an image sequence based on the transmission frame and the intermediate frame, perform super-resolution restoration on the image sequence, and obtain a restored video corresponding to the video to be transmitted at the ground station.

[0013] The present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the vehicle-ground collaborative video compression, restoration and transmission method is implemented.

[0014] The present application provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the vehicle-ground collaborative video compression, restoration and transmission method is implemented.

[0015] The vehicle-ground collaborative video compression, restoration and transmission method and system provided in the present application extract features from each video frame in the video to be transmitted collected by the vehicle, cluster based on the extracted features, and determine the key frames in the video to be transmitted; determine the transmission ratio of key frames to ordinary frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determine the transmission frames in the video to be transmitted based on the transmission ratio; generate intermediate frames between adjacent transmission frames based on the optical flow and visibility map between adjacent transmission frames; generate an image sequence based on the transmission frames and the intermediate frames, perform super-resolution restoration on the image sequence, and obtain the restored video corresponding to the video to be transmitted at the ground station; because the features of each video frame in the video to be transmitted are extracted and clustered, the key frames can be accurately extracted in complex video content , reducing the amount of transmitted data, improving the transmission efficiency and quality of the video; because the transmission ratio of key frames to ordinary frames is determined in real time according to the channel state parameters between the vehicle and the ground, the video transmission is adapted to the network communication quality, improving the transmission efficiency and stability of the video under various network communication conditions; because adjacent transmission frames are interpolated according to the optical flow and visibility map, the movement of objects can be accurately predicted, smooth and natural intermediate frames can be generated, and the quality of the video can be improved; because the obtained transmission frames are super-resolution restored, a clearer and more detailed restored image is obtained, improving the quality of the video; through the combination of the above methods, the video compression and restoration are realized as a whole, the transmission cost is saved, and efficient and stable video transmission between the vehicle and the ground is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flow chart of the vehicle-ground collaborative video compression, restoration and transmission method provided in this application.

[0019] Figure 2 It is one of the structural diagrams of the vehicle-ground collaborative video compression, restoration and transmission system provided in this application.

[0020] Figure 3 This is the second structural diagram of the vehicle-ground collaborative video compression, restoration and transmission system provided in this application.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second" etc. in the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units or modules need not be limited to those steps or units or modules clearly listed, but may include other steps or units or modules that are not clearly listed or inherent to these processes, methods, products or equipment.

[0024] Figure 1 This is one of the flow charts of the vehicle-ground collaborative video compression, restoration and transmission method provided in this application, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0025] Step 110 : extracting features from each video frame in the video to be transmitted collected by the vehicle, clustering based on the extracted features, and determining key frames in the video to be transmitted.

[0026] Specifically, the execution subject of the vehicle-ground collaborative video compression and restoration transmission method provided in the embodiment of the present application is a vehicle-ground collaborative video compression and restoration transmission system. The system can be implemented by software, such as a vehicle-ground collaborative video transmission program running in the vehicle and the ground station; it can also be implemented by hardware, such as a computer or server that executes the vehicle-ground collaborative video compression and restoration transmission method in the vehicle and the ground station.

[0027] The application scenario of the method provided in the embodiment of the present application is vehicle-to-ground communication, specifically, the vehicle transmits the video collected by itself to the ground station. The vehicle may refer to a high-speed train or car, etc. The ground station refers to a control station or a monitoring station, etc. that wirelessly communicates with the vehicle.

[0028] The video to be transmitted refers to the video obtained by a video acquisition device (such as a camera, etc.) installed in the vehicle to capture the interior space and / or the exterior environment of the vehicle. A video frame is a still image and is the basic unit of a video.

[0029] Feature extraction is the process of extracting representative information or features from video frames. These features are used to describe the content of the video frames. Common features include edges, corners, textures, color histograms, etc. By extracting these features, they can be used for tasks such as video frame classification, object detection, and video frame matching.

[0030] Clustering refers to dividing the video frames in a video into several classes or clusters, so that objects in the same class are as similar as possible, and objects in different classes are as different as possible. Among the video frames belonging to the same class, one of the video frames can be selected as the key frame.

[0031] A key frame can be used as the most representative frame in the same type of video frames. Through key frames, the main content or changes of the video can be captured. In vehicle-ground collaborative video transmission, the number of video frames that need to be transmitted can be reduced by extracting and transmitting key frames.

[0032] Step 120: Determine the transmission ratio of key frames to common frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determine the transmission frames in the video to be transmitted based on the transmission ratio.

[0033] Specifically, video transmission between the vehicle and the ground is affected by the channel state. Channel state parameters refer to parameters that describe the performance and quality of the communication channel. Based on the channel state parameters between the vehicle and the ground, the communication quality between the vehicle and the ground can be evaluated. The better the communication quality, the more suitable it is for video transmission.

[0034] Ordinary frames refer to video frames other than key frames, that is, non-key frames. When the communication quality is good, the system may transmit more ordinary frames to improve the smoothness and quality of the video; when the communication quality is poor, the system will give priority to transmitting key frames to ensure that important information can be delivered in time.

[0035] The transmission ratio refers to the ratio of the actual transmission data volume of a video frame to the original image data volume during video transmission. This ratio reflects the compression efficiency and the degree of data reduction, and is usually used to evaluate the performance of video compression algorithms.

[0036] In the embodiment of the present application, the transmission frame refers to the video frame actually transmitted. The transmission frame includes key frames and ordinary frames. It should be noted that the video transmission process is carried out in real time, so the channel state parameters between the vehicle and the ground are also obtained in real time, and the transmission ratio is also determined in real time.

[0037] Step 130: Generate an intermediate frame between adjacent transmission frames based on the optical flow and visibility map between adjacent transmission frames.

[0038] Specifically, optical flow refers to the movement or change of each pixel in a scene in consecutive image frames. It describes the direction and speed of movement of objects in an image and is often used for motion estimation, target tracking, and video stabilization.

[0039] The visibility map is used to represent the visibility of pixels in an image. Based on the visibility map, we can analyze which pixels are visible in the image and which pixels are invisible in the image.

[0040] An intermediate frame refers to a video frame obtained by interpolating between adjacent transmission frames.

[0041] Interpolating between two adjacent transmission frames (especially between key frames) to obtain intermediate frames can improve the smoothness of the video and reduce the visual frame skipping phenomenon. This method is called frame interpolation or motion estimation interpolation, which can make video playback smoother, especially when the video frame rate is low. It predicts and generates intermediate frames between two transmission frames to fill the gaps between frames, thereby improving the viewing experience and visual effects.

[0042] According to the optical flow and visibility map between adjacent transmission frames, the visibility and motion changes of pixels in the two transmission frames can be analyzed, and then the moving direction and speed of objects in the transmission frames can be predicted, and the intermediate frame can be generated according to the prediction results.

[0043] Step 140: Generate an image sequence based on the transmission frame and the intermediate frame, perform super-resolution restoration on the image sequence, and obtain a restored video corresponding to the video to be transmitted at the ground station.

[0044] Specifically, according to the time information of the transmission frame and the intermediate frame, an image sequence can be generated in time order.

[0045] Considering that the quality of the transmission frame may be degraded during the transmission process, and the quality of the intermediate frame generated by interpolation is lower than that of the transmission frame, the image sequence can be restored by super-resolution, and the images in the image sequence can be upgraded from low resolution to high resolution, making each frame in the image sequence clearer and more detailed. Based on the super-resolution restored frames, the restored video corresponding to the video to be transmitted at the ground station can be obtained.

[0046] In an embodiment of the present application, the first two steps can be implemented in a device set in the vehicle, and the last two steps can be implemented in a device set in a ground station.

[0047] The vehicle-ground collaborative video compression, restoration and transmission method provided in the embodiment of the present application performs feature extraction on each video frame in the video to be transmitted collected by the vehicle, performs clustering based on the extracted features, and determines the key frames in the video to be transmitted; determines the transmission ratio of key frames to ordinary frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determines the transmission frames in the video to be transmitted based on the transmission ratio; generates intermediate frames between adjacent transmission frames based on the optical flow and visibility map between adjacent transmission frames; generates an image sequence based on the transmission frames and the intermediate frames, performs super-resolution restoration on the image sequence, and obtains the restored video corresponding to the video to be transmitted at the ground station; since feature extraction and clustering are performed on each video frame in the video to be transmitted, key frames can be accurately extracted in complex video content , reducing the amount of transmitted data, improving the transmission efficiency and quality of the video; because the transmission ratio of key frames to ordinary frames is determined in real time according to the channel state parameters between the vehicle and the ground, the video transmission is adapted to the network communication quality, improving the transmission efficiency and stability of the video under various network communication conditions; because adjacent transmission frames are interpolated according to the optical flow and visibility map, the movement of objects can be accurately predicted, smooth and natural intermediate frames can be generated, and the quality of the video can be improved; because the obtained transmission frames are super-resolution restored, a clearer and more detailed restored image is obtained, improving the quality of the video; through the combination of the above methods, the video compression and restoration are realized as a whole, the transmission cost is saved, and efficient and stable video transmission between the vehicle and the ground is realized.

[0048] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0049] In some embodiments, feature extraction is performed on each video frame in the video to be transmitted collected by the vehicle, clustering is performed based on the extracted features, and key frames in the video to be transmitted are determined, including: Based on the feature extraction network, feature extraction is performed on each video frame in the video to be transmitted to obtain image features corresponding to each video frame; Cluster the image features corresponding to each video frame to obtain multiple classes; The video frame corresponding to the image feature closest to the cluster center in each class is used as the key frame; Among them, the feature extraction network is trained based on the sample video to be transmitted and the restored video corresponding to the sample video to be transmitted; the feature extraction network includes at least one attention feature extraction layer.

[0050] Specifically, the embodiments of the present application can utilize deep learning technology and adopt a feature extraction network to extract features from each video frame in the video to be transmitted, so as to obtain image features corresponding to each video frame.

[0051] The feature extraction network can select the autoencoder model. The autoencoder model is used to extract low-dimensional feature representations from video frames. The autoencoder consists of two parts: the encoder and the decoder. The encoder is mainly used to convert high-dimensional video frames into low-dimensional image features, which can be expressed as:

[0052] in, is the video frame, is the image feature, is the function corresponding to the encoder, is the weight matrix of the encoder, is the encoder bias vector.

[0053] The decoder is used to restore low-dimensional image features to high-dimensional video, which can be expressed as:

[0054] in, For the restored video, is the function corresponding to the decoder, is the weight matrix of the decoder, is the bias vector of the decoder.

[0055] The goal of training the autoencoder is to minimize the original video frame And the restored video frame The reconstruction error between can use the mean square error (MSE) as the loss function, which is expressed as:

[0056] In the formula, is the loss function, is the number of training samples, is the label of the training sample.

[0057] In the embodiment of the present application, the training sample may specifically be a video frame in the sample video to be transmitted. During the training process, the loss function is calculated using the video frame in the restored video corresponding to the sample video to be transmitted.

[0058] During the training phase of the feature extraction network, both the encoder and the decoder are used. The encoder compresses the high-dimensional data of the video frame into a low-dimensional feature representation, while the decoder is used to restore these low-dimensional features to high-dimensional data close to the original input, with the goal of minimizing the reconstruction error by optimizing the entire autoencoder network. During the inference phase of the feature extraction network, only the encoder is needed, which can effectively extract key low-dimensional features from the video frame, which are important bases for key frame selection. The decoder is not needed in this phase. Therefore, the key frame extraction phase only relies on the output of the encoder, and the decoder part is no longer used in this process.

[0059] In order to improve the feature extraction capability, the feature extraction network includes at least one attention feature extraction layer. Specifically, the encoder includes at least one attention feature extraction layer.

[0060] An attention feature extraction layer is added to the encoder to enhance the focus on important features through the attention mechanism. The attention mechanism highlights key features by calculating the weight of each feature. Its implementation process can be expressed as:

[0061] In the formula, is the attention weight, is the query vector, is the key vector, is a value vector, is the dimension of the key vector, is the transposition operator, is the input feature, is the weight matrix corresponding to the query vector, is the weight matrix corresponding to the key vector, is the weight matrix corresponding to the value vector.

[0062] Through the attention mechanism, it is specially optimized for video key frame extraction, which can more accurately highlight the features in the video frame that are important for subsequent processing; at the same time, the attention mechanism is combined with the deep autoencoder, so that the encoder can dynamically adjust the focus on key features while compressing features, thereby improving the accuracy of key frame extraction.

[0063] Cluster the image features corresponding to each video frame to obtain multiple classes; use the video frame corresponding to the image feature closest to the cluster center in each class as the key frame. The K-means clustering algorithm can be used to classify the extracted features and classify similar video frames into one class. The specific steps of the algorithm are as follows: Step 1: Initialization

[0064] You can choose The initial cluster centers are represented as , , …, .

[0065] Step 2: Allocation

[0066] Each feature vector Assign (calculate the similarity between each feature vector and the initial cluster center, and assign according to the similarity) to the nearest cluster center, which can be expressed as:

[0067] In the formula, is the feature vector The corresponding cluster centers are and For the label.

[0068] Step 3: Update

[0069] Recalculate the cluster center, expressed as:

[0070] Step 4: Iteration

[0071] Repeat the assignment and update steps until the cluster centers no longer change or change very little.

[0072] After clustering, select the frame closest to the center in each cluster as the key frame, which can be expressed as:

[0073] In the formula, is a key frame, is the minimum function.

[0074] The selected key frames are used for subsequent video compression and transmission to improve transmission efficiency and video quality.

[0075] The vehicle-ground collaborative video compression, restoration and transmission method provided in the embodiment of the present application uses a feature extraction network (deep autoencoder) and an attention mechanism to accurately extract key frames in the video, and then uses a clustering algorithm for further optimization and selection. It can efficiently extract key frames in complex video content, reduce the amount of transmitted data, and improve the transmission efficiency and quality of the video.

[0076] In some embodiments, determining a transmission ratio of key frames to common frames in a video to be transmitted based on a channel state parameter between a vehicle and a ground, and determining a transmission frame in the video to be transmitted based on the transmission ratio includes: Obtaining real-time values ​​of multiple channel status parameters between the vehicle and the ground; Determine the estimated value of the channel state between the vehicle and the ground based on the calculation weights corresponding to the various channel state parameters and the real-time values ​​of the various channel state parameters; Based on the channel state estimation value, determining the transmission ratio of key frames and the transmission ratio of common frames; The transmission ratio of key frames is inversely proportional to the channel state estimation value; the transmission ratio of common frames is directly proportional to the channel state estimation value.

[0077] Specifically, the embodiments of the present application may also adopt an adaptive transmission adjustment strategy during the transmission of the video, which aims to dynamically adjust the video compression rate and transmission ratio according to the channel state to ensure efficient and stable video transmission in the vehicle-to-ground communication environment. The strategy combines the channel state parameters monitored in real time, and dynamically adjusts the video transmission strategy by calculating the comprehensive channel state parameters. The video compression rate refers to the degree to which the compression algorithm compresses the data in the video frame to reduce the amount of data. The embodiments of the present application focus on the method of adjusting the transmission ratio.

[0078] The real-time values ​​of multiple channel status parameters between the vehicle and the ground can be obtained. The following channel status parameters can be monitored in real time, including but not limited to: (1) Received Signal Strength Indicator (RSSI): Indicates the strength of the received signal, usually in decibel milliwatts (dBm). (2) Signal-to-Noise Ratio (SNR): Indicates the ratio of signal to noise, usually in decibels (dB). (3) Bandwidth (B): Indicates the transmission bandwidth of the network, usually in megabits per second (Mbps). (4) Delay (D): Indicates the delay of data transmission, usually in milliseconds (ms). (5) Jitter (Jitter, J): Indicates the change in packet delay, usually in milliseconds (ms). (6) Packet Loss Rate (PLR): Indicates the proportion of packet loss, usually in percentage (%).

[0079] The set of real-time values ​​of channel state parameters can be expressed as:

[0080] The calculation weight refers to the degree of influence of each channel state parameter on the channel state. According to the calculation weight corresponding to each channel state parameter and the real-time value of each channel state parameter, the estimated value of the channel state between the vehicle and the ground is determined, which can be expressed by the formula:

[0081] In the formula, is the channel state estimate, , , , , and are the calculation weights of each channel state parameter respectively.

[0082] The channel state estimation value combines various monitored channel state parameters to form a comprehensive indicator, which can be used to guide the adjustment of the compression rate. For example, the larger the channel state estimation value, the better the channel state, the better the communication quality, and the smaller the compression rate can be to transmit more data; the smaller the channel state estimation value, the larger the compression rate can be to transmit less data.

[0083] In order to better adapt to changes in channel conditions, the transmission ratio of key frames and common frames can be dynamically adjusted according to the channel state parameter channel state estimation value. The transmission ratio of key frames is inversely proportional to the channel state estimation value; the transmission ratio of common frames is proportional to the channel state estimation value.

[0084] Taking the transmission ratio of key frames as an example, it can be expressed as:

[0085] In the formula, is the transmission ratio of key frames, is the maximum value of the key frame transmission ratio, is the minimum value of the key frame transmission ratio, is the maximum value of the channel state estimation value, is the minimum value of the channel state estimate, is the real-time value of the channel state estimation value.

[0086] That is to say, when the channel state estimation value is high (the network condition is good), the key frame transmission ratio is low, because under good network conditions, more ordinary frames can be transmitted to improve video quality; when the channel state estimation value is low (the network condition is poor), the key frame transmission ratio is high to ensure that more important information is transmitted under poor network conditions, ensuring the continuity of the video and the delivery of important content.

[0087] The vehicle-ground collaborative video compression, restoration and transmission method provided in the embodiment of the present application determines the transmission ratio of key frames to ordinary frames in real time according to the channel state parameters between the vehicle and the ground, so that the video transmission is adapted to the network communication quality, thereby improving the transmission efficiency and stability of the video under various network communication conditions.

[0088] In some embodiments, generating an intermediate frame between adjacent transmission frames based on an optical flow and a visibility map between adjacent transmission frames includes: Performing optical flow calculation on adjacent first transmission frames and second transmission frames based on an optical flow calculation network to obtain a bidirectional optical flow between the first transmission frame and the second transmission frame; Based on the bidirectional optical flow, determining a first optical flow from the intermediate frame to the first transmission frame and a second optical flow from the intermediate frame to the second transmission frame; Processing the first optical flow and the second optical flow based on the optical flow interpolation network to determine a first visibility map of the intermediate frame relative to the first transmission frame and a second visibility map of the intermediate frame relative to the second transmission frame; Based on the first visibility map and the second visibility map, interpolation is performed between the first transmission frame and the second transmission frame to generate an intermediate frame.

[0089] Specifically, the present application interpolates between two transmission frames to generate an intermediate frame, ensuring that the generated video sequence is spatially and temporally coherent with the original video. The transmission frames may include key frames and ordinary frames, and interpolation is performed preferentially between key frames to generate intermediate frames. The interpolation method includes steps such as optical flow calculation, intermediate optical flow approximation, optical flow interpolation, visibility map calculation, and intermediate frame generation.

[0090] An optical flow calculation network can be used to perform optical flow calculation on adjacent first transmission frames and second transmission frames to obtain a bidirectional optical flow between the first transmission frame and the second transmission frame. The optical flow calculation network can be implemented using a FlowNet model to perform optical flow calculation and estimate the movement of pixels in the transmission frame.

[0091] The network structure of the optical flow calculation network can also adopt an autoencoder structure. The encoder part consists of multiple convolutional layers and Leaky ReLU activation functions, and the spatial dimension is reduced through the average pooling layer. The role of the encoder is to extract the features of the input image. The decoder part increases the spatial dimension through a bilinear upsampling layer and restores the original image size. The role of the decoder is to convert the features extracted by the encoder into optical flow estimation results.

[0092] The first transmission frame can be expressed as , the second transmission frame can be expressed as Inputting the optical flow calculation network can obtain bidirectional optical flow, which can be expressed as and . It can be expressed as:

[0093] According to the bidirectional optical flow, the first optical flow from the intermediate frame to the first transmission frame can be determined by interpolation estimation. And the second optical flow from the intermediate frame to the second transmission frame , which can be expressed as:

[0094] The value range of is usually a continuous value between 0 and 1, indicating the interpolation position between two key frames. For example, when When , it indicates that it is completely at the position of the first key frame; when , it indicates that it is completely at the position of the second key frame; , it indicates an intermediate frame between two key frames.

[0095] How many intermediate frames are inserted depends on the design requirements and algorithm settings of the system. The value of can be any value between 0 and 1, or multiple discrete values ​​(for example, t = 0.1, 0.2, ..., 0.9), which determines the number of intermediate frames inserted. Therefore, the range and number of t are flexible, and the number of intermediate frames can be set as needed.

[0096] The first optical flow and the second optical flow are processed according to the optical flow interpolation network to determine a first visibility map of the intermediate frame relative to the first transmission frame. and a second visibility map of the intermediate frame relative to the second transmission frame The first visibility map is used to identify which pixels in the first transmission frame are visible in the intermediate frame, and the second visibility map is used to identify which pixels in the second transmission frame are visible in the intermediate frame.

[0097] The optical flow interpolation network is mainly used to predict visibility maps. Its network structure can adopt an autoencoder structure, more specifically, a Unet network structure. The encoder is similar to the optical flow calculation network and is used to extract input features. The decoder is used to restore the spatial dimension by upsampling. Jump connections are used to connect the corresponding layers of the encoder and decoder to retain high-resolution information.

[0098] The visibility map is used to identify which pixels are visible in the intermediate frames to handle occlusion problems and can also satisfy the following formula:

[0099] This formula ensures that a pixel Visible in Not visible.

[0100] According to the first visibility map and the second visibility map, interpolation is performed between the first transmission frame and the second transmission frame to generate an intermediate frame. The generation process of the intermediate frame comprehensively utilizes the optical flow field and the visibility map to ensure the spatial and temporal consistency of the interpolated frame. The following are the detailed steps for generating the intermediate frame: Step 1: Optical flow mapping

[0101] For each pixel position in the intermediate frame ( , ), first through the optical flow field , and , To make sure it is in the previous frame and the next frame The corresponding position in and These mapped locations are usually not integer coordinates and may fall between pixels, thus requiring further processing to determine their pixel values.

[0102] Step 2: Bilinear interpolation calculation

[0103] Since the optical flow mapping position and There may be non-integer coordinates, and their pixel values ​​cannot be directly obtained. To solve this problem, bilinear interpolation is used to calculate the pixel values ​​at these locations.

[0104] The bilinear interpolation method calculates the pixel value of the non-integer position by using the pixel values ​​of the four nearest integer coordinate points around it. and the pixel value of the next frame , the interpolation process is as follows: (a) Selection and Get the pixel values ​​of the four nearest integer coordinates around them.

[0105] (b) Using the bilinear interpolation formula, first Direction interpolation, then Direction interpolation, we get and The pixel value at and .

[0106] Step 3: Visibility graph weighs contribution

[0107] Visibility graph and Indicates whether each pixel position in the intermediate frame can be clearly observed from the previous frame or the next frame. These visibility maps are used to weigh the contribution of the previous frame and the next frame to the intermediate frame pixels.

[0108] The value of the visibility map is between 0 and 1, indicating the visibility weight of each pixel. Through these weights, the pixel values ​​of the previous and next frames can be reasonably fused into the intermediate frame.

[0109] Step 4: Visibility graph weighs contribution

[0110] Final intermediate frame ( , ) is obtained by weighted averaging the pixel values ​​of the previous frame and the next frame:

[0111] In the formula, , as a normalization factor, is used to ensure that the weighted sum of the pixel values ​​of the previous and next frames is 1. This ensures that the pixel values ​​of the intermediate frames do not show unnatural brightness or color changes.

[0112] The intermediate frame generated by the above process is a reasonable fusion of the previous and next frames, combining the position of the optical flow field mapping and the weight of the visibility map, ensuring that the generated intermediate frame is visually smooth and natural, and can maintain high-quality interpolation effects when processing complex motion scenes.

[0113] The vehicle-ground collaborative video compression, restoration and transmission method provided in the embodiment of the present application generates intermediate frames between adjacent transmission frames based on the optical flow and visibility graph between adjacent transmission frames, can accurately predict the movement of objects, generate smooth and natural intermediate frames, and improve the quality of the video.

[0114] In some embodiments, an image sequence is generated based on the transmission frame and the intermediate frame, and super-resolution restoration is performed on the image sequence to obtain a restored video corresponding to the video to be transmitted at the ground station, including: generating an initial image sequence based on the transmission frame and the intermediate frame; Preprocessing the initial image sequence based on the image preprocessing network to obtain an image sequence; The image sequence is processed based on the super-resolution restoration network to obtain the restored frames corresponding to each frame in the image sequence; Based on the restored frames corresponding to each frame, a restored video corresponding to the video to be transmitted at the ground station is generated; Among them, the image preprocessing network includes multiple residual blocks; preprocessing includes denoising and artifact removal.

[0115] Specifically, in order to further improve the video quality, super-resolution restoration is required. Before super-resolution restoration, image preprocessing can be performed.

[0116] According to the time information of the transmission frame and the intermediate frame, the initial image sequence can be generated in time order. The image preprocessing network is used to preprocess the initial image sequence to obtain an image sequence.

[0117] The image preprocessing network consists of multiple residual blocks. The residual blocks can effectively learn and suppress noise and artifacts in the image, thereby generating a clean image sequence. The residual block can be expressed as:

[0118] In the formula, is the initial input image, is the output preprocessed image, is the function of the residual block, are the trainable parameters in the residual block.

[0119] The image sequence is processed according to the super-resolution restoration network to obtain the restored frames corresponding to each frame in the image sequence, and the restored frames corresponding to each frame are combined to generate the restored video corresponding to the video to be transmitted at the ground station.

[0120] The super-resolution restoration network aims to increase the resolution of an image, generating higher quality and clearer images. The network receives a cleaned image sequence and uses temporal and spatial information to improve image quality by processing the relationship between frames.

[0121] The super-resolution restoration network can use a two-stream convolutional neural network, and the model structure can include the following key parts: (1) Feature extraction layer: The convolutional layer extracts spatial and temporal features from the low-resolution image.

[0122] (2) Temporal and spatial information fusion layer: Use 3D convolution or recurrent neural network (RNN) to fuse feature information from different frames and capture the dynamic relationship between video frames.

[0123] (3) Upsampling layer: The low-resolution image is upsampled to the target resolution through techniques such as deconvolution or sub-pixel convolution.

[0124] (4) Reconstruction layer: The upsampled feature map is converted into the final high-resolution image through the convolution layer.

[0125] The vehicle-ground collaborative video compression, restoration and transmission method provided in the embodiment of the present application processes the image sequence according to the image preprocessing network and the super-resolution restoration network, thereby obtaining a clearer and more detailed restored image and improving the quality of the video.

[0126] In some embodiments, preprocessing the initial image sequence based on the image preprocessing network to obtain the image sequence includes: determining a current frame in the initial image sequence; Preprocess the current frame based on the image preprocessing network to obtain the current iterative processing result of the current frame; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is greater than a preset difference, continuing to preprocess the current iterative processing result of the current frame based on the image preprocessing network; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is less than a preset difference, the current iterative processing result is used as the preprocessing result of the current frame, and the next frame in the initial image sequence is determined as the current frame; An image sequence is determined based on the preprocessing results of each frame in the initial image sequence.

[0127] Specifically, the image preprocessing network processes each frame of the input sequence independently. After each frame is processed by the residual block, a high-quality image sequence is generated. Input frame image, C is the image preprocessing network, then the processed image It can be expressed as:

[0128] A single pass through the image preprocessing network may not completely remove all degradation (image quality degradation). To address this problem, the present application embodiment proposes a dynamic refinement mechanism. This mechanism continuously reduces the degradation in the input image by applying the image preprocessing network multiple times.

[0129] The steps of the dynamic refinement mechanism are: The current frame is preprocessed according to the image preprocessing network to obtain the current iterative processing result of the current frame; when the difference between the current iterative processing result and the previous iterative processing result of the current frame is greater than the preset difference, the current iterative processing result of the current frame is continued to be preprocessed according to the image preprocessing network; when the difference between the current iterative processing result and the previous iterative processing result of the current frame is less than the preset difference, the current iterative processing result is used as the preprocessing result of the current frame, and the next frame in the initial image sequence is determined as the current frame. It can be expressed by the formula:

[0130] In the formula, For the current frame ( The current iteration (frame) Iterations) processing results, is the previous iteration of the current frame ( Iterations) processing results, The next iteration of the current frame ( Iterations) processing results. It is the preset difference value used to control the condition for stopping the iteration. = .

[0131] According to the preprocessing results of each frame in the initial image sequence, the image sequence can be finally determined.

[0132] The vehicle-ground collaborative video compression, restoration and transmission method provided in the embodiment of the present application calculates the difference between the results of the current iteration and the previous iteration. When the difference is greater than a preset threshold, the preprocessing network is continuously applied until the difference is less than the threshold. This mechanism ensures that the input image gradually reduces degradation during multiple iterations, thereby improving the quality of the output image.

[0133] In some embodiments, during the training process of the super-resolution restoration network, random degradation processing is performed on the training samples.

[0134] Specifically, different degradation patterns can be randomly selected and applied during the training of the super-resolution restoration network to improve the generalization ability of the model. In each training iteration, operations such as blur, noise, compression artifacts, and color distortion are applied to the video sequence, which are implemented using OpenCV and PIL (Python Imaging Library). These operations are modeled through a random walk process.

[0135] The introduction of random degradation operations reduces I / O bottlenecks, improves data transmission efficiency, and shortens training time by up to 40%. This solution simulates multiple degradation conditions in the real world and enhances the adaptability and generalization ability of the model.

[0136] A noise enhancement module for the vehicle environment can also be added to simulate noise such as electronic equipment interference and environmental vibration, further improving the robustness of the model in the vehicle environment.

[0137] The device provided in the embodiment of the present application is described below. The device described below and the method described above can be referenced to each other.

[0138] Figure 2 This is one of the structural diagrams of the vehicle-ground collaborative video compression and restoration transmission system provided by this application, such as Figure 2 As shown, the system includes: The key frame extraction module 210 is used to extract features from each video frame in the video to be transmitted collected by the vehicle, perform clustering based on the extracted features, and determine the key frames in the video to be transmitted; The video transmission module 220 is used to determine the transmission ratio of key frames to common frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determine the transmission frames in the video to be transmitted based on the transmission ratio; An intermediate frame insertion module 230, configured to generate an intermediate frame between adjacent transmission frames based on the optical flow and visibility graph between adjacent transmission frames; The video restoration module 240 is used to generate an image sequence based on the transmission frame and the intermediate frame, perform super-resolution restoration on the image sequence, and obtain a restored video corresponding to the video to be transmitted at the ground station.

[0139] The vehicle-to-ground collaborative video transmission device provided in the embodiment of the present application extracts features from each video frame in the video to be transmitted collected by the vehicle, performs clustering based on the extracted features, and determines the key frames in the video to be transmitted; determines the transmission ratio of key frames to ordinary frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determines the transmission frames in the video to be transmitted based on the transmission ratio; generates intermediate frames between adjacent transmission frames based on the optical flow and visibility map between adjacent transmission frames; generates an image sequence based on the transmission frames and the intermediate frames, performs super-resolution restoration on the image sequence, and obtains the restored video corresponding to the video to be transmitted at the ground station; because the features of each video frame in the video to be transmitted are extracted and clustered, the key frames can be accurately extracted in complex video content, reducing The amount of transmitted data is reduced, thereby improving the transmission efficiency and quality of the video; since the transmission ratio of key frames to ordinary frames is determined in real time according to the channel state parameters between the vehicle and the ground, the video transmission is adapted to the network communication quality, thereby improving the transmission efficiency and stability of the video under various network communication conditions; since adjacent transmission frames are interpolated according to the optical flow and visibility map, the movement of the object can be accurately predicted, and smooth and natural intermediate frames can be generated, thereby improving the quality of the video; since the obtained transmission frames are restored with super-resolution, a clearer and more detailed restored image is obtained, thereby improving the quality of the video; through the combination of the above methods, the video compression and restoration are realized as a whole, the transmission cost is saved, and efficient and stable video transmission between the vehicle and the ground is realized.

[0140] In some embodiments, the key frame extraction module is used to: Based on the feature extraction network, feature extraction is performed on each video frame in the video to be transmitted to obtain image features corresponding to each video frame; Cluster the image features corresponding to each video frame to obtain multiple classes; The video frame corresponding to the image feature closest to the cluster center in each class is used as the key frame; Among them, the feature extraction network is trained based on the sample video to be transmitted and the restored video corresponding to the sample video to be transmitted; the feature extraction network includes at least one attention feature extraction layer.

[0141] In some embodiments, the video transmission module is used to: Obtaining real-time values ​​of multiple channel status parameters between the vehicle and the ground; Determine the estimated value of the channel state between the vehicle and the ground based on the calculation weights corresponding to the various channel state parameters and the real-time values ​​of the various channel state parameters; Based on the channel state estimation value, determining the transmission ratio of key frames and the transmission ratio of common frames; The transmission ratio of key frames is inversely proportional to the channel state estimation value; the transmission ratio of common frames is directly proportional to the channel state estimation value.

[0142] In some embodiments, the intermediate frame insertion module is used to: Performing optical flow calculation on adjacent first transmission frames and second transmission frames based on an optical flow calculation network to obtain a bidirectional optical flow between the first transmission frame and the second transmission frame; Based on the bidirectional optical flow, determining a first optical flow from the intermediate frame to the first transmission frame and a second optical flow from the intermediate frame to the second transmission frame; Processing the first optical flow and the second optical flow based on the optical flow interpolation network to determine a first visibility map of the intermediate frame relative to the first transmission frame and a second visibility map of the intermediate frame relative to the second transmission frame; Based on the first visibility map and the second visibility map, interpolation is performed between the first transmission frame and the second transmission frame to generate an intermediate frame.

[0143] In some embodiments, the video restoration module is used to: generating an initial image sequence based on the transmission frame and the intermediate frame; Preprocessing the initial image sequence based on the image preprocessing network to obtain an image sequence; The image sequence is processed based on the super-resolution restoration network to obtain the restored frames corresponding to each frame in the image sequence; Based on the restored frames corresponding to each frame, a restored video corresponding to the video to be transmitted at the ground station is generated; Among them, the image preprocessing network includes multiple residual blocks; preprocessing includes denoising and artifact removal.

[0144] In some embodiments, the video restoration module is used to: determining a current frame in the initial image sequence; Preprocess the current frame based on the image preprocessing network to obtain the current iterative processing result of the current frame; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is greater than a preset difference, continuing to preprocess the current iterative processing result of the current frame based on the image preprocessing network; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is less than a preset difference, the current iterative processing result is used as the preprocessing result of the current frame, and the next frame in the initial image sequence is determined as the current frame; An image sequence is determined based on the preprocessing results of each frame in the initial image sequence.

[0145] In some embodiments, the system is also used to: During the training process of the super-resolution restoration network, the training samples are randomly degraded.

[0146] Figure 3 This is the second structural diagram of the vehicle-ground collaborative video compression and restoration transmission system provided by this application, such as Figure 3 As shown, the system includes: (1) Car camera

[0147] The vehicle camera module is responsible for capturing high-definition video frames inside and outside the vehicle in real time and performing preliminary image preprocessing (such as denoising, brightness and contrast adjustment) to ensure the image quality for subsequent encoding and transmission. Through high frame rate and high-resolution lenses, the dynamic environment around the vehicle is captured to provide high-quality video input for the encoding module.

[0148] (2) Encoding module

[0149] The encoding module extracts key frames from the pre-processed video frames transmitted from the on-board camera, accepts the guidance compression rate from the adaptive adjustment module, and uses the deep autoencoder and attention mechanism to perform dynamic video compression to improve transmission efficiency.

[0150] (3) Adaptive adjustment module

[0151] The adaptive adjustment module receives channel status information from the transmission module, evaluates the current network status, dynamically adjusts the video compression rate and transmission content, and guides the encoding module and transmission module to make corresponding adjustments through the feedback mechanism, ensuring efficient and stable video transmission under various network conditions.

[0152] (4) Transmission module

[0153] The transmission module is responsible for real-time monitoring of the network channel status, and adjusting the transmission content according to the guidance of the adaptive adjustment module. It transmits the processed video data efficiently and stably to the ground receiving station through the wireless communication link, and feeds back the channel status information to the adaptive adjustment module.

[0154] (5) Ground receiving station

[0155] The ground receiving station receives the video data sent by the transmission module through a wireless communication link, decodes it to restore the original video content, and transmits the decoded video data to the super-resolution restoration module for subsequent processing.

[0156] (6) Video generation and restoration module

[0157] The generation module mainly completes the frame insertion work, receives the decoded low-resolution video data, uses the convolutional neural network (CNN) and refinement network of the U-Net architecture, generates high-quality intermediate frames through bidirectional optical flow and visibility maps, and finally obtains complete video frames.

[0158] On this basis, the restoration module uses a deep learning model to perform super-resolution processing, restore low-resolution video frames to high-resolution videos, output high-quality videos for monitoring and display, and improve the clarity and detail of the video.

[0159] Figure 4 is a schematic diagram of the structure of the electronic device provided by this application, such as Figure 4 As shown, the electronic device may include: a processor (Processor) 410, a communication interface (Communications Interface) 420, a memory (Memory) 430 and a communication bus (Communications Bus) 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic command in the memory 430 to execute the method described in the above embodiment, for example: Feature extraction is performed on each video frame in the video to be transmitted collected by the vehicle, and clustering is performed based on the extracted features to determine the key frames in the video to be transmitted; the transmission ratio of key frames to ordinary frames in the video to be transmitted is determined based on the channel state parameters between the vehicle and the ground, and the transmission frames in the video to be transmitted are determined based on the transmission ratio; based on the optical flow and visibility map between adjacent transmission frames, intermediate frames between adjacent transmission frames are generated; an image sequence is generated based on the transmission frames and the intermediate frames, and the image sequence is super-resolution restored to obtain the restored video corresponding to the video to be transmitted at the ground station.

[0160] In addition, the logical commands in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several commands to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0161] The processor in the electronic device provided in the embodiment of the present application can call the logic instructions in the memory to implement the above method. Its specific implementation method is consistent with the implementation method of the aforementioned method and can achieve the same beneficial effects, which will not be repeated here.

[0162] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided in the above embodiments are implemented.

[0163] Its specific implementation is consistent with the aforementioned method implementation and can achieve the same beneficial effects, so it will not be repeated here.

[0164] An embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0165] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A vehicle-ground collaborative video compression, restoration and transmission method, characterized in that: include: Extracting features from each video frame in the video to be transmitted collected by the vehicle, clustering based on the extracted features, and determining key frames in the video to be transmitted; Determine a transmission ratio of key frames to common frames in the video to be transmitted based on a channel state parameter between the vehicle and the ground, and determine a transmission frame in the video to be transmitted based on the transmission ratio; Based on the optical flow and visibility map between adjacent transmission frames, generate an intermediate frame between the adjacent transmission frames; An image sequence is generated based on the transmission frame and the intermediate frame, and super-resolution restoration is performed on the image sequence to obtain a restored video corresponding to the video to be transmitted at the ground station.

2. The vehicle-ground collaborative video compression, restoration and transmission method according to claim 1 is characterized in that: The step of extracting features from each video frame in the video to be transmitted collected by the vehicle, clustering based on the extracted features, and determining key frames in the video to be transmitted includes: Extracting features from each video frame in the video to be transmitted based on a feature extraction network to obtain image features corresponding to each video frame; Cluster the image features corresponding to each video frame to obtain multiple classes; The video frame corresponding to the image feature closest to the cluster center in each class is used as the key frame; Wherein, the feature extraction network is trained based on the sample video to be transmitted and the restored video corresponding to the sample video to be transmitted; the feature extraction network includes at least one attention feature extraction layer.

3. The vehicle-ground collaborative video compression, restoration and transmission method according to claim 1 is characterized in that: The determining the transmission ratio of key frames to common frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determining the transmission frames in the video to be transmitted based on the transmission ratio, comprises: Obtaining real-time values ​​of multiple channel status parameters between the vehicle and the ground; Determine the estimated value of the channel state between the vehicle and the ground based on the calculation weights corresponding to the various channel state parameters and the real-time values ​​of the various channel state parameters; Determining a transmission ratio of the key frames and a transmission ratio of the common frames based on the channel state estimation value; The transmission ratio of the key frames is inversely proportional to the channel state estimation value; and the transmission ratio of the common frames is directly proportional to the channel state estimation value.

4. The vehicle-ground collaborative video compression, restoration and transmission method according to claim 1 is characterized in that: The step of generating an intermediate frame between adjacent transmission frames based on an optical flow and a visibility graph between adjacent transmission frames comprises: Performing optical flow calculation on adjacent first transmission frames and second transmission frames based on an optical flow calculation network to obtain a bidirectional optical flow between the first transmission frame and the second transmission frame; Based on the bidirectional optical flow, determining a first optical flow from the intermediate frame to the first transmission frame and a second optical flow from the intermediate frame to the second transmission frame; Processing the first optical flow and the second optical flow based on an optical flow interpolation network to determine a first visibility map of the intermediate frame relative to the first transmission frame and a second visibility map of the intermediate frame relative to the second transmission frame; Based on the first visibility map and the second visibility map, interpolation is performed between the first transmission frame and the second transmission frame to generate the intermediate frame.

5. The vehicle-ground collaborative video compression, restoration and transmission method according to claim 1 is characterized in that: The step of generating an image sequence based on the transmission frame and the intermediate frame, and performing super-resolution restoration on the image sequence to obtain a restored video corresponding to the video to be transmitted at the ground station includes: generating an initial image sequence based on the transmission frame and the intermediate frame; Preprocessing the initial image sequence based on an image preprocessing network to obtain the image sequence; Processing the image sequence based on a super-resolution restoration network to obtain a restored frame corresponding to each frame in the image sequence; Based on the restored frames corresponding to the frames, generating the restored video corresponding to the video to be transmitted at the ground station; Wherein, the image preprocessing network includes multiple residual blocks; the preprocessing includes noise reduction and artifact removal.

6. The vehicle-ground collaborative video compression, restoration and transmission method according to claim 5 is characterized in that: The preprocessing of the initial image sequence based on the image preprocessing network to obtain the image sequence includes: Determining a current frame in the initial image sequence; Preprocessing the current frame based on the image preprocessing network to obtain a current iterative processing result of the current frame; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is greater than a preset difference, continuing to preprocess the current iterative processing result of the current frame based on the image preprocessing network; When the difference between the current iterative processing result and the previous iterative processing result of the current frame is less than a preset difference, the current iterative processing result is used as the preprocessing result of the current frame, and the next frame in the initial image sequence is determined as the current frame; The image sequence is determined based on the preprocessing results of each frame in the initial image sequence.

7. The vehicle-ground collaborative video compression, restoration and transmission method according to claim 5 is characterized in that: The method further comprises: During the training process of the super-resolution restoration network, random degradation processing is performed on the training samples.

8. A vehicle-ground collaborative video compression and restoration transmission system, characterized in that: include: A key frame extraction module is used to extract features from each video frame in the video to be transmitted collected by the vehicle, cluster the extracted features, and determine the key frames in the video to be transmitted; A video transmission module, used to determine the transmission ratio of key frames to common frames in the video to be transmitted based on the channel state parameters between the vehicle and the ground, and determine the transmission frames in the video to be transmitted based on the transmission ratio; An intermediate frame insertion module, used for generating an intermediate frame between adjacent transmission frames based on the optical flow and visibility graph between the adjacent transmission frames; The video restoration module is used to generate an image sequence based on the transmission frame and the intermediate frame, perform super-resolution restoration on the image sequence, and obtain a restored video corresponding to the video to be transmitted at the ground station.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the vehicle-ground collaborative video compression, restoration and transmission method described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vehicle-ground collaborative video compression, restoration and transmission method described in any one of claims 1 to 7 is implemented.

Images