Remote desktop video coding and decoding method based on neural network multi-head attention mechanism

Abstract

The invention relates to the technical field of video coding and decoding, and discloses a remote desktop video coding and decoding method based on a neural network multi-head attention mechanism. Comprising the following steps: dividing a screen of a remote desktop into a plurality of areas by utilizing a multi-head attention mechanism; performing grading identification according to the activeness and the importance degree of each area in the screen; the encoder performs encoding sorting on each area according to the grading identifier, formulates an encoding strategy, and dynamically adjusts the encoding strategy based on a feedback mechanism; generating a coded video stream according to a formulated coding strategy, and extracting attached information, related to original coded data, of each region; synchronously sending the generated coded video stream and the attached information to a receiving end; and the decoder decodes each area according to the coding sequence, original coding data is recovered, and the remote desktop picture is recovered. The multi-head attention mechanism is utilized to perform intelligent grading and associated coding on the screen area, and the problems of low efficiency, redundant transmission and high bandwidth occupation in a traditional coding and decoding method are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding and decoding, and specifically to a remote desktop video coding and decoding method based on a neural network multi-head attention mechanism. Background Art

[0002] With the wide application of remote desktop technology, especially in environments with high-definition video transmission and low-latency requirements, how to efficiently compress remote desktop videos while maintaining picture quality has always been a difficult point in technological development. Although traditional video coding and decoding methods can meet certain data compression requirements, due to the dynamic changes in the content of the remote desktop screen, especially when it is in a frequent dynamic change state, the data volume generated by traditional coding methods shows an explosive growth, and the bit rate soars accordingly. This undoubtedly exerts a heavy pressure on the originally limited network bandwidth beyond its bearing capacity, easily causing transmission congestion; on the other hand, the decoding time-consuming is greatly extended, directly resulting in frequent lags during video playback, greatly damaging the user experience, and becoming a key bottleneck hindering the further expansion of remote desktop technology applications.

[0003] Most of the existing remote desktop video coding and decoding solutions are constructed with traditional image coding algorithms, such as H.264 and HEVC algorithms, as the core. In past practical applications, these classic algorithms have, to a certain extent, achieved the compression of video data and played a phased role in reducing the amount of transmitted data. However, they lack the ability to differentially process the screen areas of the remote desktop. In this complex visual presentation environment of the remote desktop, the screen areas are naturally divided into many categories, such as operation button areas, text display areas, dynamic image display areas, etc., according to different functions and content natures. And for each type of area, there are huge differences in the requirements for the fineness of the picture quality and the frequency of content updates. The existing coding solutions are unable to intelligently identify and judge based on the semantic information contained in each screen area, and thus it is difficult to dynamically and efficiently adjust the coding strategy to achieve refined and optimized resource allocation. Summary of the Invention

[0004] This application provides a remote desktop video coding and decoding method based on a neural network multi-head attention mechanism, which can significantly reduce the data volume and bit rate of remote desktop video transmission while ensuring high-quality pictures, and improve the efficiency and user experience of remote desktop applications. By using the multi-head attention mechanism to intelligently classify and associatively code the screen areas, it effectively solves the problems of inefficiency, redundant transmission, and high bandwidth occupancy existing in traditional coding and decoding methods. The method includes the following steps:

[0005] S1, using the multi-head attention mechanism, subdivide the screen of the remote desktop into multiple areas;

[0006] S2. Analyze each region, and grade and label it according to its activity and importance on the screen.

[0007] S3. The encoder sorts the encoding order of each region according to the grading label, formulates the encoding strategy for each region correspondingly, realizes identifying active regions, preferentially encoding active regions, and dynamically adjusts the encoding strategy based on the feedback mechanism.

[0008] S4. According to the formulated encoding strategy, compress the original encoding data to be transmitted for each region, generate an encoded video stream, and correspondingly extract the attached information related to the original encoding data for each region.

[0009] S5. Synchronously send the generated encoded video stream and the attached information to the receiving end.

[0010] S6. The decoder decodes each region according to the encoding order to restore the original encoding data, so as to realize the restoration of the remote desktop image.

[0011] Preferably, obtain the remote desktop video frame data, preprocess it to obtain the original video frame data set, and divide the original video frame data set into a training set, a validation set, and a test set.

[0012] Based on the training set, the validation set, and the test set, respectively train the encoding model and the decoding model in the encoder and the decoder through a deep learning model.

[0013] Preferably, the training process of the encoding model is as follows:

[0014] (1) Use the obtained original video frame data set.

[0015] (2) Design and build the network structure of the encoding model.

[0016] (3) Use the multi-head attention mechanism to assign different attention weights to each region according to the activity of each region, and be responsible for processing the regional features of the screen; use the convolutional neural network to extract image features.

[0017] (4) Calculate the network loss, construct a loss function, and calculate the loss between the prediction result and the true result.

[0018] (5) Set the network training parameters, and the training parameters include: learning rate and number of iterations.

[0019] (6) Use the training set to train the encoding model built in (2). After each training ends, obtain a corresponding weight file, input the validation set, evaluate the weight file using the Diss similarity coefficient as the evaluation criterion, and finally save the weight file with the largest Diss similarity coefficient value.

[0020] (7) Input the test set into the encoding model, and use the largest weight file obtained in (6) to extract video features, obtaining the compressed encoded video stream generated for each region and the corresponding accompanying information;

[0021] (8) Evaluate the performance of the encoding model through the mean intersection over union metric of the encoded video stream and the dataset.

[0022] Preferably, the training process of the decoding model is as follows:

[0023] (1) Use the obtained original video frame dataset;

[0024] (2) Design and build the network structure of the decoding model;

[0025] (3) Obtain the original encoded data and the corresponding encoded video stream, where the encoded video stream includes compressed video frame data and the accompanying information for each region;

[0026] (4) Input the training set into the built decoding model. The decoding model performs forward propagation calculation according to a preset loss function to obtain the decoded video frame data;

[0027] (5) Compare the decoded video frame data with the corresponding original video frame data in the training set, calculate the loss value, and backpropagate the loss value to each layer of the model through the backpropagation algorithm to update the decoding model parameters to reduce the difference from the original video frame data;

[0028] (6) Use the validation set to evaluate the decoding model, adjust the hyperparameters according to the validation results. When the structural similarity index value on the validation set no longer improves, reduce the learning rate to promote model convergence;

[0029] (7) During and after the training process of the decoding model, continuously optimize the quality of the decoded video frame data;

[0030] (8) Accurately restore the active region and the associated system region according to the accompanying information for each region in the encoded video stream.

[0031] Preferably, the recognition steps of the active region include:

[0032] Process the unit video frame using a convolutional neural network containing multiple convolutional layers with different specifications to extract the features of the video image of the unit video frame;

[0033] Input the extracted features into a multi-layer perceptron containing multiple fully connected layers with the number of neurons decreasing in sequence, and transform the extracted features into an embedding representation through the multi-layer perceptron;

[0034] Based on the embedding representation as a vector, generate query, key, and value vectors by setting different weight matrices;

[0035] Perform parallel calculations of multi - head attention according to their respective corresponding queries, keys, and value vectors;

[0036] Concatenate the values obtained by each attention head to obtain a comprehensive feature matrix. Input the comprehensive feature matrix into a fully - connected layer containing one or more layers of neurons, perform a linear transformation on the input features through a weight matrix and a bias vector, calculate the attention weights, and dynamically adjust them during training;

[0037] Determine the active region according to whether the calculated attention weight is higher than the set threshold.

[0038] Preferably, the attached information includes: motion information, position change, importance weight, region size change, and difference in region content.

[0039] Preferably, the generation process of the attached information is as follows:

[0040] The encoder calculates the motion vectors of each region through a motion estimation algorithm and generates the motion information of each region respectively;

[0041] The encoder generates attached position information based on the coordinate changes of each region, the movement path on the screen, and the position changes between multiple data frames;

[0042] The encoder obtains the attention weights of each region through a multi - head attention mechanism and allocates the bitrates of the encodings of each region correspondingly according to the influence of the attention weights;

[0043] The encoder calculates the pixel differences and structural change information of each region based on the difference changes between the unit video frame and the adjacent video frames, and generates the corresponding difference information.

[0044] Preferably, the remote desktop video encoding and decoding method based on the neural network multi - head attention mechanism is applied to a remote desktop environment.

[0045] Compared with the prior art, the beneficial effects of this application are as follows:

[0046] Based on the multi - head attention mechanism, this application divides the screen of the remote desktop into multiple regions, grades and identifies each region according to its activity and importance in the screen, and sorts the encoding order of each region according to the identification, so as to achieve that the active region and the associated system region will be preferentially encoded. In this way, the encoder can more accurately identify the important regions and perform high - quality encoding on them, while the unimportant regions are compressed with a lower encoding quality to reduce the data volume; to enable the encoder to accurately identify and preferentially encode the important regions in the remote desktop, reduce the transmission of redundant data, and effectively reduce the bitrate of the video stream;

[0047] Meanwhile, according to the grading identifiers of each region, corresponding coding strategies are formulated to achieve refined coding; and compression is performed according to the coding strategies, and the corresponding additional information of each region is transmitted to the decoder synchronously. When the decoder receives the coded data, it restores the content of the important regions on the screen through the accompanying information, and more accurately restores the screen content according to these additional information. By focusing on the active regions and associated system regions on the screen, more details and key information are retained during coding to ensure that the picture quality of the remote desktop is not affected, thereby improving the decoding effect and efficiency;

[0048] During transmission, by transmitting the accompanying information together with the coded video stream, the decoder can accurately restore the picture, avoiding redundant information transmission, thus significantly reducing the required bandwidth.

[0049] This application dynamically adjusts the coding strategy based on the feedback mechanism by detecting in real time which regions on the current screen are active and which regions are static. According to this feedback, the multi-head attention mechanism of the encoder can adjust the bitrate allocation of different regions in real time to maximize the compression efficiency of the video stream while ensuring that the user experience is not affected.

[0050] This application can reduce the decoding delay and improve the video transmission experience of the remote desktop by optimizing the coding and decoding strategies, and is applicable to low-latency and high-quality video transmission scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is the flow chart of the remote desktop video encoding and decoding based on the multi-head attention mechanism in this application;

[0052] Figure 2 It is the schematic diagram of the preparation process for encoding model training in this application;

[0053] Figure 3 It is the schematic diagram of the training process for the encoding and decoding model in this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] A method for remote desktop video encoding and decoding based on the multi-head attention mechanism of a neural network, the method comprising the following steps:

[0055] S1, using the multi-head attention mechanism, dividing the screen of the remote desktop into multiple regions;

[0056] In this step, the multi-head attention mechanism divides the screen into multiple regions according to the dynamic characteristics and time series characteristics of the elements in the region, so as to perform differential coding according to the importance of different regions in the follow-up;

[0057] S2. Analyze each region, and classify and label it according to its activity level and importance in the screen.

[0058] S3. The encoder sorts the encoding order of each region according to the classification label, formulates the encoding strategy for each region accordingly, realizes the identification of active regions, preferentially encodes active regions, and dynamically adjusts the encoding strategy based on the feedback mechanism.

[0059] S4. Compress the original encoded data to be transmitted for each region according to the formulated encoding strategy, generate an encoded video stream, and extract the accompanying information related to the original encoded data for each region accordingly.

[0060] S5. Synchronously send the generated encoded video stream and the accompanying information to the receiving end.

[0061] S6. The decoder decodes each region according to the encoding order to restore the original encoded data, so as to realize the restoration of the remote desktop screen.

[0062] It should be noted that:

[0063] In this application, the classification label is used to represent the priority, activity level, and importance of video content.

[0064] The encoder formulates the encoding strategy for each region according to the classification label specifically means that: the encoder performs the allocation and dynamic adjustment of the bit rate for each region based on the classification of the region's label, as well as the motion estimation, position tracking prediction, and compensation for each region.

[0065] In this application, the multi-head attention mechanism calculates in parallel through multiple independent attention heads to capture different feature information in the image. Each attention head focuses on different regions of the image. By dynamically adjusting the attention degree of each region, the allocation of attention for each region is dynamically adjusted, so that when the screen content changes violently (such as mouse clicks, window changes, etc.), more resources can be allocated to the dynamically changing regions to ensure the retention of key information; at the same time, during the encoding process, the multi-head attention mechanism encoder can allocate more bit rate to the dynamic regions; for relatively static backgrounds, the allocated bit rate can be reduced accordingly, thereby optimizing the encoding efficiency. This intelligent adjustment avoids the inefficiency problem caused by using the same encoding bit rate for all regions in traditional encoding methods.

[0066] Meanwhile, in the remote desktop scenario, changes in user operations and screen content occur in real time. Through the designed real-time feedback mechanism, it is detected which areas of the current screen are active and which are static. Based on these feedbacks, the multi-head attention mechanism of the encoder can adjust the bitrate allocation of different areas in real time, maximizing the compression efficiency of the video stream while ensuring that the user experience is not affected. And the generated encoded video stream and additional information are synchronously sent to the receiving end;

[0067] Since the additional information includes the motion information, position changes, etc. of the area, during decoding, the decoder first restores the content of the important areas on the screen through the accompanying information, and then accurately restores the screen content according to the additional information transmitted synchronously with the encoded video stream, thereby improving the decoding effect and efficiency. The multi-head attention mechanism ensures better reconstruction of these areas during decoding, maintaining the fineness and important details of the picture while avoiding unnecessary redundant information transmission;

[0068] In the application, according to the hierarchical identification of each area, the encoder will adopt different encoding parameters and compression algorithms. For highly active areas, more refined encoding parameters and a higher bitrate will be used to ensure the information integrity of the active areas; for low-active areas, a relatively simple encoding strategy will be adopted to improve the overall encoding efficiency. The dynamic adjustment of this encoding strategy is based on the feedback information from the decoding end. The decoder will send information about the decoding effect and the display quality of the user end to the encoder, and the encoder will re-evaluate the activity and importance of each area according to this feedback information and dynamically adjust the encoding strategy;

[0069] In this application, the process of the encoder dynamically adjusting the encoding strategy based on the feedback mechanism is as follows:

[0070] (1) Video frame data acquisition and encoding:

[0071] Video acquisition: Capture video frames through a camera or other video sources and convert them into digital signals.

[0072] Initial encoding: The encoder performs initial compression encoding on the video frames to generate a compressed video frame data stream. This step usually uses standard video encoding formats (such as H.264, H.265, etc.).

[0073] (2) Transmission and reception:

[0074] Data transmission: The encoded video frame data is transmitted to the receiving end (such as the user's device) through the network.

[0075] Receiving end feedback: The receiving end decodes the received video frame data and plays the video. At the same time, the receiving end will collect quality information of video playback, such as packet loss rate, latency, frame rate, image quality, etc.

[0076] (3) Feedback information collection:

[0077] Quality assessment: The receiving end sends quality reports to the sending end through feedback mechanisms such as RTCP (Real - Time Transport Control Protocol). These reports contain various problems that occur during video transmission, such as packet loss, latency, jitter, etc.

[0078] Performance monitoring: The sending end receives and analyzes this feedback information to evaluate the quality and performance of the current video transmission.

[0079] (4) Encoding strategy adjustment:

[0080] Dynamic adjustment of encoding parameters: Based on the feedback information, the encoder dynamically adjusts the encoding strategy and parameters. For example:

[0081] Bitrate control: If the feedback shows that the current network bandwidth is insufficient, the encoder can reduce the video bitrate to reduce the amount of data, thereby reducing the packet loss rate and latency.

[0082] Frame rate adjustment: If the feedback shows that the current frame rate is too high, resulting in increased latency, the encoder can reduce the frame rate to improve the transmission efficiency.

[0083] Resolution adjustment: If the feedback shows that the current resolution is too high, resulting in a decrease in image quality, the encoder can reduce the resolution to improve the image quality.

[0084] Error recovery: If the feedback shows that there is packet loss or errors, the encoder can enable error recovery mechanisms such as FEC (Forward Error Correction) or retransmission requests to improve the transmission reliability.

[0085] (5) Real - time monitoring and optimization:

[0086] Real - time monitoring: The encoder continuously monitors the feedback information and dynamically adjusts the encoding strategy according to the real - time changing network conditions and video quality.

[0087] Optimization strategy: The encoder optimizes the encoding parameters based on historical data and current feedback to achieve the best video transmission effect.

[0088] (6) Continuous improvement and adaptation:

[0089] Adaptive algorithm: The encoder adopts an adaptive algorithm to continuously improve the encoding strategy based on long - term feedback information. For example, using machine - learning algorithms to predict network conditions and adjust encoding parameters in advance.

[0090] Dynamic adaptation: The encoder can quickly adapt to new environmental conditions according to the dynamic changes in network conditions, ensuring the stability and quality of video transmission.

[0091] (7) Feedback loop:

[0092] Loop feedback: A continuous feedback loop is formed between the encoder and the receiver. The encoder continuously adjusts the encoding strategy according to the feedback information from the receiver, while the receiver continuously provides new feedback information. This loop process ensures the continuous optimization of video transmission quality.

[0093] Furthermore, obtain the remote desktop video frame data, preprocess it to obtain the original video frame dataset, and divide the original video frame dataset into a training set, a validation set, and a test set.

[0094] Train the encoding model and the decoding model in the encoder and decoder respectively based on the deep learning model.

[0095] It should be noted that:

[0096] In this application, the encoding and decoding models are trained through the deep learning model, and the multi-head attention mechanism is used to achieve dynamic partitioning and associated encoding. Through the training of a large amount of remote desktop video frame data, the encoder and decoder can adaptively adjust the encoding strategy, accurately identify the features of each area on the screen, and thus greatly reduce the amount of transmitted data while ensuring the image quality.

[0097] Furthermore, the training process of the encoding model is as follows:

[0098] (1) Preprocess the obtained remote desktop video frame data to obtain the original video frame dataset, and divide the original video frame dataset into a training set, a validation set, and a test set.

[0099] (2) Design and build the network structure of the encoding model.

[0100] (3) Use the multi-head attention mechanism to assign different attention weights to each area according to the activity of each area, and be responsible for processing the regional features of the screen; use the convolutional neural network to extract image features.

[0101] (4) Calculate the network loss, construct the loss function, and calculate the loss between the prediction result and the real result.

[0102] (5) Set the network training parameters, and the training parameters include: learning rate and number of iterations.

[0103] (6) Use the training set obtained in (1) to train the encoding model built in (2). After each training, a corresponding weight file is obtained. Input the validation set in (1), evaluate the weight file using the Diss similarity coefficient as the evaluation criterion, and finally save the weight file with the largest similarity coefficient value.

[0104] (7) Input the test set in (1) into the encoding model, use the maximum weight file obtained in (6) to extract video features, and obtain the compressed encoded video stream generated by each region and the corresponding additional information;

[0105] (8) The average intersection-over-parameter ratio of the encoded video stream and the data set is used to evaluate the performance of the coding model.

[0106] It should be noted that:

[0107] In this application, the design process of the network structure of the encoding model is:

[0108] Input layer: Input the video frame data into the convolutional neural network (CNN) for feature extraction. CNN can automatically extract low-level visual features from images, such as edges, textures, colors, etc.

[0109] Multi-head attention mechanism layer: The features extracted by CNN are input into the multi-head attention mechanism. The multi-head attention mechanism focuses on the features of different areas through multiple independent attention heads. Each head is responsible for a different area and outputs a weighted feature map. When encoding remote desktop videos, multi-head attention can dynamically adjust the encoding bit allocation of each area according to the importance of the content. For example, for the mouse activity area, the encoder will allocate more bit rate to it, while for the static area, it will reduce the bit rate, thereby achieving the effect of data compression;

[0110] Aggregation layer: Aggregate the outputs of different attention heads (such as weighted average or concatenation) to obtain a global feature that combines multiple different feature representations;

[0111] Compression layer: The feature map after multi-head attention processing will pass through the compression network (such as quantization and transform encoder) to convert it into a coding format suitable for transmission.

[0112] Furthermore, the training process of the decoding model is:

[0113] (1) Preprocessing the acquired remote desktop video frame data to obtain an original video frame dataset, and dividing the dataset into a training set, a validation set, and a test set;

[0114] (2) Design and build the network structure of the decoding model;

[0115] (3) obtaining original coded data and a corresponding coded video stream, wherein the coded video stream includes compressed video frame data and corresponding region incidental information;

[0116] (4) Input the encoded data in the training set into the built decoding model, and the decoding model performs forward propagation calculation according to the preset loss function to obtain the decoded video frame data;

[0117] (5) Compare the decoded video frame data with the corresponding original encoded video frame data in the training set, calculate the loss value, and backpropagate the loss value to each layer of the model through the backpropagation algorithm to update the model parameters, so as to reduce the difference from the original video;

[0118] (6) Evaluate the model using the validation set and adjust the hyperparameters according to the validation results. For example, when the structural similarity index value on the validation set no longer improves, reduce the learning rate to promote model convergence;

[0119] (7) During and after the model training process, continuously optimize the quality of the decoded video frame data;

[0120] (8) Accurately restore the active area and the associated system area according to the area identification information in the encoded video stream.

[0121] It should be noted that:

[0122] In this application, the design of the network structure of the decoding model should be symmetric with the network structure of the encoding model, and the size and content of the image can be gradually restored through deconvolution (or upsampling) operations.

[0123] The decoding process is as follows: The decoder receives the encoded compressed data and decodes it. The decoding model can gradually restore the size and content of the image through deconvolution (or upsampling) operations.

[0124] The decoder will process each area during the decoding process based on the multi-head attention mechanism and try to restore the important details in the image. For those areas that are assigned more bits during encoding, the decoder will focus on restoring finer details and retain as much important visual information as possible during the decoding process.

[0125] Further, the steps for identifying the active area include:

[0126] Process the unit video frame using a convolutional neural network containing multiple convolutional layers with different specifications to extract the features of the video image of the unit video frame;

[0127] Input the extracted features into a multi-layer perceptron containing multiple fully connected layers with the number of neurons decreasing in sequence, and convert the extracted features into an embedding representation through the multi-layer perceptron;

[0128] Based on the embedding representation as a vector basis, generate query, key, and value vectors by setting different weight matrices;

[0129] Perform parallel calculations of multi-head attention according to their respective corresponding query, key, and value vectors;

[0130] Concatenate the values calculated by each attention head to obtain a comprehensive feature matrix. Input the comprehensive feature matrix into a fully connected layer containing one or more layers of neurons, perform a linear transformation on the input features through a weight matrix and a bias vector, calculate the attention weights, and dynamically adjust them during training;

[0131] Determine the active regions based on whether the calculated attention weights are higher than a set threshold. Specifically, it is achieved by calculating the attention weights and comparing them with a preset threshold. If the weight is higher than the threshold, the region is regarded as active.

[0132] Furthermore, the accompanying information includes: motion information, position changes, importance weights, region size changes, and differences in region content.

[0133] Furthermore, the generation process of the accompanying information is as follows:

[0134] The encoder calculates the motion vectors of each region through a motion estimation algorithm and generates the motion information of each region respectively;

[0135] The encoder generates the accompanying position information based on the coordinate changes of each region, the movement path on the screen, and the position changes between multiple data frames;

[0136] The encoder obtains the attention weights of each region through a multi-head attention mechanism and allocates the encoding bitrates correspondingly according to the influence of the attention weights;

[0137] The encoder calculates the pixel differences and structural change information of each region based on the differences between the unit video frame and adjacent video frames, and generates the corresponding difference information.

[0138] It should be noted that:

[0139] In this application, for each recognized active region, the encoder will correspondingly generate the accompanying information of each active region, and the accompanying information helps the decoder accurately reconstruct the content of each active region during the subsequent decoding process. The accompanying information mainly includes the following types:

[0140] Motion information: used to accurately represent the motion vector of the active region, which can precisely indicate the displacement direction and change direction of the region in the picture. This is crucial for accurately capturing dynamic region behaviors, such as mouse dragging operations, window moving processes, etc., to ensure smooth encoding of dynamic pictures.

[0141] Position changes: detailed description of the position change details of the region in the screen coordinate system. When the user performs operations such as position adjustment and size scaling of the window, this information can be completely recorded, providing accurate coordinate basis for subsequent decoding and restoration.

[0142] Importance weight: Output based on the attention mechanism, the encoder can assign corresponding weights to different regions to define the relative importance of each region in the entire screen content, ensuring the encoding quality of key regions.

[0143] Change in region size: Focus on depicting the dynamic changes in the region size level, such as the increase or decrease in size that occurs synchronously during window movement, to avoid image distortion caused by size changes.

[0144] Difference in region content: By comparing consecutive frames, accurately extract the key points of differences, clarify specific changes within the region such as text updates and graphic evolutions, and assist the decoder in accurately restoring the fine content of the region with high precision.

[0145] In the actual video encoding application scenario, the encoder uses the accompanying information according to the following process: First, for each frame of the picture, use the algorithm to quickly detect the active region and calculate its motion information; at the same time, continuously track the position of the active region and record the position changes; use the attention model to evaluate the importance weight; continuously monitor the region boundary to master the size changes; compare the pixel, text, graphic and other elements of consecutive frames to extract the content differences of the active region. The decoder then uses these accompanying information received, combined with the encoded image data, to accurately restore the original video picture and achieve functions such as high-quality video playback and transmission.

[0146] In this application, the process of generating the accompanying information is as follows:

[0147] Motion estimation and vector calculation:

[0148] (1) Motion Estimation:

[0149] In this application, for the active region, at the start of the encoding process, an optimized motion estimation algorithm is innovatively used to deeply analyze the spatio-temporal changes of each pixel within the active region between consecutive video frames, accurately determine the displacement direction and amplitude of each pixel, and thus generate accurate motion vectors. These motion vectors, as key data, provide strong support for the decoder to accurately restore the dynamic evolution of the active region.

[0150] (2) Generate motion vectors:

[0151] After completing the motion estimation, the encoder constructs a motion information set for the determined active regions according to a predetermined rule. The motion information set covers core elements such as the initial positioning coordinates of the region, the direction of the motion trajectory, the accurate displacement amount, and the start and end time intervals, ensuring the integrity and usability of the motion information and providing detailed guidance for the decoder.

[0152] Position change tracking:

[0153] (1) Position tracking:

[0154] The encoder will continuously track the coordinate information of each active area and record the moving path of each active area on the screen. For those dynamically moving areas (such as windows, applications, etc.), the encoder will track their position changes between multiple data frames to ensure that no position change information is missed.

[0155] (2) Generate position change information:

[0156] The encoder generates accompanying position information based on these coordinate information. For dynamically moving areas, the encoder will record the starting and ending positions of the area to help the decoder reconstruct the spatial position of the moving area during decoding, providing a key basis for the decoder to accurately reconstruct the regional spatial layout and ensure the accuracy of the picture space restoration.

[0157] Calculation of importance weight:

[0158] (1) The auxiliary role of the multi-head attention mechanism:

[0159] The advanced multi-head attention mechanism is introduced, and the encoder uses it to deeply evaluate each area in the video stream and accurately determine its relative importance. This is to identify which areas are critical in the video stream, such as user operation hotspots, areas with significant dynamic changes, and other key areas.

[0160] (2) Calculate weight:

[0161] The multi-head attention mechanism reasonably assigns importance weights to each active area based on its activity level and influence on the video stream. Key areas with higher weights will receive more bitrate resources during the encoding phase, ensuring the richness and accuracy of the encoded information and high-quality restoration of the regional content after decoding.

[0162] Calculation of regional content differences:

[0163] (1) Analysis of differences between data frames:

[0164] The encoder performs a sophisticated data frame difference analysis process, and comprehensively compares the changes in pixels, structure, and other dimensions between the current frame and the previous frame for the active area of ​​each frame. Through complex algorithms, it accurately calculates key information such as pixel value changes and graphic structure evolution in the area, laying the foundation for subsequent difference information generation.

[0165] (2) Generate difference information:

[0166] For areas with significant changes, such as real-time text input and window scrolling updates, the encoder generates targeted difference information based on the difference analysis results between data frames. This difference information not only effectively reduces the amount of encoded data and reduces the transmission burden, but also provides key clues for the decoder to accurately restore the fine content of the area.

[0167] Core steps of the multi - head attention mechanism:

[0168] Feature extraction and embedding:

[0169] With the powerful feature extraction ability of the convolutional neural network (CNN), each frame of video image is input into it, and low - level features such as color, edge, and texture are automatically extracted. Subsequently, through a specific conversion method, these features are transformed into the form of an embedding table, so that the embedding representation of each pixel or region contains rich regional details, providing an accurate data basis for subsequent attention calculation.

[0170] Generating query (Query), key (Key), and value (Value):

[0171] (i) Query: The query vector derived from the embedded features, which serves as the core interaction medium, is used to precisely match with the key vector to initiate the attention weight allocation process.

[0172] (ii) Key: As the concrete manifestation of regional features, the key vector of each region carries its unique features. Through the matching operation with the query vector, the attention weight is accurately calculated to measure the degree of association between regions.

[0173] (iii) Value: Retaining the original representation of the original features of the region, after weighted processing, it becomes a key component of the final output, ensuring the integrity and accuracy of the output information.

[0174] Parallel computing of the multi - head attention mechanism:

[0175] (i) The multi - head attention mechanism designed in the present invention realizes the comprehensive calculation of the weighted representation of each region by deploying multiple "attention heads" in parallel. Each attention head operates independently, focuses on different concerns according to the preset learning strategy, and deeply captures diverse features.

[0176] (ii) Some attention heads focus on dynamic graphic tracking, such as accurately locking the dynamic of the mouse pointer in the active area, the instant of button click, and the real - time updated area of the application program, ensuring that no dynamic details are missed.

[0177] (iii) Another part of the attention heads focus on maintaining static areas, covering backgrounds, icons, fixed desktop areas, etc. Although the changes are slight, through accurate monitoring, the coding resources can be reasonably allocated to improve the overall coding efficiency.

[0178] Attention weight calculation and dynamic adjustment:

[0179] (i) Calculate attention scores: By using the dot product operation between the query vector and the key vector, accurately calculate the attention scores for each region. These attention scores intuitively reflect the degree of mutual attention between regions and provide key parameters for weighted averaging.

[0180] (ii) Weighted averaging: Based on the calculated attention scores, perform weighted averaging on the value vectors to generate a final accurate output representation. During this process, active regions obtain higher weights due to their dynamic characteristics, ensuring that coding resources are tilted; static regions correspond to lower weights, achieving efficient compression while ensuring the basic restoration quality.

[0181] Coding strategy based on active regions:

[0182] 1) Dynamic region coding:

[0183] For the active regions accurately identified by the multi-head attention mechanism, since they carry rich user operation information, such as key details like mouse trajectories and dynamic window changes, the encoder allocates sufficient bitrates for them during coding and adopts a fine coding algorithm to ensure the complete retention of regional details, prevent picture distortion caused by excessive compression, and ensure the visual accuracy of user operation feedback.

[0184] 2) Static region compression:

[0185] For the static regions with few changes, the encoder relies on the determination results of the multi-head attention mechanism and adopts a low-bitrate coding strategy. By optimizing the compression algorithm, while ensuring the basic visual recognition, significantly reduce the amount of coded data, improve the overall coding compression efficiency, and optimize the allocation of transmission resources.

[0186] 3) Real-time feedback and adjustment:

[0187] In interactive application scenarios such as remote desktops, user operations cause active regions to change rapidly. The multi-head attention mechanism of the present invention has the ability of real-time monitoring and feedback. Once it detects that user operations are initiated in a specific region, such as dragging, clicking, etc., it immediately dynamically increases the bitrate of that region to ensure that the picture clarity follows immediately; after the operation stops, it quickly reduces the bitrate to flexibly adapt to the dynamic changes of video content and achieve intelligent dynamic allocation of coding resources.

[0188] Furthermore, the remote desktop video encoding and decoding method based on the neural network multi-head attention mechanism is applied to the remote desktop environment.

[0189] It should be noted that the remote desktop video encoding and decoding method of this application is applicable to remote desktop environments, including but not limited to application scenarios such as remote work, distance education, cloud gaming, etc. In specific implementation, the training of the encoding and decoding models can be achieved through deep learning frameworks (such as TensorFlow, PyTorch, etc.), and deployed to the server side and the client side to encode and decode the encoded video stream of the remote desktop in real time.

[0190] Among them, the implementation process of the remote desktop video encoding and decoding method based on the multi-head attention mechanism is as Figure 1 shown.

[0191] Starting step of the process:

[0192] Enter the "start" process, and prepare to receive and process video frame data.

[0193] Execute the "capture screen content" operation, and through the adapted image acquisition module, capture all the image information presented on the screen in real time.

[0194] Execute the "partition screen area" step on the obtained screen content, divide the overall screen space into multiple regions with independent recognition features according to activity and importance, and lay a foundation for subsequent differential processing.

[0195] For each divided region, determine "whether it is an active region" according to whether the calculated attention weight of each region is higher than the set threshold.

[0196] If it is determined to be an active region, give priority to encoding the active region; if it is determined to be a non-active region, instead execute the strategy of encoding other regions with low quality. In view of the relatively static and slightly changing characteristics of such regions, enable the streamlined encoding mode, and by reducing unnecessary encoding redundancy, process these regions at a lower encoding cost on the premise of ensuring basic visual recognition, so as to achieve efficient utilization of resources.

[0197] After completing the region encoding discrimination and preliminary encoding, the encoder "extracts additional information".

[0198] Integrate and compress the encoded video frame data and the synchronously extracted additional information to "generate an encoded video stream".

[0199] Then, "transmit the encoded video stream and additional information" through the transmission channel.

[0200] The receiving end starts the process of "receiving the encoded video stream and additional information", and uses the matching receiving protocol and verification mechanism to accurately and completely receive the encoded video stream and additional information from the sending end, and make data reserves for subsequent decoding and restoration.

[0201] For the received encoded video stream, perform the "decode video stream" step, and use the decoding algorithm corresponding to the encoding end to restore the encoded data to a preliminary video frame sequence, restoring the basic picture structure of the video.

[0202] Subsequently, for the active region and the associated system region with associated logic, perform the operation of "decoding the active region and the associated system region".

[0203] Restore the screen content through the decoded video frame data and the attached information data of each region.

[0204] Push the restored screen content to the display terminal to display the decoded picture.

[0205] In this application, the training of the encoding model and the decoding model is implemented based on deep learning. The task of the encoding model is to generate an efficient encoded video stream according to the input remote desktop video frame, and the task of the decoding model is to restore the encoded video stream to the screen content. The following is the basic process of training the encoding and decoding models as shown in the appendix Figures 2 - 3 as follows:

[0206] First, perform data preparation:

[0207] Collect a large amount of original video frame data of remote desktops. The original video frame data includes scenarios with different desktop activities, different resolutions, and content changes. After preprocessing the original video frame data, construct an original frame dataset, and divide the original frame dataset into a training set, a validation set, and a test set.

[0208] Then, perform the training of the encoding model:

[0209] Introduce the multi-head attention mechanism:

[0210] Determine the parameter settings of the multi-head attention mechanism, such as the number and dimension of attention heads, etc.;

[0211] Integrate it into the architecture of the encoding model so that it can process the input video frame data.

[0212] Extract features using a convolutional neural network:

[0213] Construct a suitable convolutional neural network structure, including the number and parameters of convolutional layers, pooling layers, etc.;

[0214] Input the video frame data and let the convolutional neural network automatically learn and extract the spatial and temporal features in the video.

[0215] Train the encoding network to minimize the loss:

[0216] Define a loss function, such as mean squared error, etc., to measure the difference between the output of the encoding network and the expected output;

[0217] Using the training dataset, continuously adjust the parameters of the encoding network through the backpropagation algorithm to minimize the value of the loss function;

[0218] Divide the training set and the validation set, and regularly evaluate the model performance on the validation set during the training process to prevent overfitting.

[0219] Generate the encoded video stream and the accompanying information:

[0220] The trained encoding network processes the input video to generate an encoded video stream, which is a compressed representation of the video frame data;

[0221] Meanwhile, generate some accompanying information, such as auxiliary data for restoring video details during decoding, etc.

[0222] Next, perform the decoding model training

[0223] Receive the encoded video stream and the accompanying information:

[0224] Design the input interface of the decoding model so that it can correctly receive the encoded video stream and the accompanying information generated by the encoding model.

[0225] The decoding network restores the video content:

[0226] Construct the decoding network structure, and based on the encoded video stream and the accompanying information, through a series of operations such as deconvolution and upsampling, attempt to restore the general content of the original video.

[0227] Optimize the difference between the decoded quality and the original video:

[0228] Define the optimization objective, for example, minimize the differences between the decoded video and the original video in terms of pixel values, structure, etc.;

[0229] Use optimization algorithms, such as gradient descent, etc., to adjust the parameters of the decoding network to improve the decoding quality;

[0230] Similarly, the validation set can be used to verify the performance of the decoding model and make corresponding adjustments.

[0231] Precisely restore the active regions and the associated system regions:

[0232] Analyze the video content to determine the active regions (such as moving objects, etc.) and the associated system regions related to them.

[0233] For these regions, further optimize the parameters and processing methods of the decoding model to more precisely restore the details and coherence of the active regions and the associated system regions.

[0234] Finally, after completing the encoding model and the decoding model (when the performance of the encoding model and the decoding model on the training set and the validation set reaches the expected indicators, such as the loss value is small enough and the quality of the decoded video meets the requirements, it is considered that the training is completed), deploy the encoding model and the decoding model (specifically: deploy the trained encoding model and decoding model to an actual video processing system or platform; ensure that the operating environment of the encoding model and the decoding model meets its dependency requirements, such as hardware configuration, software libraries, etc.). Finally, use the decoding model for video encoding and decoding. Specifically: for the video to be encoded, input it into the deployed encoding model to generate an encoded video stream and accompanying information; transmit the encoded video stream and accompanying information to the receiving end, and use the decoding model for decoding to restore the video content. During the actual use process, continuously monitor the performance and effect of the model, and perform fine-tuning or updating as needed.

[0235] Through the above implementation steps, a video encoding and decoding model can be constructed and applied according to the flowchart, realizing efficient video compression and decompression while ensuring video quality.

[0236] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0237] Embodiment 1

[0238] This embodiment is the application of the remote desktop video encoding and decoding method based on the multi-head attention mechanism in the remote office scenario.

[0239] Encoding process:

[0240] 1. Data collection and preprocessing:

[0241] In the remote office environment, video frame data of the remote desktop is obtained through a specific screen capture tool. These data cover various scenarios in daily office operations, such as document editing, spreadsheet processing, slide presentation, and multi-window operations, etc. Preprocess the collected video frame data, including operations such as denoising and image enhancement, to improve the data quality for subsequent processing. After preprocessing, the original video frame data set is obtained and divided into a training set, a validation set, and a test set, with a ratio of 7:2:1.

[0242] 2. Model construction and training preparation:

[0243] Design and build the network structure of the coding model. The input layer uses a convolutional neural network (CNN). For example, the first few layers of the VGGNet structure are used to extract features from the input video frame data, automatically obtaining low-level visual features in the image, such as edge, texture, and color information.

[0244] Introduce a multi-head attention mechanism layer. According to the activity levels of different regions in the office scenario (such as areas with frequent mouse operations, areas of the document being edited, etc.), different attention weights are assigned to different regions. For example, during document editing, the position of the mouse and the text area being edited are regarded as high-activity regions and are assigned higher attention weights; while relatively static regions such as the menu bar and toolbar are assigned lower weights.

[0245] Set up an aggregation layer to perform weighted average aggregation on the features of different attention heads output by the multi-head attention mechanism layer, obtaining a global feature representation that integrates multi-region features.

[0246] Finally, through a compression layer, such as using quantization and transform encoders, the aggregated features are converted into a coding format suitable for network transmission.

[0247] Before training, calculate the network loss and construct a suitable loss function, such as the mean squared error (MSE) loss function, to measure the difference between the predicted coding result and the true coding result. At the same time, set the network training parameters. The learning rate is initialized to 0.001, the stochastic gradient descent (SGD) optimization algorithm is used, the number of iterations is set to 500 times, and the batch size is 32.

[0248] 3. Model training and optimization:

[0249] Use the training set to train the built coding model. After each training session, a corresponding weight file is obtained. Input the validation set into the model and use the structural similarity index (SSIM) as the evaluation criterion to evaluate the weight file. SSIM can comprehensively consider the brightness, contrast, and structural information of the image, more comprehensively evaluate the similarity between the coded image quality and the original image, and finally save the weight file with the largest SSIM value.

[0250] During the training process, observe the change of the loss function value as the number of iterations increases. If it is found that the loss function value no longer decreases significantly after a certain number of iterations, the learning rate can be appropriately adjusted, for example, multiplied by a decay factor less than 1 (such as 0.1) to promote further convergence of the model.

[0251] 4. Coding and generation of additional information:

[0252] Input the test set into the trained encoding model, and use the saved optimal weight file to extract video features, generating the encoded video stream compressed for each region. Meanwhile, the encoder calculates the motion vectors of active regions (such as document scrolling regions, window moving regions) through the motion estimation algorithm to generate motion information; generates position information based on the coordinate changes of active regions (such as window position adjustment); uses the multi-head attention mechanism to obtain the attention weights of each active region, and allocates the encoding bit rate according to the weights; calculates the pixel differences and structural change information within the active region by comparing the differences between the unit video frame and adjacent video frames to generate difference information. These motion information, position information, importance weights, regional size changes, and differences in regional content together constitute the accompanying information.

[0253] Decoding process:

[0254] 1. Data preparation and model construction:

[0255] The training data preparation of the decoding model corresponds to that of the encoding model. Obtain the encoded video stream and its original screen content, where the encoded video stream includes the compressed video frame data and the accompanying regional information. Design and construct the network structure of the decoding model to be symmetric with the encoding model, and use the transposed convolutional layer (or upsampling layer) to gradually restore the size and content of the image.

[0256] 2. Model training and adjustment:

[0257] Input the encoded data in the training set into the constructed decoding model. The decoding model performs forward propagation calculation according to the preset loss function (such as the same MSE loss function as the encoding model) to obtain the decoded video frame data. Compare the decoded video frame data with the corresponding original encoded video frame data in the training set, calculate the loss value, and backpropagate the loss value to each layer of the model through the backpropagation algorithm to update the model parameters to reduce the difference from the original video.

[0258] Use the validation set to evaluate the model. When the structural similarity index (SSIM) value on the validation set no longer improves, reduce the learning rate (such as multiplying by 0.5) to promote model convergence. During and after the model training process, continuously optimize the quality of the decoded video frame data by adjusting network parameters, optimization algorithms, etc.

[0259] 3. Decoding and picture restoration:

[0260] The decoder receives the encoded video stream and accompanying information transmitted from the encoding end. According to the encoding order, the accompanying information is first parsed to obtain relevant information about the active area and the associated system area. Using this information, the decoder can decode the encoded video stream more accurately, and preferentially restore the image content of the active area (such as the area of the document content being updated, the operation feedback area) and the associated system area (such as the menu pop-up area related to the operation). During the decoding process, each area is processed based on the multi-head attention mechanism. According to the bit rate allocated during encoding and the accompanying information, it focuses on restoring the details of important areas to ensure high-quality restoration of the important content of the picture. For example, for the document editing area, it can accurately restore information such as text content, format, and editing position; for the window operation area, it can clearly restore the position, size, and display content of the window. Finally, the original screen content is restored according to the decoding result, and the decoded picture is displayed at the receiving end to achieve smooth and high-quality desktop video transmission in the remote work scenario.

[0261] Embodiment 2

[0262] This embodiment is an application of the remote desktop video encoding and decoding method based on the multi-head attention mechanism in the distance education scenario.

[0263] Encoding process:

[0264] 1. Data collection and preprocessing:

[0265] In the distance education environment, video frame data of the teacher's remote desktop is collected, including scenarios such as courseware display, classroom interaction (such as questions, answers, whiteboard writing, etc.), and video playback. These data are preprocessed to ensure clear images and accurate colors. The dataset is divided into a training set, a validation set, and a test set according to the same ratio as in Embodiment 1.

[0266] 2. Model construction and training preparation:

[0267] The network structure design of the encoding model is similar to that of Embodiment 1, but it is optimized according to the characteristics of the distance education scenario. In the multi-head attention mechanism layer, attention weights are allocated according to the importance of the courseware content (such as key knowledge point areas, example explanation areas) and the activity of the interaction areas (such as whiteboard writing areas, student question feedback areas). For example, when explaining key knowledge, the key areas marked by the teacher and the content area being explained are given higher weights; during the interaction session, the whiteboard writing area and related feedback areas become the focus of attention.

[0268] Similarly, an aggregation layer and a compression layer are constructed, and appropriate loss functions (such as cross-entropy loss function to better handle classification problems, such as courseware content classification, etc.) and training parameters (learning rate 0.002, using Adam optimization algorithm, number of iterations 600 times, batch size 64) are set.

[0269] 3. Model Training and Optimization:

[0270] Use the training set to train the encoding model. After each training, evaluate the weight file with the validation set, using the peak signal-to-noise ratio (PSNR) as the evaluation criterion. The higher the PSNR value, the better the image quality. Save the weight file with the maximum PSNR value. During the training process, adjust the learning rate according to the change of the PSNR value. If the PSNR value increases by less than 0.1 in 10 consecutive iteration cycles, reduce the learning rate (such as multiplying by 0.8).

[0271] 4. Encoding and Generation of Supplementary Information:

[0272] Process the test set through the trained encoding model to generate a compressed encoded video stream and supplementary information. The encoder calculates the motion vectors and motion information of the active regions (such as the page-turning animation region of the courseware, the erasing and rewriting region of the whiteboard); tracks the position changes of the active regions (such as the position change of the video playback window) to generate position information; allocates bitrates according to the importance weights determined by the multi-head attention mechanism; analyzes the inter-frame differences (such as the update of the courseware content) to generate difference information.

[0273] Decoding Process:

[0274] 1. Data Preparation and Model Construction:

[0275] Prepare the training data for the decoding model, and build a network structure symmetric to the encoding model, including a transposed convolutional layer for restoring the image size.

[0276] 2. Model Training and Adjustment:

[0277] Input the encoded data for decoding model training, calculate the loss value according to the preset loss function and backpropagate to update the parameters. Use the validation set to evaluate the model. When the PSNR value no longer improves, adjust the learning rate (such as multiplying by 0.6). Continuously optimize the decoding quality to ensure that the restored video frame data is as close as possible to the original data.

[0278] 3. Decoding and Image Restoration:

[0279] The decoder receives the encoded video stream and supplementary information, and preferentially decodes the active regions (such as the key regions explained by the teacher, the interactive operation regions) and the associated system regions (such as the tool button regions related to the courseware). Use the supplementary information to accurately restore the content of these regions, such as the clear display of the courseware text and graphics, and the accurate presentation of interactive operations. Finally, restore the original screen content and display it on the student side, providing high-quality and low-latency video transmission for distance education to ensure the teaching effect.

[0280] Example Three

[0281] This embodiment is the application of a remote desktop video encoding and decoding method based on a multi-head attention mechanism in a cloud gaming scenario.

[0282] Coding process:

[0283] 1. Data collection and preprocessing:

[0284] In the cloud gaming scenario, remote desktop video frame data is collected during the game, including game scene switching, character actions, special effects display and other rich and diverse images. The data is preprocessed to enhance the image contrast and clarity to meet the high-quality requirements of the game screen. The data set is divided into training set, validation set and test set in proportion.

[0285] 2. Model construction and training preparation:

[0286] Construct the encoding model network structure, and in the multi-head attention mechanism layer, assign attention weights according to the activity of dynamic elements in the game screen (such as character movement, skill release effects) and important areas (such as game key prompt areas, character health and status display areas). For example, in a character battle scene, the character and skill effects areas are assigned high weights, while the relatively static game background areas are assigned lower weights.

[0287] Set appropriate aggregation layers and compression layers, select appropriate loss functions (such as perceptual loss function, which focuses more on the perceptual quality of the image) and training parameters (learning rate 0.003, use RMSProp optimization algorithm, 800 iterations, and batch size 128).

[0288] 3. Model training and optimization:

[0289] Use the training set to train the encoding model, and use the validation set to evaluate the weight file after training. Use subjective evaluation indicators (such as mean subjective opinion score, MOS) combined with objective indicators (such as PSNR) for comprehensive evaluation. MOS is obtained by inviting players to subjectively rate the quality of the decoded game screen (1-5 points). Save the weight file with the best comprehensive evaluation index. During the training process, adjust the learning rate according to the changes in MOS and PSNR. If MOS has not improved for 5 consecutive iterations and the PSNR improvement is less than 0.05, reduce the learning rate (such as multiplying by 0.7).

[0290] 4. Coding and accompanying information generation:

[0291] The test set is encoded to generate an encoded video stream and accompanying information. The encoder calculates the motion vectors and motion information of active areas (such as the area where the character runs and jumps, and the area where the explosion special effects occur); records the position information of active areas (such as the position changes when the game map is switched); allocates the bit rate according to the weights obtained by the multi-head attention mechanism; and analyzes the differences between frames (such as the update changes of scene elements) to generate difference information.

[0292] Decoding process:

[0293] 1. Data preparation and model building:

[0294] Prepare the decoding model training data and build a network structure symmetrical to the encoding model to ensure that the details and smoothness of the game screen can be accurately restored.

[0295] 2. Model training and adjustment:

[0296] Input the encoded data to train the decoding model, calculate the loss value and back-propagate to update the parameters. Use the validation set to evaluate the model and adjust the learning rate (e.g. multiply by 0.5) based on the evaluation results of MOS and PSNR. Continue to optimize the quality of the decoded game screen to make it close to the original screen.

[0297] 3. Decoding and picture restoration:

[0298] The decoder receives the encoded video stream and accompanying information, and gives priority to decoding active areas (such as character action areas, key areas of the game plot) and related system areas (such as game operation button areas). The accompanying information is used to accurately restore the content of these areas to ensure the smoothness, color accuracy and detail integrity of the game screen. Finally, the restored game screen is displayed on the player side, providing cloud game players with a high-quality, low-latency gaming experience.

[0299] Through the above three embodiments, the effectiveness and adaptability of the remote desktop video encoding and decoding method based on the multi-head attention mechanism in different remote desktop application scenarios are fully demonstrated. It can realize efficient video encoding, transmission and decoding according to the characteristics and requirements of different scenarios, and provide users with high-quality remote desktop services.

Claims

1. A remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism, characterized in that: The method comprises the following steps: S1, uses the multi-head attention mechanism to divide the remote desktop screen into multiple areas; S2, analyze each area and classify it according to its activity and importance on the screen; S3, the encoder sorts the coding order of each area according to the hierarchical identification, formulates the coding strategy for each area accordingly, realizes the identification of active areas, prioritizes the coding of active areas, and dynamically adjusts the coding strategy based on the feedback mechanism; S4, according to the formulated encoding strategy, compressing the original encoded data to be transmitted in each region to generate an encoded video stream, and correspondingly extracting the accompanying information related to the original encoded data in each region; S5, synchronously sending the generated coded video stream and the accompanying information to the receiving end; S6, the decoder decodes each area according to the encoding order and restores the original encoded data to restore the remote desktop screen.

2. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 1 is characterized in that: Acquire remote desktop video frame data, preprocess it, obtain an original video frame data set, and divide the original video frame data set into a training set, a validation set, and a test set; Based on the training set, the validation set and the test set, the encoding model and the decoding model in the encoder and the decoder are trained respectively by a deep learning model.

3. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 2 is characterized in that: The training process of the encoding model is: (1) Using the original video frame dataset obtained in claim 2; (2) Design and build the network structure of the encoding model; (3) Use a multi-head attention mechanism to assign different attention weights to each region according to its activity level, and be responsible for processing the regional features of the screen; use a convolutional neural network to extract image features; (4) Calculate network loss, construct loss function, and calculate the loss between predicted results and actual results; (5) Setting network training parameters, wherein the training parameters include: learning rate and number of iterations; (6) Use the training set to train the encoding model built in (2). After each training, obtain a corresponding weight file, input the validation set, use the Diss similarity coefficient as the evaluation standard to evaluate the weight file, and finally save the weight file with the largest Diss similarity coefficient value; (7) Input the test set into the encoding model, use the maximum weight file obtained in (6) to extract video features, and obtain the compressed encoded video stream generated by each region and the corresponding accompanying information; (8) The performance of the coding model is evaluated by the average intersection-over-parameter of the encoded video stream and the data set.

4. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 2 is characterized in that: The training process of the decoding model is: (1) Using the original video frame dataset obtained in claim 2; (2) Design and build the network structure of the decoding model; (3) obtaining original coded data and a corresponding coded video stream, wherein the coded video stream includes compressed video frame data and accompanying information of each region; (4) Input the training set into the built decoding model, and the decoding model performs forward propagation calculation according to the preset loss function to obtain the decoded video frame data; (5) Compare the decoded video frame data with the corresponding original video frame data in the training set, calculate the loss value, propagate the loss value back to each layer of the model through the back propagation algorithm, and update the decoding model parameters to reduce the difference with the original video frame data; (6) Use the validation set to evaluate the decoding model and adjust the hyperparameters based on the validation results. When the structural similarity index value on the validation set no longer increases, reduce the learning rate to promote model convergence. (7) During and after the decoding model training process, continuously optimize the quality of the decoded video frame data; (8) Based on the accompanying information of each area in the encoded video stream, the active area and the associated system area are accurately restored.

5. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 1 is characterized in that: The step of identifying the active area comprises: A convolutional neural network including multiple convolutional layers of different specifications is used to process the unit video frame to extract the features of the video image of the unit video frame; The extracted features are input into a multi-layer perceptron containing multiple fully connected layers with decreasing numbers of neurons, and the extracted features are converted into embedded representations through the multi-layer perceptron; Using the embedding representation as the vector basis, query, key, and value vectors are generated by setting different weight matrices; Perform parallel computation of multi-head attention according to their corresponding query, key and value vectors; The values ​​calculated by each attention head are concatenated to obtain a comprehensive feature matrix, which is then input into a fully connected layer containing one or more layers of neurons. The input features are linearly transformed through the weight matrix and the bias vector to calculate the attention weights, which are dynamically adjusted during training. The active area is determined based on whether the calculated attention weight is higher than the set threshold.

6. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 1 is characterized in that: The incidental information includes: motion information, position change, importance weight, region size change and region content difference.

7. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 1 is characterized in that: The generation process of the incidental information is as follows: The encoder calculates the motion vector of each region through a motion estimation algorithm, and generates motion information of each region respectively; The encoder generates additional position information based on the coordinate changes of each area, the movement path on the screen, and the position changes between multiple data frames; The encoder obtains the attention weight of each region through a multi-head attention mechanism, and allocates the encoding bit rate of each region accordingly based on the influence of the attention weight; Based on the difference between the unit video frame and the adjacent video frames, the encoder calculates the pixel difference and structural change information of each area and generates corresponding difference information.

8. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claims 1-7 is characterized in that: The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism is applied to a remote desktop environment.

Images