Remote desktop video coding method based on neural network multi-head attention mechanism

Abstract

The application relates to the field of video coding technology and discloses a remote desktop video coding method based on a neural network multi-head attention mechanism.The method comprises the following steps: a multi-head attention mechanism is used to divide a remote desktop screen into multiple regions; hierarchical identification is performed according to the activity and importance of each region in the screen; an encoder performs coding sequencing and formulates a coding strategy for each region according to the hierarchical identification, and dynamically adjusts the coding strategy based on a feedback mechanism; a coded video stream is generated according to the formulated coding strategy, and the accompanying information related to the original coded data of each region is extracted; the generated coded video stream and the accompanying information are synchronously sent to a receiving end; and a decoder decodes each region according to the coding sequence, restores the original coded data, and realizes the restoration of the remote desktop picture. The application utilizes the multi-head attention mechanism to intelligently grade and correlatively code the screen regions, and solves the problems of low efficiency, redundant transmission and high bandwidth occupation existing in the traditional coding and decoding method.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, specifically to a remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism. Background Technology

[0002] With the widespread application of remote desktop technology, especially in environments requiring high-definition video transmission and low latency, efficiently compressing remote desktop videos while maintaining image quality has always been a challenge for technological development. While traditional video encoding and decoding methods can meet certain data compression requirements, the dynamic nature of remote desktop screen content, particularly during frequent updates, leads to an explosive increase in data volume and a surge in bitrate. This undoubtedly places an excessive burden on already limited network bandwidth, easily causing transmission congestion. Furthermore, the significantly extended decoding time directly results in frequent video playback stuttering, severely degrading the user experience and becoming a key bottleneck hindering the further expansion of remote desktop technology applications.

[0003] Most existing remote desktop video encoding and decoding solutions are built upon traditional image encoding algorithms such as H.264 and HEVC. While these classic algorithms have achieved video data compression to some extent in past applications, effectively reducing the amount of data transmitted, they lack the ability to differentiate between different areas of the remote desktop screen. In the complex visual environment of a remote desktop, different screen areas are naturally divided into various categories based on their function and content nature, such as operation button areas, text display areas, and dynamic image display areas. Each type of area has vastly different requirements for image quality detail and content update frequency. Existing encoding solutions are unable to intelligently identify and judge based on the semantic information contained in each area of ​​the screen, making it difficult to dynamically adjust encoding strategies flexibly and efficiently to achieve refined and optimized resource allocation. Summary of the Invention

[0004] This application provides a remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism. This method significantly reduces the data volume and bitrate of remote desktop video transmission while maintaining high-quality video, thereby improving the efficiency and user experience of remote desktop applications. By utilizing a multi-head attention mechanism for intelligent hierarchical and associative encoding of screen areas, it effectively solves the problems of inefficiency, redundant transmission, and high bandwidth consumption in traditional encoding and decoding methods. The method includes the following steps:

[0005] S1 utilizes a multi-head attention mechanism to subdivide the remote desktop screen into multiple areas;

[0006] S2 analyzes each region and classifies and labels them according to their activity and importance on the screen;

[0007] S3, the encoder sorts the encoding order of each region according to the hierarchical identifier, formulates the encoding strategy for each region accordingly, realizes the identification of active regions, prioritizes the encoding of active regions, and dynamically adjusts the encoding strategy based on the feedback mechanism;

[0008] S4, based on the established encoding strategy, compresses the original encoded data to be transmitted in each region to generate an encoded video stream, and extracts the corresponding information related to the original encoded data in each region.

[0009] S5, synchronously sends the generated encoded video stream and accompanying information to the receiving end;

[0010] S6, the decoder decodes each area according to the encoding order to restore the original encoded data, thereby restoring the remote desktop screen.

[0011] Preferably, remote desktop video frame data is acquired and preprocessed to obtain the original video frame dataset, which is then divided into a training set, a validation set, and a test set.

[0012] Based on the training set, validation set, and test set, the encoding model and decoding model in the encoder and decoder are trained respectively using deep learning models.

[0013] Preferably, the training process for the encoding model is as follows:

[0014] (1) Use the obtained original video frame dataset;

[0015] (2) Design and build the network structure of the coding model;

[0016] (3) Use a multi-head attention mechanism to assign different attention weights to each region according to the activity of each region, and be responsible for processing the regional features of the screen; use a convolutional neural network to extract image features;

[0017] (4) Calculate the network loss, construct the loss function, and calculate the loss between the predicted result and the actual result;

[0018] (5) Set network training parameters, including: learning rate and number of iterations;

[0019] (6) Use the training set to train the encoding model built in (2). After each training, a corresponding weight file is obtained. Input the validation set and use the Diss similarity coefficient as the evaluation standard to evaluate the weight file. Finally, save the weight file with the largest Diss similarity coefficient value.

[0020] (7) Input the test set into the coding model, and use the largest weight file obtained in (6) to extract video features, and obtain the compressed coded video stream and corresponding information for each region.

[0021] (8) The performance of the coding model is evaluated by the average intersection-union ratio of the encoded video stream and the dataset.

[0022] Preferably, the training process for the decoding model is as follows:

[0023] (1) Use the obtained original video frame dataset;

[0024] (2) Design and build the network structure of the decoding model;

[0025] (3) Obtain the original encoded data and the corresponding encoded video stream, wherein the encoded video stream includes compressed video frame data and additional information for each region;

[0026] (4) Input the training set into the built decoding model. The decoding model performs forward propagation calculation according to the preset loss function to obtain the decoded video frame data.

[0027] (5) Compare the decoded video frame data with the corresponding original video frame data in the training set, calculate the loss value, and backpropagate the loss value to each layer of the model through the backpropagation algorithm to update the parameters of the decoded model in order to reduce the difference with the original video frame data.

[0028] (6) Use the validation set to evaluate the decoding model, adjust the hyperparameters according to the validation results, and reduce the learning rate when the structural similarity index value on the validation set no longer increases in order to promote model convergence.

[0029] (7) Continuously optimize the quality of the decoded video frame data during and after the training of the decoding model;

[0030] (8) Based on the accompanying information of each region in the encoded video stream, accurately restore the active region and the associated system region.

[0031] Preferably, the active region identification step includes:

[0032] A convolutional neural network containing multiple layers of different sizes is used to process a unit video frame and extract the features of the video image in the unit video frame.

[0033] The extracted features are input into a multilayer perceptron containing multiple fully connected layers with the number of neurons decreasing sequentially. The multilayer perceptron then transforms the extracted features into embedded representations.

[0034] Based on embedded representations, query, key, and value vectors are generated by setting different weight matrices.

[0035] Parallel computation of multi-head attention is performed according to their respective query, key, and value vectors;

[0036] The values ​​calculated by each attention head are concatenated to obtain a comprehensive feature matrix. The comprehensive feature matrix is ​​then input into a fully connected layer containing one or more neurons. The input features are linearly transformed by the weight matrix and bias vector to calculate the attention weights, which are then dynamically adjusted during training.

[0037] The active region is determined based on whether the calculated attention weight is higher than a set threshold.

[0038] Preferably, the accompanying information includes: motion information, position changes, importance weights, changes in region size, and differences in region content.

[0039] Preferably, the process for generating the accompanying information is as follows:

[0040] The encoder calculates the motion vectors for each region using a motion estimation algorithm and generates motion information for each region accordingly.

[0041] The encoder generates additional position information based on the coordinate changes of each region, the movement path on the screen, and the position changes between multiple data frames;

[0042] The encoder obtains the attention weights of each region through a multi-head attention mechanism, and allocates the bit rate of each region according to the influence of the attention weights.

[0043] The encoder calculates the pixel differences and structural changes in each region based on the differences between a unit video frame and its adjacent video frames, and generates the corresponding difference information.

[0044] Preferably, the remote desktop video encoding and decoding method based on the neural network multi-head attention mechanism is applied to a remote desktop environment.

[0045] Compared with the prior art, the beneficial effects of this application are as follows:

[0046] This application divides the remote desktop screen into multiple regions based on a multi-head attention mechanism. Each region is hierarchically labeled according to its activity and importance on the screen, and the encoding order of each region is sorted according to the label. This ensures that active regions and related system regions are encoded first. In this way, the encoder can more accurately identify important regions and encode them with high quality, while unimportant regions are compressed with lower encoding quality to reduce the amount of data. This enables the encoder to accurately identify and prioritize the encoding of important regions in the remote desktop, reduce the transmission of redundant data, and effectively reduce the bitrate of the video stream.

[0047] Simultaneously, based on the hierarchical identification of each region, corresponding encoding strategies are formulated to achieve refined encoding; and compression is performed according to the encoding strategy, while the additional information corresponding to each region is synchronously transmitted to the decoder. When the decoder receives the encoded data, it restores the content of important areas in the screen through the attached information, and more accurately recovers the screen content based on this additional information. By focusing on the active areas and related system areas in the screen, more details and key information are preserved in the encoding, ensuring that the picture quality of the remote desktop is not affected, thereby improving the decoding effect and efficiency.

[0048] During transmission, by sending accompanying information along with the encoded video stream, the decoder can accurately reproduce the image, avoiding the repeated transmission of redundant information, thereby significantly reducing the required bandwidth.

[0049] This application dynamically adjusts the encoding strategy based on a feedback mechanism by detecting in real time which areas of the screen are active and which are static. Based on this feedback, the encoder's multi-head attention mechanism can adjust the bitrate allocation of different areas in real time, maximizing the compression efficiency of the video stream while ensuring that the user experience is not affected.

[0050] This application optimizes encoding and decoding strategies to reduce decoding latency and improve the video transmission experience on remote desktops, making it suitable for low-latency, high-quality video transmission scenarios. Attached Figure Description

[0051] Figure 1 This is a flowchart of the remote desktop video encoding and decoding process based on the multi-head attention mechanism in this application;

[0052] Figure 2 A schematic diagram illustrating the process for preparing the coding model for training in this application;

[0053] Figure 3 This is a schematic diagram of the encoding / decoding model training process in this application. Detailed Implementation

[0054] A remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism, the method comprising the following steps:

[0055] S1 utilizes a multi-head attention mechanism to subdivide the remote desktop screen into multiple areas;

[0056] In this step, the multi-head attention mechanism divides the screen into multiple regions based on the dynamic and temporal characteristics of elements within the region, so that different regions can be encoded differently in the subsequent process.

[0057] S2 analyzes each region and classifies and labels them according to their activity and importance on the screen;

[0058] S3, the encoder sorts the encoding order of each region according to the hierarchical identifier, formulates the encoding strategy for each region accordingly, realizes the identification of active regions, prioritizes the encoding of active regions, and dynamically adjusts the encoding strategy based on the feedback mechanism;

[0059] S4, based on the established encoding strategy, compresses the original encoded data to be transmitted in each region to generate an encoded video stream, and extracts the corresponding information related to the original encoded data in each region.

[0060] S5, synchronously sends the generated encoded video stream and accompanying information to the receiving end;

[0061] S6, the decoder decodes each area according to the encoding order to restore the original encoded data, thereby restoring the remote desktop screen.

[0062] It should be noted that:

[0063] In this application, the rating identifier is used to indicate the priority, activity level, and importance of video content.

[0064] The encoder formulates the encoding strategy for each region based on the hierarchical identifier. Specifically, the encoder allocates and dynamically adjusts the bit rate of each region based on the identifier of each region, as well as perform motion estimation, position tracking prediction and compensation for each region.

[0065] In this application, a multi-head attention mechanism employs multiple independent attention heads to perform parallel computations, capturing different feature information in an image. Each attention head focuses on a different region of the image, dynamically adjusting the attention allocation to each region. This allows for the allocation of more resources to dynamically changing regions during periods of drastic screen content change (e.g., mouse clicks, window changes), ensuring the preservation of critical information. Simultaneously, during encoding, the multi-head attention mechanism encoder can allocate a higher bitrate to dynamic regions, while reducing the bitrate allocated to relatively static backgrounds, thus optimizing encoding efficiency. This intelligent adjustment avoids the inefficiency caused by using the same bitrate for all regions in traditional encoding methods.

[0066] Meanwhile, in remote desktop scenarios, user actions and screen content changes occur in real time. A designed real-time feedback mechanism detects which areas of the screen are active and which are static. Based on this feedback, the encoder's multi-head attention mechanism can adjust the bitrate allocation of different areas in real time, maximizing the compression efficiency of the video stream while ensuring an unaffected user experience. The generated encoded video stream and additional information are then synchronously sent to the receiving end.

[0067] Because the additional information includes motion information and positional changes in the area, during decoding, the decoder first reconstructs the content of important areas on the screen using the additional information, and then accurately recovers the screen content based on the additional information transmitted synchronously with the encoded video stream, thereby improving decoding effect and efficiency. The multi-head attention mechanism ensures that these areas are better reconstructed during decoding, maintaining the image's fineness and important details, while avoiding unnecessary redundant information transmission.

[0068] In this application, based on the hierarchical identification of each region, the encoder will employ different encoding parameters and compression algorithms. For highly active regions, more refined encoding parameters and a higher bit rate will be used to ensure the integrity of information in active regions; for inactive regions, a relatively simple encoding strategy will be adopted to improve overall encoding efficiency. This dynamic adjustment of the encoding strategy is based on feedback information from the decoder. The decoder will send information to the encoder regarding the decoding effect and the display quality at the user end. Based on this feedback information, the encoder will reassess the activity and importance of each region and dynamically adjust the encoding strategy accordingly.

[0069] In this application, the encoder dynamically adjusts the encoding strategy based on a feedback mechanism as follows:

[0070] (1) Video frame data acquisition and encoding:

[0071] Video capture: Capturing video frames using a camera or other video source and converting them into digital signals.

[0072] Preliminary encoding: The encoder performs preliminary compression encoding on the video frames to generate a compressed video frame data stream. This step usually uses standard video encoding formats (such as H.264, H.265, etc.).

[0073] (2) Transmission and reception:

[0074] Data transmission: The encoded video frame data is transmitted to the receiving end (such as the user's device) via the network.

[0075] Receiver feedback: The receiver decodes the received video frame data and plays the video. Simultaneously, the receiver collects video playback quality information, such as packet loss rate, latency, frame rate, and image quality.

[0076] (3) Feedback information collection:

[0077] Quality assessment: The receiving end sends quality reports to the sending end through feedback mechanisms such as RTCP (Real-time Transmission Control Protocol). These reports include various problems that occur during video transmission, such as packet loss, latency, and jitter.

[0078] Performance monitoring: The sending end receives and analyzes this feedback information to evaluate the quality and performance of the current video transmission.

[0079] (4) Encoding strategy adjustment:

[0080] Dynamically adjust encoding parameters: Based on feedback information, the encoder dynamically adjusts its encoding strategy and parameters. For example:

[0081] Bitrate control: If the feedback indicates that the current network bandwidth is insufficient, the encoder can reduce the video bitrate to reduce the amount of data, thereby reducing packet loss and latency.

[0082] Frame rate adjustment: If feedback indicates that the current frame rate is too high, causing increased latency, the encoder can reduce the frame rate to improve transmission efficiency.

[0083] Resolution Adjustment: If feedback indicates that the current resolution is too high and causes a decrease in image quality, the encoder can reduce the resolution to improve image quality.

[0084] Error recovery: If feedback indicates packet loss or errors, the encoder can enable error recovery mechanisms, such as FEC (Forward Error Correction) or retransmission requests, to improve transmission reliability.

[0085] (5) Real-time monitoring and optimization:

[0086] Real-time monitoring: The encoder continuously monitors feedback information and dynamically adjusts the encoding strategy based on real-time changes in network conditions and video quality.

[0087] Optimization strategy: The encoder optimizes encoding parameters based on historical data and current feedback to achieve the best video transmission effect.

[0088] (6) Continuous improvement and adaptation:

[0089] Adaptive Algorithms: The encoder employs adaptive algorithms to continuously improve its encoding strategy based on long-term feedback. For example, machine learning algorithms are used to predict network conditions and adjust encoding parameters in advance.

[0090] Dynamic Adaptation: The encoder can quickly adapt to new environmental conditions based on dynamic changes in network conditions, ensuring the stability and quality of video transmission.

[0091] (7) Feedback loop:

[0092] Looping feedback: A continuous feedback loop is formed between the encoder and the receiver. The encoder continuously adjusts its encoding strategy based on feedback information from the receiver, while the receiver continuously provides new feedback information. This cyclical process ensures continuous optimization of video transmission quality.

[0093] Furthermore, remote desktop video frame data is acquired and preprocessed to obtain the original video frame dataset, which is then divided into training set, validation set and test set.

[0094] The encoding and decoding models in the encoder and decoder are trained based on deep learning models, respectively.

[0095] It should be noted that:

[0096] In this application, deep learning models are used to train the encoding and decoding models, and a multi-head attention mechanism is used to achieve dynamic partitioning and associative encoding. The encoder and decoder are trained on a large amount of remote desktop video frame data, enabling them to adaptively adjust the encoding strategy and accurately identify the features of each region on the screen, thereby significantly reducing the amount of data transmitted while ensuring image quality.

[0097] Furthermore, the training process for the encoding model is as follows:

[0098] (1) The acquired remote desktop video frame data is preprocessed to obtain the original video frame dataset, and the original video frame dataset is divided into training set, validation set and test set.

[0099] (2) Design and build the network structure of the coding model;

[0100] (3) Use a multi-head attention mechanism to assign different attention weights to each region according to the activity of each region, and be responsible for processing the regional features of the screen; use a convolutional neural network to extract image features;

[0101] (4) Calculate the network loss, construct the loss function, and calculate the loss between the predicted result and the actual result;

[0102] (5) Set network training parameters, including: learning rate and number of iterations;

[0103] (6) Use the training set obtained in (1) to train the encoding model built in (2). After each training, a corresponding weight file is obtained. Input the validation set in (1) and use the Diss similarity coefficient as the evaluation standard to evaluate the weight file. Finally, save the weight file with the largest similarity coefficient value.

[0104] (7) Input the test set in (1) into the coding model, and use the largest weight file obtained in (6) to extract video features, and obtain the compressed coded video stream and corresponding additional information for each region.

[0105] (8) The average intersection-to-union ratio of the encoded video stream and the dataset is used to evaluate the performance of the coding model.

[0106] It should be noted that:

[0107] In this application, the design process of the network structure of the coding model is as follows:

[0108] Input layer: Video frame data is fed into a convolutional neural network (CNN) for feature extraction. CNNs can automatically extract low-level visual features from images, such as edges, textures, and colors;

[0109] Multi-head attention mechanism layer: Features extracted by the CNN are input into the multi-head attention mechanism. The multi-head attention mechanism uses multiple independent attention heads to focus on features in different regions, with each head responsible for a different region, outputting a weighted feature map. When encoding remote desktop video, multi-head attention can dynamically adjust the bit allocation for each region based on the importance of the content. For example, the encoder allocates more bitrate to the mouse-active area, while reducing the bitrate for static areas, thus achieving data compression.

[0110] Aggregation layer: Aggregates the outputs of different attention heads (such as weighted averaging or concatenation) to obtain a global feature that integrates multiple different feature representations;

[0111] Compression layer: The feature maps after multi-head attention processing will be passed through a compression network (e.g., quantization and transform encoder) to transform them into an encoding format suitable for transmission.

[0112] Furthermore, the training process for the decoding model is as follows:

[0113] (1) Preprocess the acquired remote desktop video frame data to obtain the original video frame dataset, and divide the dataset into training set, validation set and test set;

[0114] (2) Design and build the network structure of the decoding model;

[0115] (3) Obtain the original encoded data and the corresponding encoded video stream, wherein the encoded video stream includes compressed video frame data and corresponding regional information;

[0116] (4) Input the encoded data in the training set into the built decoding model. The decoding model performs forward propagation calculation according to the preset loss function to obtain the decoded video frame data.

[0117] (5) Compare the decoded video frame data with the corresponding original encoded video frame data in the training set, calculate the loss value, and backpropagate the loss value to each layer of the model through the backpropagation algorithm to update the model parameters in order to reduce the difference from the original video.

[0118] (6) Use the validation set to evaluate the model and adjust the hyperparameters according to the validation results. For example, when the structural similarity index value on the validation set no longer increases, reduce the learning rate to promote model convergence.

[0119] (7) Continuously optimize the quality of the decoded video frame data during and after model training;

[0120] (8) Accurately recover active regions and associated system regions based on the region identifier information in the encoded video stream.

[0121] It should be noted that:

[0122] In this application, the network structure of the decoding model should be symmetrical to that of the encoding model, and should be able to gradually recover the size and content of the image through deconvolution (or upsampling) operations.

[0123] The decoding process is as follows: the decoder receives the encoded compressed data and decodes it. The decoding model can gradually recover the size and content of the image through deconvolution (or upsampling) operations.

[0124] The decoder processes each region during the decoding process using a multi-head attention mechanism, attempting to recover important details from the image. For regions that were assigned more bits during encoding, the decoder focuses on recovering finer details, preserving as much important visual information as possible during the decoding process.

[0125] Further steps in identifying active regions include:

[0126] A convolutional neural network containing multiple layers of different sizes is used to process a unit video frame and extract the features of the video image in the unit video frame.

[0127] The extracted features are input into a multilayer perceptron containing multiple fully connected layers with the number of neurons decreasing sequentially. The multilayer perceptron then transforms the extracted features into embedded representations.

[0128] Based on embedded representations, query, key, and value vectors are generated by setting different weight matrices.

[0129] Parallel computation of multi-head attention is performed according to their respective query, key, and value vectors;

[0130] The values ​​calculated by each attention head are concatenated to obtain a comprehensive feature matrix. The comprehensive feature matrix is ​​then input into a fully connected layer containing one or more neurons. The input features are linearly transformed by the weight matrix and bias vector to calculate the attention weights, which are then dynamically adjusted during training.

[0131] Active regions are determined based on whether the calculated attention weight is higher than a set threshold. Specifically, this is achieved by calculating the attention weight and comparing it with a preset threshold; if the weight is higher than the threshold, the region is considered active.

[0132] Furthermore, the accompanying information includes: motion information, position changes, importance weights, changes in region size, and differences in region content.

[0133] Furthermore, the process for generating the accompanying information is as follows:

[0134] The encoder calculates the motion vectors for each region using a motion estimation algorithm and generates motion information for each region accordingly.

[0135] The encoder generates additional position information based on the coordinate changes of each region, the movement path on the screen, and the position changes between multiple data frames;

[0136] The encoder obtains the attention weights of each region through a multi-head attention mechanism, and allocates the corresponding bit rate for encoding based on the influence of the attention weights;

[0137] The encoder calculates the pixel differences and structural changes in each region based on the differences between a unit video frame and its adjacent video frames, and generates the corresponding difference information.

[0138] It should be noted that:

[0139] In this application, for each identified active region, the encoder generates accompanying information for that active region. This accompanying information helps the decoder accurately reconstruct the content of each active region during subsequent decoding. The accompanying information mainly includes the following types:

[0140] Motion information: Motion vectors used to accurately represent active areas, precisely indicating the displacement and direction of change of that area in the image. This is crucial for accurately capturing dynamic area behaviors, such as mouse dragging operations and window movement, ensuring smooth encoding of dynamic images.

[0141] Position Change: Describes in detail the positional changes of the area in the screen coordinate system. This information can be fully recorded when the user performs operations such as adjusting the position or resizing the window, providing accurate coordinate basis for subsequent decoding and reconstruction.

[0142] Importance weights: Based on the attention mechanism output, the encoder can assign corresponding weights to different regions, thereby defining the relative importance of each region in the overall screen content and ensuring the encoding quality of key regions.

[0143] Area size changes: Focus on depicting dynamic changes in area size, such as the increase or decrease in size that occurs synchronously during window movement, to avoid image distortion caused by size changes.

[0144] Differences in regional content: By comparing previous and next frames, key differences can be accurately extracted, and specific changes such as text updates and graphic evolution within the region can be identified, helping the decoder to accurately recover detailed content in the region.

[0145] In practical video encoding applications, the encoder utilizes the accompanying information in the following process: First, for each frame, it uses algorithms to quickly detect active regions and calculate their motion information; simultaneously, it tracks the position of active regions in real time and records position changes; it uses an attention model to evaluate the importance weights; it continuously monitors region boundaries to understand size changes; and it compares pixels, text, graphics, and other elements from previous and subsequent frames to extract differences in the content of active regions. The decoder then uses this received accompanying information, combined with the encoded image data, to accurately reconstruct the original video, enabling high-quality video playback and transmission.

[0146] In this application, the process for generating accompanying information is as follows:

[0147] Motion estimation and vector calculation:

[0148] (1) Motion Estimation:

[0149] This application, targeting active regions, innovatively employs an optimized motion estimation algorithm at the start of the encoding process to deeply analyze the spatiotemporal changes of each pixel within the active region across consecutive video frames. It precisely measures the displacement direction and amplitude of each pixel, thereby generating accurate motion vectors. These motion vectors serve as crucial data, providing strong support for the subsequent decoder to accurately reconstruct the dynamic evolution of the active region.

[0150] (2) Generate motion vectors:

[0151] After completing motion estimation, the encoder constructs a motion information set for regions identified as active according to predetermined rules. The motion information set covers core elements such as the region's initial positioning coordinates, motion trajectory direction, precise displacement, and start and end time intervals, ensuring the integrity and usability of the motion information and providing detailed guidance for the decoder.

[0152] Location change tracking:

[0153] (1) Position tracking:

[0154] The encoder continuously tracks the coordinates of each active region and records its movement path on the screen. For dynamically moving regions (such as windows, applications, etc.), the encoder tracks their position changes across multiple data frames to ensure that no position change information is missed.

[0155] (2) Generate location change information:

[0156] The encoder generates additional position information based on these coordinates. For dynamically moving areas, the encoder records the start and end positions of the area, helping the decoder to reconstruct the spatial position of the moving area during decoding. This provides a key basis for the decoder to accurately reconstruct the spatial layout of the area and ensures the accuracy of the image spatial restoration.

[0157] Calculation of importance weights:

[0158] (1) The auxiliary role of multi-head attention mechanism:

[0159] By introducing an advanced multi-head attention mechanism, the encoder performs in-depth evaluation of various regions in the video stream, accurately determining their relative importance. This identifies which regions are critical in the video stream, such as user interaction hotspots and areas of significant dynamic change.

[0160] (2) Calculate the weights:

[0161] The multi-head attention mechanism allocates importance weights to each active region based on its activity level and influence on the video stream. Key regions with higher weights receive more bitrate resources during the encoding stage, ensuring the richness and accuracy of encoded information and guaranteeing high-quality reproduction of the region's content after decoding.

[0162] Calculation of regional content differences:

[0163] (1) Data frame difference analysis:

[0164] The encoder performs a sophisticated inter-frame difference analysis process, comprehensively comparing the changes in pixels, structure, and other dimensions between the current frame and the previous frame for the active regions of each frame. Through complex algorithms, it accurately calculates key information such as pixel value changes and graphic structure evolution within the region, laying the foundation for subsequent difference information generation.

[0165] (2) Generate difference information:

[0166] For areas exhibiting significant changes, such as real-time text input or window scrolling updates, the encoder generates targeted difference information based on inter-frame difference analysis. This difference information not only effectively reduces the amount of encoded data and alleviates the transmission burden, but also provides crucial clues for the decoder to accurately recover the fine details of the area.

[0167] Core steps of multi-head attention mechanism:

[0168] Feature extraction and embedding:

[0169] Leveraging the powerful feature extraction capabilities of Convolutional Neural Networks (CNNs), each frame of video image is input into the network, automatically extracting low-level features such as color, edges, and texture. Subsequently, these features are transformed into embeddings using a specific transformation method, ensuring that the embedding representation of each pixel or region contains rich region details, providing a precise data foundation for subsequent attention calculations.

[0170] Generate queries, keys, and values:

[0171] (i) Query: The query vector derived from the embedded features serves as the core interaction medium, used to accurately match the key vector and initiate the attention weight allocation process.

[0172] (ii) Key: As a concrete representation of regional features, the key vector of each region carries its unique features. Through matching operations with the query vector, the attention weight is accurately calculated to measure the degree of correlation between regions.

[0173] (iii) Value: The original representation of the original features of the region is preserved. After weighted processing, it becomes a key component of the final output, ensuring the integrity and accuracy of the output information.

[0174] Parallel computation using multi-head attention mechanisms:

[0175] (i) The multi-head attention mechanism designed in this invention achieves comprehensive computation of the weighted representation of each region by deploying multiple "attention heads" in parallel. Each attention head operates independently, focusing on different points of interest according to a preset learning strategy, and deeply capturing diverse features.

[0176] (ii) Partial attention is focused on dynamic graphics tracking, such as precisely locking the mouse pointer movement in active areas, the instant of button clicks, and areas that are updated in real time by the application, to ensure that no dynamic details are missed.

[0177] (iii) Another part of the attention focuses on the maintenance of static areas, including background, icons, and fixed areas on the desktop. Although the changes are slight, precise monitoring can reasonably allocate coding resources and improve the overall coding efficiency.

[0178] Attention weight calculation and dynamic adjustment:

[0179] (i) Calculate attention score: The attention score of each region is accurately calculated by using the dot product operation between the query vector and the key vector. The attention score directly reflects the degree of mutual attention between regions and provides a key parameter for weighted averaging.

[0180] (ii) Weighted averaging: Based on the calculated attention scores, the value vector is weighted and averaged to generate the final accurate output representation. In this process, active regions receive higher weights due to their dynamic characteristics, ensuring that coding resources are allocated accordingly; static regions receive lower weights, achieving efficient compression while ensuring basic restoration quality.

[0181] Encoding strategy based on active regions:

[0182] 1) Dynamic region coding:

[0183] Active areas, precisely identified through a multi-head attention mechanism, are given ample bitrate during encoding due to the rich user operation information they contain, such as mouse trajectories and dynamic window changes. A refined encoding algorithm is used to ensure that the details of the area are fully preserved, preventing image distortion caused by over-compression and ensuring the visual accuracy of user operation feedback.

[0184] 2) Static region compression:

[0185] For static regions with little change, the encoder relies on the multi-head attention mechanism to determine the result and adopts a low bit rate encoding strategy. By optimizing the compression algorithm, the amount of encoded data is significantly reduced while ensuring basic visual recognition, thereby improving the overall encoding and compression efficiency and optimizing the allocation of transmission resources.

[0186] 3) Real-time feedback and adjustment:

[0187] In highly interactive application scenarios such as remote desktops, user operations cause the active area to change rapidly. The multi-head attention mechanism of this invention has real-time monitoring and feedback capabilities. Once a user operation is detected in a specific area, such as dragging or clicking, the bitrate of that area is immediately and dynamically increased to ensure that the image clarity keeps up in real time. After the operation stops, the bitrate is quickly reduced, flexibly adapting to the dynamic changes in video content and realizing intelligent dynamic allocation of encoding resources.

[0188] Furthermore, the remote desktop video encoding and decoding method based on the neural network multi-head attention mechanism is applied to a remote desktop environment.

[0189] It should be noted that the remote desktop video encoding and decoding method of this application is applicable to remote desktop environments, including but not limited to application scenarios such as remote office, remote education, and cloud gaming. In specific implementation, the encoding and decoding models can be trained using deep learning frameworks (such as TensorFlow, PyTorch, etc.) and deployed to the server and client sides to enable real-time encoding and decoding of encoded video streams from remote desktops.

[0190] The implementation process of the remote desktop video encoding and decoding method based on the multi-head attention mechanism is as follows: Figure 1 As shown.

[0191] Process start steps:

[0192] Enter the "Start" process to prepare to receive and process video frame data.

[0193] The "Capture Screen Content" operation is executed, which uses an adapted image acquisition module to capture all image information displayed on the screen in real time.

[0194] The acquired screen content is divided into "screen region partitioning" steps, which divide the overall screen space into multiple regions with independent identification characteristics based on activity and importance, laying the foundation for subsequent differentiated processing.

[0195] For each of the defined regions, the system determines whether a region is an "active region" based on whether the attention weight calculated for each region is higher than a set threshold.

[0196] If a region is determined to be active, it is prioritized for encoding. If a region is determined to be inactive, a strategy of encoding other regions with low quality is adopted. Given the relatively static and minimally changing nature of such regions, a simplified encoding mode is enabled. By reducing unnecessary encoding redundancy, these regions are processed at a lower encoding cost while ensuring basic visual recognition, thus achieving efficient resource utilization.

[0197] After completing the region coding discrimination and preliminary coding, the encoder "extracts the attached information".

[0198] The encoded video frame data and the synchronously extracted accompanying information are integrated and compressed to "generate an encoded video stream".

[0199] Then, the encoded video stream and accompanying information are transmitted through the transmission channel.

[0200] The receiving end initiates the "receive encoded video stream and accompanying information" process, using a matching receiving protocol and verification mechanism to accurately and completely receive the encoded video stream and accompanying information from the sending end, thus preparing data reserves for subsequent decoding and restoration.

[0201] For the received encoded video stream, the "decode video stream" step is performed. Using the decoding algorithm corresponding to the encoding end, the encoded data is restored to a preliminary video frame sequence, and the basic picture structure of the video is recovered.

[0202] Then, for the active regions and the related system regions with associated logic, the operation of "decoding the active regions and related system regions" is carried out.

[0203] The screen content is recovered by decoding the video frame data and the accompanying information data of each area.

[0204] The restored screen content is pushed to the display terminal to display the decoded image.

[0205] In this application, the encoding and decoding models are trained using deep learning. The encoding model's task is to generate an efficient encoded video stream from the input remote desktop video frames, while the decoding model's task is to reconstruct the screen content from the encoded video stream. The basic training process for the encoding and decoding models is shown in the attached figure. Figure 2-3 As shown:

[0206] First, prepare the data:

[0207] A large amount of raw video frame data from remote desktops is collected. The raw video frame data includes scenes with different desktop activities, different resolutions, and content changes. After preprocessing the raw video frame data, a raw frame dataset is constructed and divided into a training set, a validation set, and a test set.

[0208] Then, the encoding model is trained:

[0209] Introducing a multi-head attention mechanism:

[0210] Determine the parameter settings for the multi-head attention mechanism, such as the number and dimensions of the attention heads;

[0211] This is integrated into the architecture of the coding model, enabling it to process input video frame data.

[0212] Convolutional neural networks extract features:

[0213] Construct a suitable convolutional neural network structure, including the number and parameters of convolutional layers, pooling layers, etc.

[0214] Input video frame data and let the convolutional neural network automatically learn and extract the spatial and temporal features in the video.

[0215] Minimize the loss when training the encoding network:

[0216] Define a loss function, such as mean squared error, to measure the difference between the output of the encoding network and the expected output;

[0217] Using the training dataset, the parameters of the encoding network are continuously adjusted through the backpropagation algorithm to minimize the value of the loss function;

[0218] Divide the model into training and validation sets, and periodically evaluate the model performance on the validation set during training to prevent overfitting.

[0219] Generate encoded video stream and accompanying information:

[0220] The trained coding network processes the input video to generate a coded video stream, which is a compressed representation of video frame data.

[0221] It also generates some additional information, such as auxiliary data used to recover video details during decoding.

[0222] Next, the decoding model is trained.

[0223] Receive encoded video stream and accompanying information:

[0224] Design the input interface of the decoding model so that it can correctly receive the encoded video stream and accompanying information generated by the encoding model.

[0225] Decode the network to restore the video content:

[0226] A decoding network structure is constructed, and based on the encoded video stream and accompanying information, a series of operations such as deconvolution and upsampling are used to attempt to reconstruct the approximate content of the original video.

[0227] Optimize decoding quality to match the original video:

[0228] Define optimization objectives, such as minimizing the differences between the decoded video and the original video in terms of pixel values, structure, etc.

[0229] Use optimization algorithms, such as gradient descent, to adjust the parameters of the decoding network to improve decoding quality;

[0230] Similarly, a validation set can be used to verify the performance of the decoding model and make corresponding adjustments.

[0231] Precisely restore active and associated system regions:

[0232] Analyze video content to identify active areas (such as moving objects) and related system areas.

[0233] For these regions, the parameters and processing methods of the decoding model are further optimized to more accurately restore the details and coherence of active regions and related system regions.

[0234] Finally, after completing the encoding and decoding models (the models are considered trained when their performance on the training and validation sets meets expected metrics, such as sufficiently small loss values ​​and satisfactory decoded video quality), the models are deployed (specifically, the trained models are deployed to the actual video processing system or platform; ensuring the operating environment of the models meets their dependencies, such as hardware configuration and software libraries). The decoding model is then used for video encoding and decoding. Specifically, for the video to be encoded, it is input into the deployed encoding model to generate an encoded video stream and accompanying information; the encoded video stream and accompanying information are transmitted to the receiving end, where the decoding model is used to decode and reconstruct the video content. During actual use, the model's performance and effectiveness are continuously monitored, and fine-tuning or updates are made as needed.

[0235] By following the steps outlined above, a video encoding / decoding model can be built and applied according to the flowchart to achieve efficient video compression and decompression while ensuring video quality.

[0236] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0237] Example 1

[0238] This embodiment demonstrates the application of a remote desktop video encoding / decoding method based on a multi-head attention mechanism in a remote work scenario.

[0239] Encoding process:

[0240] 1. Data Acquisition and Preprocessing:

[0241] In a remote work environment, video frame data from the remote desktop is acquired using specific screen capture tools. This data covers various scenarios in daily office operations, such as document editing, spreadsheet processing, PowerPoint presentations, and multi-window operations. The acquired video frame data undergoes preprocessing, including noise reduction and image enhancement, to improve data quality and facilitate subsequent processing. After preprocessing, the original video frame dataset is obtained and divided into training, validation, and test sets in a 7:2:1 ratio.

[0242] 2. Model building and training preparation:

[0243] Design and build the network structure for the encoding model. The input layer uses a convolutional neural network (CNN), such as the first few layers of a VGGNet structure, to extract features from the input video frame data and automatically obtain low-level visual features in the image, such as edges, textures, and colors.

[0244] A multi-head attention mechanism layer is introduced to assign different attention weights to different areas based on their activity level in the office scenario (such as areas with frequent mouse operations, areas of documents being edited, etc.). For example, during document editing, the mouse position and the area of ​​text being edited are considered high-activity areas and are assigned higher attention weights; while relatively static areas such as menu bars and toolbars are assigned lower weights.

[0245] An aggregation layer is set up to perform a weighted average aggregation of the features from different attention heads output by the multi-head attention mechanism layer, resulting in a global feature representation that integrates features from multiple regions.

[0246] Finally, a compression layer, such as a quantization and transform encoder, is used to transform the aggregated features into an encoding format suitable for network transmission.

[0247] Before training, the network loss is calculated, and a suitable loss function, such as the mean squared error (MSE) loss function, is constructed to measure the difference between the predicted and actual encoding results. Simultaneously, the network training parameters are set, with the learning rate initialized to 0.001, the stochastic gradient descent (SGD) optimization algorithm used, the number of iterations set to 500, and the batch size set to 32.

[0248] 3. Model Training and Optimization:

[0249] The built encoding model is trained using the training set. After each training iteration, a corresponding weight file is obtained. The validation set is input into the model, and the structural similarity index (SSIM) is used as the evaluation criterion to evaluate the weight files. SSIM can comprehensively consider the brightness, contrast, and structural information of the image, and more comprehensively evaluate the quality of the encoded image and its similarity to the original image. Finally, the weight file with the highest SSIM value is saved.

[0250] During training, observe the changes in the loss function value as the number of iterations increases. If the loss function value no longer decreases significantly after a certain number of iterations, the learning rate can be adjusted appropriately, for example, by multiplying it by a decay factor less than 1 (such as 0.1) to promote further convergence of the model.

[0251] 4. Encoding and Generation of Attached Information:

[0252] The test set is input into the trained encoding model, and video features are extracted using the saved optimal weight file to generate compressed encoded video streams for each region. Simultaneously, the encoder calculates motion vectors for active regions (such as document scrolling areas and window movement areas) using motion estimation algorithms, generating motion information; it generates positional information based on coordinate changes in active regions (such as window position adjustments); it obtains attention weights for each active region using a multi-head attention mechanism and allocates the encoding bitrate according to these weights; and it calculates pixel differences and structural changes within active regions by comparing the differences between a single video frame and adjacent video frames, generating difference information. This motion information, positional information, importance weights, region size changes, and differences in region content collectively constitute supplementary information.

[0253] Decoding process:

[0254] 1. Data preparation and model building:

[0255] The training data preparation for the decoding model corresponds to that of the encoding model. The encoded video stream and its original screen content are obtained, where the encoded video stream includes compressed video frame data and accompanying region information. The network structure of the decoding model is designed and built to be symmetrical with that of the encoding model. Deconvolutional layers (or upsampling layers) are used to progressively recover the image size and content.

[0256] 2. Model Training and Adjustment:

[0257] The encoded data from the training set is input into the constructed decoding model. The decoding model performs forward propagation calculations based on a preset loss function (such as the same MSE loss function as the encoding model) to obtain the decoded video frame data. The decoded video frame data is compared with the corresponding original encoded video frame data in the training set, the loss value is calculated, and the loss value is backpropagated to each layer of the model through the backpropagation algorithm to update the model parameters and reduce the difference from the original video.

[0258] The model is evaluated using a validation set. When the Structural Similarity Index (SSIM) value on the validation set no longer improves, the learning rate is reduced (e.g., multiplied by 0.5) to promote model convergence. During and after model training, the quality of the decoded video frame data is continuously optimized by adjusting network parameters and optimizing algorithms.

[0259] 3. Decoding and image restoration:

[0260] The decoder receives the encoded video stream and accompanying information from the encoding end. Based on the encoding order, it first parses the accompanying information to obtain information about active areas and related system areas. Using this information, the decoder can decode the encoded video stream more accurately, prioritizing the recovery of image content in active areas (such as areas displaying updated document content or operation feedback) and related system areas (such as menu pop-ups related to operations). During decoding, a multi-head attention mechanism is used to process each area, focusing on recovering details of important areas based on the bit rate allocated during encoding and the accompanying information, ensuring high-quality restoration of important content. For example, for document editing areas, it can accurately restore text content, formatting, and editing position; for window operation areas, it can clearly restore the window's position, size, and displayed content. Finally, the original screen content is restored based on the decoding results, and the decoded image is displayed at the receiving end, achieving smooth, high-quality desktop video transmission in remote work scenarios.

[0261] Example 2

[0262] This embodiment demonstrates the application of a remote desktop video encoding / decoding method based on a multi-head attention mechanism in a distance education scenario.

[0263] Encoding process:

[0264] 1. Data Acquisition and Preprocessing:

[0265] In a distance education environment, video frame data from the teacher's remote desktop is collected, including scenarios such as courseware presentations, classroom interactions (e.g., asking and answering questions, writing on the whiteboard), and video playback. This data is preprocessed to ensure image clarity and color accuracy. The dataset is then divided into training, validation, and test sets according to the same proportions as in Example 1.

[0266] 2. Model building and training preparation:

[0267] The network structure design of the encoding model is similar to that of Implementation Example 1, but it is optimized for the characteristics of distance education scenarios. In the multi-head attention mechanism layer, attention weights are assigned based on the importance of the courseware content (e.g., key knowledge points, example explanations) and the activity level of interactive areas (e.g., whiteboard writing areas, student question and feedback areas). For example, when explaining key knowledge, the key areas marked by the teacher and the content being explained are given higher weights; during interactive sessions, the whiteboard writing area and related feedback areas become the focus of attention.

[0268] Similarly, we constructed an aggregation layer and a compression layer, and set appropriate loss functions (such as cross-entropy loss function to better handle classification problems, such as courseware content classification) and training parameters (learning rate 0.002, Adam optimization algorithm, 600 iterations, batch size 64).

[0269] 3. Model Training and Optimization:

[0270] The encoding model is trained using the training set. After each training iteration, the weight files are evaluated using the validation set, with Peak Signal-to-Noise Ratio (PSNR) as the evaluation metric. A higher PSNR value indicates better image quality. The weight file with the highest PSNR value is saved. During training, the learning rate is adjusted based on changes in the PSNR value. If the PSNR value increases by less than 0.1 for 10 consecutive iterations, the learning rate is reduced (e.g., multiplied by 0.8).

[0271] 4. Encoding and Generation of Attached Information:

[0272] The trained encoding model processes the test set to generate a compressed encoded video stream and accompanying information. The encoder calculates motion vectors and motion information for active regions (such as the page-turning animation area of ​​courseware and the whiteboard erase and rewrite area); tracks position changes of active regions (such as changes in the position of the video playback window) to generate position information; allocates bit rate according to the importance weights determined by the multi-head attention mechanism; and analyzes inter-frame differences (such as courseware content updates) to generate difference information.

[0273] Decoding process:

[0274] 1. Data preparation and model building:

[0275] Prepare training data for the decoding model, and build a network structure symmetrical to the encoding model, including deconvolution layers to restore image size.

[0276] 2. Model Training and Adjustment:

[0277] Input encoded data to train the decoding model. Calculate the loss value according to the preset loss function and update the parameters through backpropagation. Evaluate the model using a validation set. When the PSNR value no longer improves, adjust the learning rate (e.g., multiply by 0.6). Continuously optimize the decoding quality to ensure that the recovered video frame data is as close as possible to the original data.

[0278] 3. Decoding and image restoration:

[0279] The decoder receives the encoded video stream and accompanying information, prioritizing the decoding of active areas (such as key areas explained by the teacher and interactive operation areas) and related system areas (such as tool buttons related to the courseware). Using the accompanying information, it accurately restores the content of these areas, ensuring clear display of courseware text and graphics, and accurate presentation of interactive operations. Finally, it restores the original screen content and displays it on the student's end, providing high-quality, low-latency video transmission for distance education and guaranteeing teaching effectiveness.

[0280] Example 3

[0281] This embodiment demonstrates the application of a remote desktop video encoding / decoding method based on a multi-head attention mechanism in a cloud gaming scenario.

[0282] Encoding process:

[0283] 1. Data Acquisition and Preprocessing:

[0284] In cloud gaming scenarios, remote desktop video frame data is collected during game execution, including a variety of scenes such as game scene transitions, character actions, and special effects. The data is preprocessed to enhance image contrast and clarity to meet the high-quality requirements of game visuals. The dataset is then proportionally divided into training, validation, and test sets.

[0285] 2. Model building and training preparation:

[0286] A network structure for the encoding model is constructed. In the multi-head attention mechanism layer, attention weights are assigned based on the activity and importance of dynamic elements in the game screen (such as character movement and skill release effects) in different regions (e.g., key game prompt areas, character health and status display areas). For example, in a character combat scene, the character and skill effect areas are assigned high weights, while the relatively static game background areas are assigned lower weights.

[0287] Set appropriate aggregation and compression layers, select suitable loss functions (such as perceptual loss function, which focuses more on the perceptual quality of the image) and training parameters (learning rate 0.003, using RMSProp optimization algorithm, 800 iterations, batch size 128).

[0288] 3. Model Training and Optimization:

[0289] The encoding model is trained using the training set, and the weight file is evaluated using the validation set after training. A comprehensive evaluation is performed using subjective evaluation metrics (such as Mean Opinion Score, MOS) combined with objective metrics (such as PSNR). MOS is obtained by inviting players to subjectively rate the quality of the decoded game graphics (1-5 points). The weight file with the optimal comprehensive evaluation metrics is saved. During training, the learning rate is adjusted based on changes in MOS and PSNR. If MOS does not improve for five consecutive iterations and the PSNR improvement is less than 0.05, the learning rate is reduced (e.g., multiplied by 0.7).

[0290] 4. Encoding and Generation of Attached Information:

[0291] The test set is encoded to generate an encoded video stream and accompanying information. The encoder calculates motion vectors and motion information for active regions (such as areas where characters run and jump, and areas with explosion effects); records position information for active regions (such as position changes when the game map switches); allocates bitrates based on weights obtained from the multi-head attention mechanism; and analyzes inter-frame differences (such as updates and changes in scene elements) to generate difference information.

[0292] Decoding process:

[0293] 1. Data preparation and model building:

[0294] Prepare training data for the decoding model and build a network structure symmetrical to the encoding model to ensure accurate restoration of the details and smoothness of the game screen.

[0295] 2. Model Training and Adjustment:

[0296] Train the decoding model using the input encoded data, calculate the loss value, and backpropagate to update the parameters. Evaluate the model using a validation set, and adjust the learning rate (e.g., multiply by 0.5) based on the evaluation results of MOS and PSNR. Continuously optimize the quality of the decoded game visuals to make them close to the original visuals.

[0297] 3. Decoding and image restoration:

[0298] The decoder receives the encoded video stream and accompanying information, prioritizing the decoding of active areas (such as character action areas and key plot areas) and related system areas (such as game control button areas). It then uses the accompanying information to accurately reconstruct the content of these areas, ensuring smooth gameplay, accurate color reproduction, and complete detail. Finally, the reconstructed game visuals are displayed on the player's device, providing cloud gaming players with a high-quality, low-latency gaming experience.

[0299] The above three embodiments fully demonstrate the effectiveness and adaptability of the remote desktop video encoding and decoding method based on the multi-head attention mechanism in different remote desktop application scenarios. It can achieve efficient video encoding, transmission and decoding according to the characteristics and needs of different scenarios, and provide users with high-quality remote desktop services.

Claims

1. A remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism, characterized in that, The method includes the following steps: S1 utilizes a multi-head attention mechanism to analyze the screen content of the remote desktop in real time. Based on user operation information, including mouse clicks and window changes, the remote desktop screen is subdivided into multiple areas. S2 analyzes each region and calculates the attention weight of each region based on its real-time activity and content importance on the screen, and then classifies and labels them accordingly. S3, the encoder sorts the encoding order of each region according to the hierarchical identifier, formulates the encoding strategy for each region accordingly, realizes the identification of active regions, prioritizes the encoding of active regions, and dynamically adjusts the encoding strategy based on the feedback mechanism; The encoding strategy includes allocating the bit rate of each region based on the attention weight, detecting active and static regions in the current screen in real time based on a feedback mechanism, and dynamically adjusting the attention weight and encoding strategy through a multi-head attention mechanism according to the detection results to achieve priority encoding of active regions. S4. Based on the established encoding strategy, the original encoded data to be transmitted in each region is compressed to generate an encoded video stream, and the accompanying information related to the original encoded data in each region is extracted accordingly. S5, synchronously send the generated encoded video stream and accompanying information to the receiving end; S6, the decoder decodes each region according to the encoding order and the received additional information, prioritizing the recovery of the original encoded data of the active region and the associated system region, so as to restore the remote desktop screen; The accompanying information includes: motion information, positional changes, attention weights for each region, and differences in region content. The process for generating the accompanying information is as follows: The encoder calculates the motion vectors of each region using a motion estimation algorithm and generates motion information for each region accordingly. The encoder also generates additional position information based on the coordinate changes of each region, the movement path on the screen, and the position changes between multiple data frames. The encoder obtains the attention weights of each region through a multi-head attention mechanism, and allocates the bit rate of each region according to the influence of the attention weights. The encoder calculates the pixel differences and structural changes in each region based on the differences between a unit video frame and its adjacent video frames, and generates the corresponding difference information. The coding order of each region is sorted according to the identifier, so that active regions and associated system regions are coded first; By transmitting the accompanying information along with the encoded video stream, the decoder can accurately reconstruct the image, avoiding the repeated transmission of redundant information and thus significantly reducing the required bandwidth. The encoding strategy is dynamically adjusted based on a feedback mechanism by detecting in real time which areas of the screen are active and which are static. Based on this feedback, the encoder's multi-head attention mechanism can adjust the bit rate allocation of different areas in real time, maximizing the compression efficiency of the video stream while ensuring that the user experience is not affected.

2. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 1, characterized in that, Acquire remote desktop video frame data and preprocess it to obtain the raw video frame dataset. Divide the raw video frame dataset into training set, validation set and test set. Based on the training set, validation set, and test set, the encoding model and decoding model in the encoder and decoder are trained respectively using deep learning models.

3. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 2, characterized in that, The training process of the encoding model is as follows: (1) Using the original video frame dataset obtained in claim 2; (2) Design and build the network structure of the coding model; (3) Use a multi-head attention mechanism to assign different attention weights to each region according to the activity of each region, and be responsible for processing the regional features of the screen; use a convolutional neural network to extract image features; (4) Calculate the network loss, construct the loss function, and calculate the loss between the predicted result and the actual result; (5) Set network training parameters, including: learning rate and number of iterations; (6) Use the training set to train the encoding model built in (2). After each training, a corresponding weight file is obtained. Input the validation set and use the Diss similarity coefficient as the evaluation standard to evaluate the weight file. Finally, save the weight file with the largest Diss similarity coefficient value. (7) Input the test set into the coding model, and use the largest weight file obtained in (6) to extract video features, and obtain the compressed coded video stream and corresponding information for each region. (8) The performance of the coding model is evaluated by the average intersection-union ratio of the encoded video stream and the dataset.

4. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 2, characterized in that, The training process of the decoding model is as follows: (1) Using the original video frame dataset obtained in claim 2; (2) Design and build the network structure of the decoding model; (3) Obtain the original encoded data and the corresponding encoded video stream, wherein the encoded video stream includes compressed video frame data and additional information for each region; (4) Input the training set into the built decoding model. The decoding model performs forward propagation calculation according to the preset loss function to obtain the decoded video frame data. (5) Compare the decoded video frame data with the corresponding original video frame data in the training set, calculate the loss value, and backpropagate the loss value to each layer of the model through the backpropagation algorithm to update the parameters of the decoded model in order to reduce the difference with the original video frame data. (6) Use the validation set to evaluate the decoding model, adjust the hyperparameters according to the validation results, and reduce the learning rate when the structural similarity index value on the validation set no longer increases in order to promote model convergence. (7) Continuously optimize the quality of the decoded video frame data during and after the training of the decoding model; (8) Based on the accompanying information of each region in the encoded video stream, accurately restore the active region and the associated system region.

5. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to claim 1, characterized in that, The steps for identifying the active region include: A convolutional neural network containing multiple layers of different sizes is used to process a unit video frame and extract the features of the video image in the unit video frame. The extracted features are input into a multilayer perceptron containing multiple fully connected layers with the number of neurons decreasing sequentially. The multilayer perceptron then transforms the extracted features into embedded representations. Based on embedded representations, query, key, and value vectors are generated by setting different weight matrices. Parallel computation of multi-head attention is performed according to their respective query, key, and value vectors; The values ​​calculated by each attention head are concatenated to obtain a comprehensive feature matrix. The comprehensive feature matrix is ​​then input into a fully connected layer containing one or more neurons. The input features are linearly transformed by the weight matrix and bias vector to calculate the attention weights, which are then dynamically adjusted during training. The active region is determined based on whether the calculated attention weight is higher than a set threshold.

6. The remote desktop video encoding and decoding method based on a neural network multi-head attention mechanism according to any one of claims 1-5, characterized in that, The remote desktop video encoding and decoding method based on the neural network multi-head attention mechanism is applied to the remote desktop environment.

Images