Intelligent coding and decoding method and system based on video content in cloud desktop scene

Abstract

The invention discloses an intelligent coding and decoding method and system based on video content in a cloud desktop scene, and the method comprises the steps: determining a change region image block and a content type corresponding to the change region image block through an image difference and content recognition technology for a current frame image of a cloud host system desktop; determining a coding mode of the current frame image based on the size, the distribution and the content type of the image blocks in the change area within the coding and decoding capability range of the cloud host system; coding the current frame image based on the coding mode to obtain coded data; and encrypting the coded data, transmitting the obtained encrypted data to a client, and correspondingly decrypting and decoding the obtained encrypted data by the client to obtain an updated complete frame image.

Description

Technical Field

[0001] The present invention relates to the fields of cloud computing and cloud desktop applications, video encoding and decoding, and encrypted communication, and in particular to an intelligent encoding and decoding method and system based on video content in a cloud desktop scenario. Background Art

[0002] Cloud desktop is a desktop virtualization technology based on cloud computing. It migrates the traditional desktop computing environment to the cloud server. Users can access and use applications and files on the cloud desktop through various terminal devices (such as PC, tablet, mobile phone, etc.). Cloud desktop technology has the advantages of centralized resource management, elastic expansion, high availability, and security. It is widely used in enterprise office, distance education, graphic design and other fields. Video content encoding and decoding technology refers to the technology for compressing and decompressing video data. In the cloud desktop scenario, the transmission and presentation of video content need to rely on efficient encoding and decoding technology to reduce the amount of data, improve transmission efficiency, and ensure video quality. Common video encoding algorithms include H.264, H.265 (HEVC), VP8, VP9, ​​AV1, etc. These algorithms use different compression strategies and optimization methods to achieve video data compression. However, traditional video encoding and decoding technology still faces many challenges in cloud desktop scenarios. For example, although algorithms such as H.264 and H.265 have significantly improved in compression efficiency, they often find it difficult to achieve sufficiently low latency and sufficiently high compression rate while ensuring video quality when facing complex and changeable video content in cloud desktops. Therefore, there is an urgent need to propose an intelligent encoding and decoding technology based on video content, which can make full use of the advantages of cloud computing and artificial intelligence, meet the challenges of real-time and efficient compression in cloud desktop scenarios, and provide users with a smoother, clearer and more intelligent cloud desktop experience. Summary of the invention

[0003] In order to meet the challenges of real-time performance and efficient compression in cloud desktop scenarios, the purpose of the present invention is to provide an intelligent encoding and decoding method and system based on video content in cloud desktop scenarios. The technical solutions adopted are as follows: In a first aspect, the present application discloses an intelligent encoding and decoding method based on video content in a cloud desktop scenario, the method being applied to a server, comprising: S1. For the current frame image of the cloud host system desktop, determine the image block in the changed area and the content type corresponding to the image block in the changed area through image difference and content recognition technology; S2. Determine the encoding method of the current frame image based on the size, distribution, and content type of the image blocks in the changed area within the encoding and decoding capabilities of the cloud host system; S3, encoding the current frame image based on the encoding method to obtain encoded data; S4. Encrypt the encoded data and transmit the obtained encrypted data to the client, and the client performs corresponding decryption and decoding on the obtained encrypted data, and then displays the current frame image.

[0004] Furthermore, in step S1, for the current frame image of the cloud host system desktop, the image block of the changed area and the content type corresponding to the image block of the changed area are determined by image difference and content recognition technology, including: S11, obtaining the previous frame image and the current frame image of the cloud host system desktop; S12, determining the image change area through pixel difference analysis based on the previous frame image and the current frame image; S13, identifying a target image area to be encoded based on the image change area; S14: performing image segmentation and image content recognition on the target image area to determine image blocks in the changed area and content types corresponding to the image blocks in the changed area.

[0005] Furthermore, in step S2, the encoding method determined based on the size, distribution, and content type of the image blocks in the changed area includes: According to the application scenario requirements, a first encoding method or a second encoding method is used to perform global incremental encoding on a first frame image, a preset key frame image, or a current frame image having a change area image block whose size is greater than or equal to a preset threshold, using a streaming encoding form or a picture encoding form; For a current frame image in which there are multiple second change area image blocks with sizes smaller than a preset threshold, a third encoding method is used to perform regional incremental encoding using an adaptive hybrid encoding form according to the content type corresponding to each image block in the image. The hybrid encoding form includes at least one of a PNG lossy encoding form, an LZ4 lossless encoding form, an H265 lossless encoding form, an H264 lossy encoding form, and a JPEG lossy encoding form.

[0006] Further, in step S2, for the current frame image having multiple second change area image blocks whose sizes are smaller than a preset threshold, if the content type corresponding to one of the image blocks is text office content, the image block is incrementally encoded in the PNG lossy encoding form; if the content type corresponding to one of the image blocks is design and drawing content and the cloud host system supports lossless hardware acceleration, the image is incrementally encoded in the H265 lossless encoding form; if the content type corresponding to one of the image blocks is design and drawing content and the cloud host system does not support lossless hardware acceleration, the image block is incrementally encoded in the LZ4 lossless encoding form; if the content type corresponding to one of the image blocks is rendering entertainment content and the cloud host system supports lossy hardware acceleration, the image block is incrementally encoded in the H264 lossy encoding form; if the content type corresponding to one of the image blocks is rendering entertainment content and the cloud host system does not support lossy hardware acceleration, the image block is incrementally encoded in the JPEG lossy encoding form.

[0007] Further, for a current frame image having multiple second change region image blocks whose sizes are smaller than a preset threshold, if the multiple image blocks are spatially adjacent and have the same content type, the multiple image blocks are merged to obtain a new merged change region; for the current frame image including the merged change region, the encoding method determined in step S2 also includes: When it is determined that the size of the merged change region is greater than or equal to a preset threshold, according to application scenario requirements, a fourth encoding method of performing global incremental encoding on the current frame image using the first encoding method or the second encoding method; When it is determined that there are multiple merged change regions whose sizes are smaller than the preset threshold, a fifth encoding method of performing region incremental encoding on the current frame image using the third encoding method.

[0008] Further, in step S3, encoding the current frame image based on the encoding method to obtain encoded data includes: S31. For a current frame image using a global incremental encoding method, a global incremental encoder is used to encode the current frame image to obtain corresponding encoded data, wherein the encoded data is accompanied by image encoding method information.

[0009] Further, in step S3, encoding the current frame image based on the encoding method to obtain encoded data includes: S31, for a current frame image using a region increment coding method, independently encode each changed region image block in the current frame image to generate a corresponding coded segment, wherein the coded segment is accompanied by coding method information used for the corresponding changed region image block and Rect information indicating the region position and size of the image block; S32, for each coding segment, serialization processing is performed based on the coding segment, the corresponding coding mode information, and the Rect information to obtain a corresponding structured data segment; S33: Combine the structured data segments to obtain corresponding encoded data.

[0010] Furthermore, in step S4, during the process of displaying the current frame image, the method includes: If the client decodes and recovers a complete image that is consistent with the original picture, the complete image is directly used as the currently displayed frame image and cached for subsequent use; If the client decodes and recovers multiple image blocks in the changed region, the multiple image blocks in the changed region are merged with the previous frame image according to their position information to update the currently displayed frame image.

[0011] In a second aspect, the present application discloses an intelligent encoding and decoding system based on video content in a cloud desktop scenario, the system comprising a server and a client, wherein: The server is used to determine the image blocks in the changed area and the content types corresponding to the image blocks in the changed area by using image difference and content recognition technology for the current frame image of the cloud host system desktop; determine the encoding method of the current frame image based on the size, distribution, and content type of the image blocks in the changed area within the encoding and decoding capability of the cloud host system; encode the current frame image based on the encoding method to obtain encoded data; encrypt the encoded data and transmit the obtained encrypted data to the client; The client is used to perform corresponding decryption and decoding processing on the acquired encrypted data and then display the current frame image.

[0012] The present invention has the following beneficial effects: 1) Through image difference and content recognition technology, the image blocks of the changed area in the current frame image of the cloud host system desktop can be accurately identified, avoiding indiscriminate encoding of the entire frame image, thereby reducing unnecessary encoding and decoding consumption and data transmission; 2) Dynamically select the most suitable encoding method according to the size, distribution and content type of the image blocks in the changing area, which can further improve the encoding efficiency and transmission speed; and select the encoding method according to the content type to ensure that different types of video content are properly processed during the transmission process, thereby maintaining the stability and clarity of the video quality; 3) Encrypting the encoded data can effectively prevent the data from being stolen or tampered with during transmission, ensuring the security of the cloud desktop system and the privacy of user data. After the encrypted data is transmitted to the client, the client performs the corresponding decryption and decoding processing, achieving end-to-end data protection and further enhancing the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A method flow chart of an intelligent encoding and decoding method based on video content in a cloud desktop scenario provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of the overall process of regional incremental coding using an adaptive hybrid coding form; Figure 3 A system structure diagram of an intelligent encoding and decoding system based on video content in a cloud desktop scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of an intelligent encoding and decoding method and system based on video content in a cloud desktop scenario proposed by the present invention, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0016] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0017] The following is a detailed description of a specific solution of an intelligent encoding and decoding method and system based on video content in a cloud desktop scenario provided by the present invention in conjunction with the accompanying drawings.

[0018] See also Figure 1 , which shows a method flow chart of an intelligent encoding and decoding method based on video content in a cloud desktop scenario provided by an embodiment of the present invention, the method comprising: Step S1, for the current frame image of the cloud host system desktop, determine the changed area image block and the content type corresponding to the changed area image block through image difference and content recognition technology.

[0019] Specifically, this application will compare the current frame image with the previous frame image to obtain the image change area. Taking into account the need to improve coding efficiency and decoding quality, this application will perform image segmentation and classification based on image content based on the image change area. This process will carefully analyze the image features in the image change area, such as color, texture, shape, etc., to obtain more accurate image blocks in the change area, as well as the content types corresponding to these image blocks (such as text, images, videos, etc.). Such processing not only helps to reduce the redundancy of encoded data and improve coding efficiency, but also can more accurately restore the original image during decoding, thereby improving decoding quality.

[0020] Step S2, within the coding and decoding capability of the cloud host system, based on the size, distribution, and content type of the image blocks in the changed area, determine the coding method for the current frame image.

[0021] Specifically, when determining the encoding method of the current frame image, the server will take multiple factors into consideration. Among them: (1) For the first frame, considering that it is the starting frame in the video sequence and there is no previous frame for comparison, this application uses global incremental coding to encode the image. Global incremental coding means encoding the entire image area of ​​the current frame, not just the changed area, because for the first frame, the entire image is new and needs to be transmitted. Therefore, global incremental coding can ensure that the first frame can be completely and accurately encoded and transmitted to the client, providing a basis for incremental coding of subsequent frames; (2) For key frame images preset in the video sequence (such as I frames, i.e., intra-coded frames), considering that these key frame images need to be decoded independently during the decoding process, and they serve as reference frames for subsequent frames and need to contain complete image information, the server will use global incremental encoding to encode the image. This processing method ensures the high quality and high fidelity of the key frame images, and provides a solid foundation for the accurate decoding and display of subsequent frames.

[0022] (3) The server will also analyze the size of the image blocks in the changing area. For larger image blocks, considering that these large image blocks may contain more image details or complex texture information, if block-based regional incremental coding is used, the coding efficiency may be reduced or the image quality may be damaged. Therefore, the server will use the global incremental coding method (i.e., the global coding method for the large image block) to encode the image. For the case where there are multiple smaller image blocks, considering that these small image blocks may contain different types of image content, and each content type has different requirements for coding efficiency and image quality. Therefore, the server will use adaptive hybrid coding to perform regional incremental coding according to the content type corresponding to each image block. This processing method can ensure that the most appropriate encoding method is selected for different types of image content, thereby maximizing the coding efficiency and data compression ratio while maintaining image quality. By intelligently analyzing the content of each small image block and selecting the best encoding strategy accordingly, the server can optimize the transmission of video streams, reduce bandwidth usage, and improve user experience.

[0023] Step S3, encoding the current frame image based on the encoding method to obtain encoded data.

[0024] Specifically, the server will perform actual encoding operations on the current frame image according to the previously determined encoding method (which may be global incremental encoding or regional incremental encoding). If the current frame image is determined to be a key frame image or contains a large image block, the server may use global incremental encoding to encode the entire frame image to ensure the integrity and high-quality transmission of the image information. For non-key frame images that contain multiple smaller image blocks, the server will encode each image block separately using regional incremental encoding according to the content type of each image block (such as text, image, video, etc.). After the encoding process, a bit stream containing the encoded data of the current frame image will be obtained. These encoded data will be transmitted to the client for subsequent decoding and display operations.

[0025] Step S4, encrypting the encoded data and transmitting the obtained encrypted data to the client, which performs corresponding decryption and decoding on the obtained encrypted data and displays the current frame image.

[0026] Specifically, the server will first encrypt the encoded data to ensure the security and privacy of the data during transmission. The encryption process may use advanced encryption algorithms and technologies, such as symmetric encryption algorithms (such as AES), asymmetric encryption algorithms (such as RSA) or hybrid encryption algorithms, and the specific choice depends on factors such as security requirements, computing resources and transmission efficiency. After the encryption is completed, the server will transmit the obtained encrypted data to the client through the network. After receiving the encrypted data, the client will perform the corresponding decryption process. The decryption process needs to use the same encryption and decryption algorithm and key or corresponding key pair as the server encryption to ensure the correct restoration of the data. After the decryption is completed, the client will obtain the original encoded data. Then, the client will decode the obtained encoded data. The decoding process is the opposite of the encoding process. It uses the same technology as the encoding process to restore the encoded data to the original image data. Finally, the client will display the decoded image data and present the current frame image on the user interface.

[0027] As can be seen from the above, the intelligent encoding and decoding method based on video content in a cloud desktop scenario disclosed in this application can accurately identify the image blocks of the changed area in the current frame image of the cloud host system desktop through image difference and content recognition technology, avoiding indiscriminate encoding of the entire frame image, thereby reducing unnecessary encoding and decoding consumption and data transmission; dynamically selecting the most suitable encoding method according to the size, distribution and content type of the image blocks in the changed area can further improve the encoding efficiency and transmission speed; and selecting the encoding method according to the content type can ensure that different types of video content are properly processed during the transmission process, thereby maintaining the stability and clarity of the video quality; encrypting the encoded data can effectively prevent the data from being stolen or tampered with during the transmission process, thereby ensuring the security of the cloud desktop system and the privacy of user data. After the encrypted data is transmitted to the client, the client performs corresponding decryption and decoding processing, realizing end-to-end data protection and further enhancing the security of the system.

[0028] In one embodiment, in step S1, for the current frame image of the cloud host system desktop, the image block of the changed area and the content type corresponding to the image block of the changed area are determined by image difference and content recognition technology, including: Step S11, obtaining the previous frame image and the current frame image of the cloud host system desktop.

[0029] Specifically, this step is to obtain the basic data and the changed data for comparison. In the Windows system, DXGI is used to achieve higher performance image capture based on content changes.

[0030] Step S12: Based on the previous frame image and the current frame image, determine the image change area through pixel difference analysis.

[0031] Specifically, by comparing the pixel differences between the previous frame image and the current frame image, it is further identified which areas in the current frame image have changed. That is, image difference analysis is performed based on the previous frame image and the current frame image to determine the changed areas in the current frame image. It should be noted that these changed areas include various changes caused by user operations, such as newly appeared windows, moved icons, and changed texts.

[0032] Step S13: identifying a target image region to be encoded based on the image change region.

[0033] Step S14: performing image segmentation and image content recognition on the target image area to determine the image blocks in the changed area and the content types corresponding to the image blocks in the changed area.

[0034] Based on step S13 to step S14, it should be noted that the present application determines which areas are the target image areas of interest and need to be encoded by performing feature analysis (such as color, texture, and shape feature analysis) and judgment on the content of the image change area. Afterwards, an image segmentation algorithm (such as an edge-based image segmentation algorithm, a threshold-based image segmentation algorithm, etc.) is used to accurately segment the target image area to extract the corresponding change area image blocks (it should be noted that what is obtained here may be one change area image block or multiple change area image blocks, which depends on the complexity of the image change area). Subsequently, the image content recognition technology is used to classify each segmented change area image block to determine the corresponding content type (such as text area, icon area, image area, video window, etc.).

[0035] In one of the embodiments, in step S2, the encoding method determined based on the size, distribution, and content type of the image blocks in the changing area includes: according to the application scenario requirements, a first encoding method or a second encoding method is used to perform global incremental encoding on the first frame image, a preset key frame image, or a current frame image with a changing area image block whose size is greater than or equal to a preset threshold using a streaming encoding form or a picture encoding form; for the current frame image with multiple second changing area image blocks whose sizes are less than the preset threshold, a third encoding method is used to perform regional incremental encoding using an adaptive hybrid encoding form according to the content type corresponding to each image block in the figure, and the hybrid encoding form includes at least one of a PNG lossy encoding form, an LZ4 lossless encoding form, an H264 lossy encoding form, and a JPEG lossy encoding form.

[0036] Specifically, first, for the first frame image, the preset key frame image, or the current frame image with a change area image block whose size is greater than or equal to the preset threshold, the server will use streaming encoding to perform global incremental encoding. This encoding method is suitable for application scenarios that require real-time transmission and playback of video streams, such as remote desktop sharing, video conferencing, etc. Through streaming encoding, the server can send the encoded data to the client in real time, and the client can decode and display while receiving, thereby reducing delays and freezes. Secondly, for images of the same type (i.e., the first frame image, the key frame image, or the frame image containing a large change area), the server will choose to use picture encoding for global incremental encoding. This encoding method focuses more on maintaining image quality and improving compression efficiency, and is suitable for application scenarios with high requirements for image quality and low requirements for real-time performance, such as image storage, image transmission, etc. Finally, for the current frame image with multiple second change area image blocks whose sizes are less than the preset threshold, the server will use adaptive hybrid encoding according to the content type corresponding to each image block. This encoding method combines the advantages of multiple encoding technologies and can select the most appropriate encoding method according to the content type of the image block.

[0037] In one of the embodiments, in step S2, for the current frame image having multiple second change area image blocks whose sizes are smaller than a preset threshold, if the content type corresponding to one of the image blocks is text office content, the image block is incrementally encoded in PNG lossy encoding; if the content type corresponding to one of the image blocks is design and drawing content and the cloud host system supports lossless hardware acceleration, the image is incrementally encoded in H265 lossless encoding; if the content type corresponding to one of the image blocks is design and drawing content and the cloud host system does not support lossless hardware acceleration, the image block is incrementally encoded in LZ4 lossless encoding; if the content type corresponding to one of the image blocks is rendering entertainment content and the cloud host system supports lossy hardware acceleration, the image block is incrementally encoded in H264 lossy encoding; if the content type corresponding to one of the image blocks is rendering entertainment content and the cloud host system does not support lossy hardware acceleration, the image block is incrementally encoded in JPEG lossy encoding.

[0038] Specifically, for image blocks whose content type is text office content, since the text information needs to maintain a high degree of clarity and readability, the server will select PNG lossy encoding for regional incremental encoding. Among them, the PNG format supports lossless compression and provides transparency support, which is very effective for content such as text and simple graphics that need to maintain clear edges. Although it is lossy encoding, by adjusting the compression level, a good compression effect can be achieved while ensuring text readability. For image blocks whose content type is design drawing content, since design drawings usually contain complex lines, shapes, and color gradients, lossless compression can better preserve these details. If the cloud host system supports lossless hardware acceleration, the server will select H265 lossless encoding for regional incremental encoding. Among them, H265 (HEVC) is an advanced video compression standard that supports efficient lossless compression. While maintaining or improving video quality, it significantly reduces the bit rate of the video, thereby reducing the demand for storage space and transmission bandwidth. With the support of hardware acceleration, H265 can achieve efficient lossless compression and real-time processing, which is very suitable for the transmission of design drawing content. If the cloud host system does not support lossless hardware acceleration, the server will select LZ4 lossless encoding for regional incremental encoding. Among them, LZ4 is an efficient compression algorithm that can achieve fast compression and decompression speed while maintaining data integrity, which is very suitable for the transmission of design drawing content. For image blocks whose content type is rendering entertainment content, the server will select the encoding form based on whether the cloud host system supports lossy hardware acceleration. If the cloud host system supports lossy hardware acceleration, the server will select H264 lossy encoding for regional incremental encoding. Among them, H264 is an advanced video compression standard that uses lossy compression technology to reduce the amount of data by removing redundant information in the image and using time correlation. With the support of hardware acceleration, H264 can achieve efficient compression and real-time processing, which is very suitable for the transmission of rendering entertainment content. If the cloud host system does not support lossy hardware acceleration, the server will select JPEG lossy encoding for regional incremental encoding. Among them, JPEG is a widely used image compression format that also uses lossy compression technology to reduce the amount of data. Although JPEG will lose some image details during the compression process, by adjusting the compression quality parameters, better compression effects can be achieved while ensuring acceptable image quality.

[0039] In one of the embodiments, for a current frame image having multiple second change area image blocks whose sizes are smaller than a preset threshold, if multiple image blocks are spatially adjacent and have the same content type, the multiple image blocks are merged to obtain a new merged change area; for the current frame image including the merged change area, the encoding method determined in step S2 also includes: when it is determined that the size of the merged change area is greater than or equal to the preset threshold, according to the application scenario requirements, a fourth encoding method of using the first encoding method or the second encoding method to perform global incremental encoding on the current frame image; when it is determined that there are multiple merged change areas whose sizes are smaller than the preset threshold, a fifth encoding method of using the third encoding method to perform regional incremental encoding on the current frame image.

[0040] In one embodiment, in step S3, encoding the current frame image based on the encoding method to obtain encoded data includes: Step S31, for the current frame image using the global incremental encoding method, a global incremental encoder is used to encode the current frame image to obtain corresponding encoded data, wherein the encoded data is accompanied by image encoding method information.

[0041] In one embodiment, in step S3, encoding the current frame image based on the encoding method to obtain encoded data includes: Step S31, for the current frame image using the area incremental encoding method, each changed area image block in the current frame image is independently encoded to generate a corresponding encoding segment, wherein the encoding segment is accompanied by the encoding method information used by the corresponding changed area image block, and the Rect information representing the area position and size of the image block.

[0042] For details, please refer to Figure 2 For the current frame image using the regional incremental encoding method, the server will first detect the difference between the current frame image and the previous frame image, and identify all the changed area image blocks. Then, each changed area image block is encoded independently. During the encoding process, a coding segment corresponding to the image block will be generated. In this coding segment, in addition to the actual encoding data, the encoding method information used by the image block and the Rect information representing the position and size of the image block area will also be included (usually including the x, y coordinates of the upper left corner and the height and width information of the image block). This information is necessary for subsequent decoding, so that the image can be correctly decoded and reconstructed.

[0043] Step S32: For each coding segment, serialization processing is performed based on the coding segment, the corresponding coding mode information, and the Rect information to obtain a corresponding structured data segment.

[0044] Specifically, the server organizes the coded data, the coding method information, and the Rect information according to a predetermined format, and converts the organized data into a format that is easy to store and transmit, such as Figure 2 The purpose of serialization is to keep the structure and content of the data unchanged during storage or transmission. After serialization, each coded segment will generate a corresponding structured data segment. These structured data segments contain all the necessary decoding information, which makes it easy to decode and reconstruct the image.

[0045] Step S33, combining the structured data segments to obtain corresponding encoded data.

[0046] For details, please refer to Figure 2 The server will integrate all structured data fragments, and the integrated data will be output as the final coded data. These coded data contain all the changed area information of the current frame image, and are organized in a structured way to facilitate the decoder to parse and reconstruct the image. In one of the embodiments, in step S4, during the display of the current frame image, the method includes: if the client decodes and recovers a complete image consistent with the original picture, directly using the complete image as the currently displayed frame image, and caching it for subsequent use; if the client decodes and recovers a plurality of changed area image blocks, merging the plurality of changed area image blocks with the previous frame image according to their position information to update the currently displayed frame image.

[0047] Specifically, if the client decodes and recovers a complete image that is consistent with the original image, it means that it contains complete image information, and there is no need to perform image merging operations. The client will use the decoded complete image as the currently displayed frame image. At the same time, the client will also cache this complete image for subsequent use. It should be noted that the purpose of caching is to reduce the need for repeated decoding and improve playback efficiency. When a subsequent frame image needs this image as a reference frame again, it can be read directly from the cache without re-decoding.

[0048] Specifically, if the client decodes and recovers multiple changed area image blocks, it will accurately place these image blocks at the corresponding positions of the previous frame image according to the position information (i.e., Rect information) of each changed area image block obtained by decoding. After completing the merging of all changed area image blocks, the client will obtain the updated current frame image, which contains all the latest change information and is ready to be displayed.

[0049] Please refer to Figure 3, the present application discloses an intelligent encoding and decoding system based on video content in a cloud desktop scenario, the system includes a server and a client, wherein: The server is used to determine the image blocks in the changed area and the content types corresponding to the image blocks in the changed area by using image difference and content recognition technology for the current frame image of the cloud host system desktop; determine the encoding method of the current frame image based on the size, distribution, and content type of the image blocks in the changed area within the encoding and decoding capability of the cloud host system; encode the current frame image based on the encoding method to obtain encoded data; encrypt the encoded data and transmit the obtained encrypted data to the client.

[0050] The client is used to perform corresponding decryption and decoding processing on the acquired encrypted data and then display the current frame image.

[0051] In one of the embodiments, the server is also used to implement the steps illustrated in any of the above method embodiments, which is not limited in this application.

[0052] As can be seen from the above, the intelligent encoding and decoding system based on video content in a cloud desktop scenario disclosed in this application can accurately identify the image blocks of the changed area in the current frame image of the cloud host system desktop through image difference and content recognition technology, avoiding indiscriminate encoding of the entire frame image, thereby reducing unnecessary encoding consumption and data transmission; dynamically selecting the most suitable encoding method according to the size, distribution and content type of the image blocks in the changed area can further improve the encoding efficiency and transmission speed; and selecting the encoding method according to the content type can ensure that different types of video content are properly processed during the transmission process, thereby maintaining the stability and clarity of the video quality; encrypting the encoded data can effectively prevent the data from being stolen or tampered with during the transmission process, thereby ensuring the security of the cloud desktop system and the privacy of user data. After the encrypted data is transmitted to the client, the client performs corresponding decryption and decoding processing, realizing end-to-end data protection and further enhancing the security of the system.

[0053] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0054] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. An intelligent encoding and decoding method based on video content in a cloud desktop scenario, characterized in that: The method is applied to the server, and includes: S1. For the current frame image of the cloud host system desktop, determine the image block in the changed area and the content type corresponding to the image block in the changed area through image difference and content recognition technology; S2. Determine the encoding method of the current frame image based on the size, distribution, and content type of the image blocks in the changed area within the encoding and decoding capabilities of the cloud host system; S3, encoding the current frame image based on the encoding method to obtain encoded data; S4. Encrypt the encoded data and transmit the obtained encrypted data to the client, which performs corresponding decryption and decoding on the obtained encrypted data and then displays the current frame image.

2. The method according to claim 1, characterized in that In step S1, for the current frame image of the cloud host system desktop, the image difference and content recognition technology are used to determine the changed area image block and the content type corresponding to the changed area image block, including: S11, obtaining the previous frame image and the current frame image of the cloud host system desktop; S12, determining the image change area through pixel difference analysis based on the previous frame image and the current frame image; S13, identifying a target image area to be encoded based on the image change area; S14: performing image segmentation and image content recognition on the target image area to determine image blocks in the changed area and content types corresponding to the image blocks in the changed area.

3. The method according to claim 1, characterized in that In step S2, the encoding method determined based on the size, distribution, and content type of the image blocks in the changed area includes: According to the application scenario requirements, a first encoding method or a second encoding method is used to perform global incremental encoding on a first frame image, a preset key frame image, or a current frame image having a change area image block whose size is greater than or equal to a preset threshold, using a streaming encoding form or a picture encoding form; For a current frame image in which there are multiple second change area image blocks with sizes smaller than a preset threshold, a third encoding method is used to perform regional incremental encoding using an adaptive hybrid encoding form according to the content type corresponding to each image block in the image. The hybrid encoding form includes at least one of a PNG lossy encoding form, an LZ4 lossless encoding form, an H264 lossy encoding form, and a JPEG lossy encoding form.

4. The method according to claim 3, characterized in that In step S2, for the current frame image having multiple second change area image blocks whose sizes are smaller than a preset threshold, if the content type corresponding to one of the image blocks is text office content, the image block is incrementally encoded in the PNG lossy encoding form; if the content type corresponding to one of the image blocks is design and drawing content and the cloud host system supports lossless hardware acceleration, the image is incrementally encoded in the H265 lossless encoding form; if the content type corresponding to one of the image blocks is design and drawing content and the cloud host system does not support lossless hardware acceleration, the image block is incrementally encoded in the LZ4 lossless encoding form; if the content type corresponding to one of the image blocks is rendering entertainment content and the cloud host system supports lossy hardware acceleration, the image block is incrementally encoded in the H264 lossy encoding form; if the content type corresponding to one of the image blocks is rendering entertainment content and the cloud host system does not support lossy hardware acceleration, the image block is incrementally encoded in the JPEG lossy encoding form.

5. The method according to claim 3, characterized in that: For a current frame image having multiple second change region image blocks whose sizes are smaller than a preset threshold, if the multiple image blocks are spatially adjacent and have the same content type, the multiple image blocks are merged to obtain a new merged change region; for the current frame image including the merged change region, the encoding method determined in step S2 also includes: When it is determined that the size of the merged change region is greater than or equal to a preset threshold, according to application scenario requirements, a fourth encoding method of performing global incremental encoding on the current frame image using the first encoding method or the second encoding method; When it is determined that there are multiple merged change regions whose sizes are smaller than the preset threshold, a fifth encoding method of performing region incremental encoding on the current frame image using the third encoding method.

6. The method according to claim 1, characterized in that In step S3, encoding the current frame image based on the encoding method to obtain encoded data includes: S31. For a current frame image using a global incremental encoding method, a global incremental encoder is used to encode the current frame image to obtain corresponding encoded data, wherein the encoded data is accompanied by image encoding method information.

7. The method according to claim 1, characterized in that In step S3, encoding the current frame image based on the encoding method to obtain encoded data includes: S31, for a current frame image using a region increment coding method, independently encode each changed region image block in the current frame image to generate a corresponding coded segment, wherein the coded segment is accompanied by coding method information used for the corresponding changed region image block and Rect information indicating the region position and size of the image block; S32, for each coding segment, serialization processing is performed based on the coding segment, the corresponding coding mode information, and the Rect information to obtain a corresponding structured data segment; S33: Combine the structured data segments to obtain corresponding encoded data.

8. The method according to claim 1, characterized in that In step S4, during displaying the current frame image, the method includes: If the client decodes and recovers a complete image that is consistent with the original picture, the complete image is directly used as the currently displayed frame image and cached for subsequent use; If the client decodes and recovers multiple image blocks in the changed region, the multiple image blocks in the changed region are merged with the previous frame image according to their position information to update the currently displayed frame image.

9. An intelligent encoding and decoding system based on video content in a cloud desktop scenario, characterized in that: The system includes a server and a client, wherein: The server is used to determine the image blocks in the changed area and the content types corresponding to the image blocks in the changed area by using image difference and content recognition technology for the current frame image of the cloud host system desktop; determine the encoding method of the current frame image based on the size, distribution, and content type of the image blocks in the changed area within the encoding and decoding capability of the cloud host system; encode the current frame image based on the encoding method to obtain encoded data; encrypt the encoded data and transmit the obtained encrypted data to the client; The client is used to perform corresponding decryption and decoding processing on the acquired encrypted data and then display the current frame image.

Images