Picture content hybrid coding transmission method, device, equipment and storage medium

By splitting the screen content into lossless and lossy encoded frames according to network status in cloud desktop applications, the problem of distortion of text and line content in cloud desktop applications is solved, improving user experience and reducing latency and lag risks .

CN117395429BActive Publication Date: 2024-11-12SHANGHAI JIUCHI NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311347417.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-17
Publication Date
2024-11-12
Estimated Expiration
2043-10-17

AI Technical Summary

Technical Problem

In cloud desktop applications, the text and line content distortion caused by video encoders such as H264/H265, and the network bandwidth exceeding the limit caused by direct lossless encoding transmission, resulting in user experience problems such as high latency, lag and disconnection.

Method used

By obtaining the pending picture frame, the lossless encoding parameters are determined based on the current network channel status parameters and the preset mapping relationship, the lossless encoding parameters are split into lossless encoding frames and lossy encoding frames, and mixed encoding is performed, and appropriate encoding parameters are selected in combination with the network status to achieve Extraction and transmission of lossless encoded content.

Benefits of technology

It effectively reduces picture distortion, ensures real-time transmission, reduces the risks of delay, lag and disconnection, and improves user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117395429B_ABST
    Figure CN117395429B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a method, device, equipment and storage medium for hybrid coding transmission of picture content, the method comprising obtaining a picture frame to be processed; determining the lossless coding parameters corresponding to the picture frame to be processed according to the state parameters of the current network channel and the preset mapping relationship; splitting the picture frame to be processed into a lossless coding frame and a lossy coding frame based on the macroblock division result and the lossless coding parameters of the picture frame to be processed; performing hybrid coding based on the lossless coding frame and the lossy coding frame to obtain target coding information; outputting the target coding information to the terminal through the current network channel, so that the terminal decodes and renders the target picture frame based on the target coding information. It is realized that the appropriate lossless coding parameters are selected for hybrid coding in combination with the current network status, which can effectively reduce the picture distortion while ensuring the real-time transmission of the picture, reduce the risk of transmission delay, freeze and disconnection, and improve the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, device and storage medium for hybrid coding and transmission of picture content. Background Art

[0002] Cloud desktop applications collect, encode and transmit the desktop content of the cloud server, and decode and render it on the terminal after receiving it, so as to finally achieve the effect of being able to control the cloud desktop on any local machine. Generally speaking, the screen content of the cloud desktop usually contains text, lines, icons, images, videos and other contents with different characteristics, which are often directly encoded and transmitted using video encoders such as H264 / H265.

[0003] However, video encoders such as H264 / H265 are lossy encoders. The human eye is usually sensitive to the distortion of text and lines after encoding. The text and line content encoded using the aforementioned video encoders will have obvious distortion, which provides a poor experience in cloud desktop office scenarios. In addition, direct lossless encoding and transmission of the picture content will easily exceed the network bandwidth limit of the network transmission, resulting in high latency, freezes, disconnections, and other problems when users use cloud desktop applications. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for hybrid coding transmission of picture content, which solves the problem that a single coding transmission mode in the process of picture content transmission easily leads to blurred transmission picture or high delay, jamming and disconnection, and realizes that lossless coding content that meets the attention of the human eye in the picture frame to be processed can be extracted by setting lossless coding parameters, and at the same time, suitable lossless coding parameters are selected for hybrid coding in combination with the current network status, which can effectively reduce picture distortion while ensuring the real-time transmission of picture, reduce the risk of transmission delay, jamming and disconnection, and improve the user experience.

[0005] In a first aspect, an embodiment of the present application provides a method for hybrid encoding and transmitting picture content, including:

[0006] Get the picture frame to be processed;

[0007] Determine, according to the state parameters of the current network channel and a preset mapping relationship, a lossless coding parameter corresponding to the picture frame to be processed, wherein the preset mapping relationship is an association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract lossless coding content from the picture frame to be processed according to the content information amount;

[0008] Based on the macroblock division result of the picture frame to be processed and the lossless coding parameter, splitting the picture frame to be processed into a lossless coding frame and a lossy coding frame;

[0009] Performing mixed coding based on the lossless coded frame and the lossy coded frame to obtain target coding information;

[0010] The target coding information is output to the terminal through the current network channel, so that the terminal performs decoding and rendering based on the target coding information to obtain a target picture frame.

[0011] Optionally, determining the lossless coding parameters corresponding to the to-be-processed picture frame according to the state parameters of the current network channel and a preset mapping relationship includes:

[0012] Estimate the available transmission capacity of the current network channel to obtain the current available network bandwidth;

[0013] Allocating the currently available network bandwidth based on a preset allocation rule to obtain a lossless encoding bit rate;

[0014] According to the lossless encoding bit rate and the preset mapping relationship, the lossless encoding parameters corresponding to the picture frame to be processed are determined.

[0015] Optionally, allocating the currently available network bandwidth based on a preset allocation rule to obtain a lossless encoding bit rate includes:

[0016] The set bit rate allocation coefficient is multiplied by the current available network bandwidth to obtain the lossless encoding bit rate;

[0017] Or, the currently available network bandwidth is input into a set bit rate allocation function relationship to obtain the lossless encoding bit rate.

[0018] Optionally, determining the lossless encoding parameter corresponding to the to-be-processed picture frame according to the lossless encoding bit rate and a preset mapping relationship includes:

[0019] The lossless coding parameters corresponding to the picture frame to be processed are obtained by inputting the function mapping relation of the lossless coding bit rate setting;

[0020] Or, the set parameter mapping list is traversed, and the lossless coding parameters stored in association with the lossless coding bit rate are determined as the lossless coding parameters corresponding to the picture frame to be processed.

[0021] Optionally, the lossless encoding parameters include a lossless content determination threshold and a single-frame lossless macroblock limit;

[0022] The step of splitting the picture frame to be processed into a lossless coding frame and a lossy coding frame based on the macroblock division result of the picture frame to be processed and the lossless coding parameter comprises:

[0023] According to a preset macroblock size and a preset segmentation direction, the picture frame to be processed is segmented to obtain a plurality of macroblocks;

[0024] Based on the lossless content determination threshold and the single-frame lossless macroblock limit, the multiple macroblocks are divided into lossless coded macroblocks and lossy coded macroblocks;

[0025] According to the position distribution information of the multiple macroblocks in the picture frame to be processed and the macroblock division result, the picture frame to be processed is divided into a lossless coding frame and a lossy coding frame.

[0026] Optionally, the performing hybrid coding based on the lossless coded frame and the lossy coded frame to obtain target coding information includes:

[0027] Performing lossless encoding on the lossless encoded frame to obtain first encoding information, and performing lossy encoding on the lossy encoded frame to obtain second encoding information;

[0028] The first coding information and the second coding information are integrated to obtain target coding information.

[0029] Optionally, after outputting the target coding information to the terminal through the current network channel, the method further includes:

[0030] Based on the data capacity of the first coding information, the output bit rate corresponding to the target coding information and the currently available network bandwidth, the set bit rate allocation coefficient is adjusted, and the preset mapping relationship is calibrated.

[0031] In a second aspect, an embodiment of the present application further provides a picture content mixed coding transmission device, the picture content mixed coding transmission device comprising:

[0032] An acquisition unit, configured to acquire a picture frame to be processed;

[0033] a parameter determination unit, configured to determine the lossless coding parameters corresponding to the to-be-processed picture frame according to the state parameters of the current network channel and a preset mapping relationship, wherein the preset mapping relationship is an association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract lossless coding content from the to-be-processed picture frame according to the content information amount;

[0034] A coding frame determining unit, configured to split the picture frame to be processed into a lossless coding frame and a lossy coding frame based on the macroblock division result of the picture frame to be processed and the lossless coding parameter;

[0035] A hybrid coding unit, configured to perform hybrid coding based on the lossless coding frame and the lossy coding frame to obtain target coding information;

[0036] The information output unit is configured to output the target coding information to the terminal through the current network channel, so that the terminal decodes and renders the target picture frame based on the target coding information.

[0037] Optionally, the parameter determination unit includes:

[0038] A network bandwidth estimation module is configured to estimate the available transmission capacity of the current network channel to obtain the current available network bandwidth;

[0039] A lossless bit rate determination module, configured to allocate the currently available network bandwidth based on a preset allocation rule to obtain a lossless encoding bit rate;

[0040] The lossless parameter determination module is configured to determine the lossless encoding parameters corresponding to the to-be-processed picture frame according to the lossless encoding bit rate and a preset mapping relationship.

[0041] Optional, lossless bit rate determination module, specifically configured as follows

[0042] The set bit rate allocation coefficient is multiplied by the current available network bandwidth to obtain the lossless encoding bit rate;

[0043] Or, the currently available network bandwidth is input into a set bit rate allocation function relationship to obtain the lossless encoding bit rate.

[0044] Optional, lossless parameter determination module, specifically configured as

[0045] The lossless coding parameters corresponding to the picture frame to be processed are obtained by inputting the function mapping relation of the lossless coding bit rate setting;

[0046] Or, the set parameter mapping list is traversed, and the lossless coding parameters stored in association with the lossless coding bit rate are determined as the lossless coding parameters corresponding to the picture frame to be processed.

[0047] Optionally, the lossless encoding parameters include a lossless content determination threshold and a single-frame lossless macroblock limit;

[0048] The coding frame determination unit comprises:

[0049] A picture segmentation module, configured to segment the picture frame to be processed into a plurality of macroblocks according to a preset macroblock size and a preset segmentation direction;

[0050] A macroblock division module, configured to divide the plurality of macroblocks into lossless coding macroblocks and lossy coding macroblocks based on the lossless content determination threshold and the single-frame lossless macroblock limit;

[0051] The picture frame splitting module is configured to split the picture frame to be processed into a lossless coding frame and a lossy coding frame according to the position distribution information of the multiple macroblocks in the picture frame to be processed and the macroblock division result.

[0052] Optionally, a hybrid coding unit includes:

[0053] a hybrid coding module, configured to perform lossless coding on the lossless coded frame to obtain first coding information, and perform lossy coding on the lossy coded frame to obtain second coding information;

[0054] The coding integration module is configured to integrate the first coding information and the second coding information to obtain target coding information.

[0055] Optionally, a parameter calibration unit is also included, configured as follows:

[0056] Based on the data capacity of the first coding information, the output bit rate corresponding to the target coding information and the currently available network bandwidth, the set bit rate allocation coefficient is adjusted, and the preset mapping relationship is calibrated.

[0057] In a third aspect, an embodiment of the present application further provides a picture content hybrid coding transmission device, the device comprising:

[0058] one or more processors;

[0059] a storage device configured to store one or more programs,

[0060] When the one or more programs are executed by the one or more processors, the one or more processors implement the picture content mixed coding transmission method described in the embodiment of the present application.

[0061] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are configured to execute the mixed coding and transmission method for picture content described in the embodiment of the present application.

[0062] In the embodiment of the present application, by obtaining the picture frame to be processed; according to the state parameters of the current network channel and the preset mapping relationship, the lossless coding parameters corresponding to the picture frame to be processed are determined, the preset mapping relationship is the association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract the lossless coding content from the picture frame to be processed according to the content information; based on the macroblock division result of the picture frame to be processed and the lossless coding parameters, the picture frame to be processed is split into a lossless coding frame and a lossy coding frame; based on the lossless coding frame and the lossy coding frame, mixed coding is performed to obtain the target coding information; the target coding information is output to the terminal through the current network channel, so that the terminal decodes and renders the target picture frame based on the target coding information. It is realized that the lossless coding content that meets the attention of the human eye in the picture frame to be processed can be extracted by setting the lossless coding parameters, and at the same time, the appropriate lossless coding parameters are selected for mixed coding in combination with the current network state, which can effectively reduce the picture distortion while ensuring the real-time transmission of the picture, reduce the risk of transmission delay, freeze and disconnection, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of a method for hybrid encoding and transmitting picture content provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of the content composition of a cloud desktop screen frame as an example;

[0065] Figure 3 A flowchart of a method for determining lossless coding parameters corresponding to a picture frame to be processed provided in an embodiment of the present application;

[0066] Figure 4 A flowchart of a method for splitting a picture frame to be processed into a lossless coded frame and a lossy coded frame provided in an embodiment of the present application;

[0067] Figure 5 A flowchart of a method for obtaining target coding information by hybrid coding provided in an embodiment of the present application;

[0068] Figure 6 A structural block diagram of a device for hybrid encoding and transmitting picture content provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the structure of a mixed encoding and transmission device for picture content provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

[0071] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0072] In the mixed coding and transmission method of picture content provided in the embodiment of the present application, the executor of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as PC (Personal Computer), server and other devices, and the embodiment of the present application does not limit this.

[0073] Figure 1 The present invention provides a flowchart of a method for transmitting mixed coding of picture content. The method can be implemented with a mixed coding transmission device for picture content as the execution subject, and is suitable for application scenarios where picture data from a cloud server is transmitted to a local terminal device, such as cloud desktop applications, remote control applications, etc. Figure 1 As shown, the picture content mixed coding transmission method specifically includes the following steps:

[0074] Step S101, obtaining a picture frame to be processed.

[0075] The picture frame to be processed may be the current picture frame of the interface content generated by the cloud server. Taking the cloud desktop application as an example, the picture frame to be processed may be the current picture frame corresponding to the desktop content generated by the cloud server. Figure 2 A schematic diagram of the content composition of a cloud desktop screen frame is shown as an example. Figure 2As shown, the content of the picture frame to be processed may include content with different characteristics such as text, lines, icons, images and videos, wherein the first area 101 may include picture content such as icons, images and videos, and the second area 102 may include picture content such as text and lines. The picture content hybrid coding transmission method provided in the embodiment of the present application aims to smoothly transmit the content of the picture frame to be processed while keeping the picture frame to be processed undistorted to the greatest extent.

[0076] Step S102: determine the lossless coding parameters corresponding to the picture frame to be processed according to the state parameters of the current network channel and the preset mapping relationship. The preset mapping relationship is the association relationship between the state parameters and the lossless coding parameters. The lossless coding parameters are used to extract lossless coding content from the picture frame to be processed according to the amount of content information.

[0077] Among them, the current network channel can be a data transmission channel corresponding to the network connection between the cloud server and the local terminal, and the specific network transmission protocol is not limited, and can be TCP protocol or UDP protocol. The state parameter corresponding to the current network channel can be the network connection type, the maximum transmission capacity of the network channel, and the current available transmission capacity, etc., and the preset mapping relationship can be an association relationship constructed based on the state parameters corresponding to the network channel under different network environments and the corresponding matching lossless coding parameters. It is worth noting that the lossless coding parameters are used to extract lossless coding content from the to-be-processed picture frame according to the amount of content information, and the size of the content that needs to be losslessly encoded for the to-be-processed picture frame can be determined, which is equivalent to being used to adjust the content ratio of lossy encoding and lossless encoding for the to-be-processed picture frame. Since lossless coding will perform lossless encoding on the input content without losing any information or details, the output bit rate requirement is relatively high. Therefore, by setting the preset mapping relationship between the state parameters of the network channel and the lossless coding parameters, the appropriate lossless coding parameters can be matched in combination with the available transmission capacity of the current network channel to achieve an adaptive network environment, which is conducive to reducing the risk of transmission delay, jamming and disconnection.

[0078] In addition, since the human eye has different sensitivities to different characteristics of the content in the picture frame to be processed, the text and lines have relatively small information volume and low complexity in the picture frame to be processed, so the contrast is higher, which is sensitive to the human eye and can use lossless coding, while the icons, images and video pictures have relatively high information volume and high complexity in the picture frame to be processed, so the contrast is relatively low, which is not sensitive to the human eye and can use lossy coding without affecting the user's viewing experience. Then the lossless coding parameters can be used to divide the lossless coding content according to the amount of information in the content of the picture frame to be processed. Specifically, the picture frame to be processed can be divided into macroblocks and the information entropy of each macroblock can be determined, where the information entropy is a metric used to measure the information content of a certain information source. Therefore, the larger the information entropy, the richer the information contained in the macroblock, and vice versa, the smaller the information entropy, the less information contained in the macroblock. The unit of information entropy is usually expressed in bits or nits. For example, lossless coding parameters may include a lossless content determination threshold and a single-frame lossless macroblock limit. The lossless content determination threshold may be used to determine whether the information entropy size corresponding to the macroblock is suitable for lossless coding or lossy coding. The specific threshold size may be set according to the critical information entropy of the human eye-sensitive content and the non-human eye-sensitive content. The single-frame lossless macroblock limit may be the maximum number of lossless coded macroblocks in the picture frame to be processed. Thus, the lossless coded content in the picture frame to be processed may be determined by the lossless coding parameters. Combined with the correspondence between the amount of content information and the sensitivity of the human eye, the lossless coded content that meets the attention of the human eye can be effectively extracted, thereby improving the user's perception of use and facilitating the terminal to restore the target picture frame smoothly and without distortion.

[0079] In one embodiment, Figure 3 A flowchart of a method for determining lossless encoding parameters corresponding to a picture frame to be processed provided in an embodiment of the present application, such as Figure 3 As shown, the specific process of determining the lossless encoding parameters corresponding to the picture frame to be processed includes the following steps:

[0080] Step S1021: Estimate the available transmission capacity of the current network channel to obtain the current available network bandwidth.

[0081] The available transmission capacity of a network channel refers to the currently available maximum rate or bandwidth for transmitting data through the channel, which can represent the amount of data that can be transmitted through the network channel per unit time. It is usually measured in bits per second or bytes per second, depending on the current network environment, including the performance of network equipment, network topology, signal interference, and transmission protocol limitations. The current available network bandwidth obtained by estimating the available transmission capacity of the current network channel can determine the capacity of the subsequent lossless encoding content of the picture frames to be processed, thereby achieving the purpose of hybrid encoding transmission in the adaptive network.

[0082] Step S1022: Allocate the currently available network bandwidth based on a preset allocation rule to obtain a lossless encoding bit rate.

[0083] The preset allocation rule may be to define the allocation ratio of the current available network bandwidth between lossy coding and lossless coding, so as to balance the bit rate output of lossless coding and lossy coding, and reduce the risk that the output bit rate of hybrid coding cannot adapt to the current network environment due to the large difference in bit rate output between lossless coding and lossy coding. In addition, the lossless coding bit rate and the lossy coding bit rate may be used to set the expected file size of lossless coding output and lossy coding output, so as to achieve the purpose of pre-allocating the lossy coding content and lossless coding content in the processed picture frame.

[0084] Optionally, the specific implementation process of obtaining the lossless encoding bit rate based on the preset allocation rule may include:

[0085] The lossless encoding bit rate is calculated by multiplying the set bit rate allocation coefficient with the current available network bandwidth;

[0086] Alternatively, the currently available network bandwidth is input into a set bit rate allocation function to obtain the lossless encoding bit rate.

[0087] Specifically, the rate allocation coefficient may be a distribution ratio coefficient of the lossless coding rate and the lossy coding rate. For example, the current available network bandwidth is BW, and the rate allocation coefficient is α. Then the corresponding lossless coding rate may be BW×α, and the lossy coding rate may be BW×(1-α). The rate allocation coefficient may be adjusted according to the actual network environment and the output rate after hybrid coding. In addition, the rate allocation function relationship may be a corresponding relationship obtained by fitting the combined data of the available network bandwidth and the lossless coding rate recorded in the historical records. By inputting the current available network bandwidth into the rate allocation function relationship, the lossless coding rate adapted to the current network environment may be converted, thereby improving the data reliability of the lossless coding rate and effectively balancing the rate allocation of lossless coding and lossy coding.

[0088] Step S1023: Determine the lossless coding parameters corresponding to the picture frame to be processed according to the lossless coding bit rate and the preset mapping relationship.

[0089] Among them, the preset mapping relationship can be a specific association relationship based on the lossless coding bit rate and the lossless coding parameter setting. Since the lossless coding parameters can determine the lossless coding content capacity of the picture frame to be processed, then by using the lossless coding bit rate suitable for the current network environment to match the corresponding lossless coding parameters, it can be ensured that the lossless coding content capacity is adapted to the current network environment, which is conducive to the smooth transmission of the encoded content after hybrid encoding.

[0090] Optionally, the specific process of determining the lossless encoding parameters corresponding to the picture frame to be processed may include:

[0091] The lossless coding bit rate is input into the function mapping relationship set to obtain the lossless coding parameters corresponding to the picture frame to be processed;

[0092] Or, the set parameter mapping list is traversed, and the lossless coding parameters stored in association with the lossless coding bit rate are determined as the lossless coding parameters corresponding to the picture frame to be processed.

[0093] Specifically, the function mapping relationship can be to perform function fitting estimation on the lossless coding bit rate and lossless coding parameters corresponding to the content transmission of multiple recorded historical picture frames, and establish a function mapping relationship. For example, if the lossless coding bit rate is B and the lossless coding parameters are β_1,…β_N, then the established mapping relationship can be B=f(β_1,…β_N). Thus, according to the reverse calculation of the mapping relationship, β_1,…β_N=f^(-1)(B) can be obtained, thereby accurately calculating the lossless coding parameters corresponding to the lossless coding bit rate adapted to the current network state. In addition, a parameter mapping list can also be set, and the parameter mapping list can be a list-like associated storage of the better lossless coding parameters corresponding to each lossless coding bit rate or lossless coding bit rate range in the historical coding process. Therefore, by traversing the parameter mapping list, the appropriate lossless coding parameters can be located based on the lossless coding bit rate corresponding to the current network environment, which is consistent with the historical experience data and improves the data credibility of the lossless coding parameters.

[0094] Step S103 : splitting the picture frame to be processed into a lossless coding frame and a lossy coding frame based on the macroblock division result and lossless coding parameters of the picture frame to be processed.

[0095] Among them, since the lossless coding parameters are used to extract lossless coding content from the picture frame to be processed according to the amount of content information, in order to facilitate the judgment and processing of the amount of content information, the picture frame to be processed can be divided into macroblocks, and the specific macroblock size can be determined according to the sampling format of the picture frame to be processed. The macroblock division result of the picture frame to be processed may include the macroblock size, the number of macroblocks and the macroblock position distribution. The lossless coding parameters may include a lossless content determination threshold and a single-frame lossless macroblock limit, wherein the lossless content determination threshold is determined based on the macroblock information entropy associated with the sensitivity of the human eye, and thus combined with the information entropy corresponding to each macroblock, the macroblock can be divided into lossless coding macroblocks and lossy coding macroblocks according to the lossless content determination threshold. Of course, since the number of lossless coding macroblocks divided from the picture frame to be processed based on the lossless content determination threshold may be large, the number of lossless coding macroblocks can be limited in combination with the single-frame lossless macroblock limit to balance the ratio of lossy coding macroblocks and lossless coding macroblocks. After determining the division of lossy coded macroblocks and lossless coded macroblocks of a picture frame to be processed, the picture frame to be processed can be split into lossless coded frames and lossy coded frames, wherein the frame sizes of the lossless coded frames and the lossy coded frames are consistent with those of the picture frame to be processed, the lossless coded frames only include lossless coded macroblocks, and the position distribution of the lossless coded macroblocks is consistent with the position distribution in the picture frame to be processed. Similarly, the lossy coded frames only include lossy coded macroblocks, and the position distribution of the lossless coded macroblocks is consistent with the position distribution in the picture frame to be processed.

[0096] In one embodiment, Figure 4 A flowchart of a method for splitting a to-be-processed picture frame into a lossless coded frame and a lossy coded frame is provided in an embodiment of the present application, such as Figure 4 As shown, the lossless coding parameters include a lossless content determination threshold and a single-frame lossless macroblock limit. The specific process of splitting the to-be-processed picture frame into a lossless coding frame and a lossy coding frame includes the following steps:

[0097] Step S1031 : dividing the picture frame to be processed into a plurality of macroblocks according to a preset macroblock size and a preset division direction.

[0098] Among them, the preset macroblock size can be determined based on the basic coding unit adapted to the coding standard, such as 4×4, 8×8 and 16×16, and the preset segmentation direction can be from top to bottom, from left to right according to the size of the preset macroblock size, or it can be other segmentation directions that comply with the coding regulations. By segmenting the picture frame to be processed to obtain multiple macroblocks, it is equivalent to dividing the content of the picture frame to be processed into multiple content segments for processing, which is conducive to the subsequent determination of lossless coding content.

[0099] Step S1032: divide the multiple macroblocks into lossless coding macroblocks and lossy coding macroblocks based on the lossless content determination threshold and the single-frame lossless macroblock limit.

[0100] Specifically, the information entropy of each macroblock can be compared with the lossless content determination threshold according to the set macroblock traversal direction, such as the Z-order, when the information entropy of the macroblock is lower than the lossless content determination threshold, the macroblock can be determined as a lossless coding macroblock, when the information entropy of the block is higher than the lossless content determination threshold, the macroblock can be determined as a lossy coding macroblock, when the number of lossless coding macroblocks gradually accumulates to reach the single-frame lossless macroblock limit, the coding division of the macroblock can be stopped, and the remaining macroblocks can be directly determined as lossy coding macroblocks, thereby efficiently determining the lossless coding macroblocks that meet the preset number. In addition, in order to improve the accuracy of the division, all macroblocks can be sorted according to the information entropy, and the macroblocks can be traversed from low to high according to the information entropy to determine the candidate macroblocks that meet the corresponding range of the lossless content determination threshold, and then based on the single-frame lossless macroblock limit, the target macroblock is selected from the candidate macroblocks according to the sorting information or the set information entropy range as the lossless coding macroblock.

[0101] Step S1033: split the picture frame to be processed into a lossless coding frame and a lossy coding frame according to the position distribution information of the multiple macroblocks in the picture frame to be processed and the macroblock division result.

[0102] Among them, according to the macroblock division result, the result of dividing the picture frame to be processed into lossy coded macroblocks and lossless coded macroblocks can be determined, and according to the position information of the lossy coded macroblocks and the position information of the lossless coded macroblocks, the picture frame to be processed can be split into lossless coded frames and lossy coded frames, wherein the frame sizes of the lossless coded frames and the lossy coded frames are consistent with the picture frame to be processed, the lossless coded frames only include lossless coded macroblocks, and the lossy coded frames only include lossy coded macroblocks. Therefore, the purpose of hybrid coding is achieved by independently performing lossless coding on the lossless coded frames and lossy coding on the lossy coded frames.

[0103] Step S104: performing hybrid coding based on the lossless coding frame and the lossy coding frame to obtain target coding information.

[0104] Specifically, Figure 5 A flowchart of a method for obtaining target coding information by hybrid coding is provided in an embodiment of the present application, such as Figure 5 As shown, the specific process of obtaining the target coding information through hybrid coding includes the following steps:

[0105] Step S1041: Losslessly encode the lossless coded frame to obtain first coded information, and lossily encode the lossy coded frame to obtain second coded information.

[0106] Step S1042: Integrate the first coding information and the second coding information to obtain target coding information.

[0107] Among them, the first coding information can be a binary file obtained by lossless encoding a lossless coded frame, and the second coding information can be a binary file obtained by lossy encoding a lossy coded frame. The first coding information and the second coding information can be integrated, and the corresponding binary files can be spliced ​​or fused, thereby obtaining a mixed coding result of the picture frame to be processed.

[0108] Step S105: output the target coding information to the terminal through the current network channel, so that the terminal performs decoding and rendering based on the target coding information to obtain a target picture frame.

[0109] Optionally, after the target encoding information is output to the terminal through the current network channel, the following process may also be included:

[0110] Based on the data capacity of the first coding information, the output bit rate corresponding to the target coding information and the currently available network bandwidth, the set bit rate allocation coefficient is adjusted, and the preset mapping relationship is calibrated.

[0111] Specifically, when the output bit rate corresponding to the target coding information does not match the currently available network bandwidth, it may be that the distribution ratio of the lossless coding content to the lossy coding content is inappropriate. For example, the initial value of the bit rate allocation coefficient can be set to 0.5. When the output bit rate is too low compared to the currently available network bandwidth, the bit rate allocation coefficient can be appropriately increased in combination with the data capacity of the first coding information. When the output bit rate is too high compared to the currently available network bandwidth, the bit rate allocation coefficient can be appropriately reduced in combination with the data capacity of the first coding information. In addition, the correspondence between the lossless coding bit rate and the lossless coding parameters in the preset mapping relationship can also be calibrated. After the bit rate allocation coefficient is adjusted, the lossless coding bit rate is associated with a more suitable lossless coding parameter based on the lossless coding bit rate corresponding to the first coding information, providing a more accurate reference association relationship between the lossless coding bit rate and the lossless coding parameter set based on the network environment.

[0112] In the above, by obtaining the picture frame to be processed; according to the state parameters of the current network channel and the preset mapping relationship, the lossless coding parameters corresponding to the picture frame to be processed are determined, the preset mapping relationship is the association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract the lossless coding content from the picture frame to be processed according to the content information; based on the macroblock division result of the picture frame to be processed and the lossless coding parameters, the picture frame to be processed is split into a lossless coding frame and a lossy coding frame; based on the lossless coding frame and the lossy coding frame, mixed coding is performed to obtain the target coding information; the target coding information is output to the terminal through the current network channel, so that the terminal decodes and renders based on the target coding information to obtain the target picture frame. It is realized that the lossless coding content that meets the attention of the human eye in the picture frame to be processed can be extracted by setting the lossless coding parameters, and at the same time, the appropriate lossless coding parameters are selected for mixed coding in combination with the current network status, which can effectively reduce the picture distortion while ensuring the real-time transmission of the picture, reduce the risk of transmission delay, freeze and disconnection, and improve the user experience.

[0113] Figure 6 This is a structural block diagram of a mixed coding transmission device for picture content provided in an embodiment of the present application. The device is configured to execute the mixed coding transmission method for picture content provided in the above embodiment, and has the corresponding functional modules and beneficial effects of the execution method. Figure 6 As shown, the device specifically includes:

[0114] The acquisition unit 201 is configured to acquire a picture frame to be processed;

[0115] The parameter determination unit 202 is configured to determine the lossless coding parameters corresponding to the picture frame to be processed according to the state parameters of the current network channel and a preset mapping relationship, wherein the preset mapping relationship is an association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract the lossless coding content from the picture frame to be processed according to the content information amount;

[0116] The coding frame determining unit 203 is configured to split the picture frame to be processed into a lossless coding frame and a lossy coding frame based on the macroblock division result and lossless coding parameters of the picture frame to be processed;

[0117] A hybrid coding unit 204, configured to perform hybrid coding based on the lossless coding frame and the lossy coding frame to obtain target coding information;

[0118] The information output unit 205 is configured to output the target coding information to the terminal through the current network channel, so that the terminal performs decoding and rendering based on the target coding information to obtain the target picture frame.

[0119] As mentioned above, the hybrid coding and transmission device for picture content realizes the extraction of lossless coding content that meets the attention of the human eye in the picture frame to be processed by setting lossless coding parameters, and selects appropriate lossless coding parameters for hybrid coding in combination with the current network status, which can effectively reduce picture distortion while ensuring the real-time transmission of picture, reduce the risk of transmission delay, freeze and disconnection, and improve the user experience.

[0120] In a possible embodiment, the parameter determination unit 202 includes:

[0121] A network bandwidth estimation module is configured to estimate the available transmission capacity of the current network channel to obtain the current available network bandwidth;

[0122] A lossless bit rate determination module, configured to allocate the currently available network bandwidth based on a preset allocation rule to obtain a lossless encoding bit rate;

[0123] The lossless parameter determination module is configured to determine the lossless encoding parameters corresponding to the picture frame to be processed according to the lossless encoding bit rate and the preset mapping relationship.

[0124] In a possible embodiment, the lossless bit rate determination module is specifically configured as follows:

[0125] The lossless encoding bit rate is calculated by multiplying the set bit rate allocation coefficient with the current available network bandwidth;

[0126] Alternatively, the currently available network bandwidth is input into a set bit rate allocation function to obtain the lossless encoding bit rate.

[0127] In a possible embodiment, the lossless parameter determination module is specifically configured as follows:

[0128] The lossless coding bit rate is input into the function mapping relationship set to obtain the lossless coding parameters corresponding to the picture frame to be processed;

[0129] Or, the set parameter mapping list is traversed, and the lossless coding parameters stored in association with the lossless coding bit rate are determined as the lossless coding parameters corresponding to the picture frame to be processed.

[0130] In a possible embodiment, the lossless coding parameters include a lossless content determination threshold and a single-frame lossless macroblock limit;

[0131] The coding frame determination unit 203 comprises:

[0132] A picture segmentation module, configured to segment the picture frame to be processed into a plurality of macroblocks according to a preset macroblock size and a preset segmentation direction;

[0133] A macroblock division module is configured to divide the plurality of macroblocks into lossless coding macroblocks and lossy coding macroblocks based on a lossless content determination threshold and a single-frame lossless macroblock limit;

[0134] The picture frame splitting module is configured to split the picture frame to be processed into a lossless coding frame and a lossy coding frame according to the position distribution information of multiple macroblocks in the picture frame to be processed and the macroblock division result.

[0135] In a possible embodiment, the hybrid coding unit 204 includes:

[0136] A hybrid coding module, configured to perform lossless coding on the lossless coded frame to obtain first coding information, and perform lossy coding on the lossy coded frame to obtain second coding information;

[0137] The coding integration module is configured to integrate the first coding information and the second coding information to obtain target coding information.

[0138] In a possible embodiment, it further includes a parameter calibration unit configured as follows:

[0139] Based on the data capacity of the first coding information, the output bit rate corresponding to the target coding information and the currently available network bandwidth, the set bit rate allocation coefficient is adjusted, and the preset mapping relationship is calibrated.

[0140] Figure 7 A structural diagram of a mixed encoding transmission device for picture content provided in an embodiment of the present application, such as Figure 7 As shown, the device includes a processor 301, a memory 302, an input device 303 and an output device 304; the number of processors 301 in the device can be one or more. Figure 7 A processor 301 is taken as an example; the processor 301, memory 302, input device 303 and output device 304 in the device can be connected by a bus or other means. Figure 7 The example of connecting via a bus is taken. The memory 302, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the mixed coding transmission method of the picture content in the embodiment of the present application. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 302, that is, realizes the above-mentioned mixed coding transmission method of the picture content. The input device 303 can be configured to receive input digital or character information, and generate key signal input related to the user settings and function control of the device. The output device 304 may include a display device such as a display screen.

[0141] The above-mentioned picture content mixed coding transmission device can be used to execute the picture content mixed coding transmission method provided by any of the above-mentioned embodiments, and has corresponding functions and beneficial effects.

[0142] An embodiment of the present application also provides a non-volatile storage medium containing computer executable instructions, which, when executed by a computer processor, is configured to execute a method for hybrid coding and transmission of picture content described in an embodiment of the present application, including: obtaining a picture frame to be processed; determining lossless coding parameters corresponding to the picture frame to be processed according to state parameters of a current network channel and a preset mapping relationship, wherein the preset mapping relationship is an association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract lossless coding content from the picture frame to be processed according to the amount of content information; based on a macroblock division result of the picture frame to be processed and the lossless coding parameters, splitting the picture frame to be processed into a lossless coding frame and a lossy coding frame; performing hybrid coding based on the lossless coding frame and the lossy coding frame to obtain target coding information; and outputting the target coding information to a terminal through the current network channel, so that the terminal performs decoding and rendering based on the target coding information to obtain a target picture frame.

[0143] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media and optical storage; registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.

[0144] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present application is not limited to the mixed coding transmission method of picture content as described above, and can also execute related operations in the mixed coding transmission method of picture content provided in any embodiment of the present application.

[0145] It is worth noting that in the embodiment of the above-mentioned picture content mixed coding transmission system, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the embodiments of the present application.

[0146] It should be noted that the numbering of each step in this scheme is only used to describe the overall design framework of this scheme, and does not represent the necessary order relationship between the steps. On the basis that the overall implementation process conforms to the overall design framework of this scheme, it belongs to the protection scope of this scheme, and the order of precedence in the form of text during description is not an exclusive limitation on the specific implementation process of this scheme. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. In a typical configuration, a computing device includes one or more processors (CPU), an input / output interface, a network interface, and a memory. The memory may include non-permanent memory in a computer-readable medium, a random access memory (RAM) and / or a non-volatile memory in the form of a read-only memory (ROM) or a flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0147] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0148] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for transmitting mixed coding of picture contents, characterized in that: include: Get the picture frame to be processed; Determine, according to the state parameters of the current network channel and a preset mapping relationship, a lossless coding parameter corresponding to the picture frame to be processed, wherein the preset mapping relationship is an association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract lossless coding content from the picture frame to be processed according to the content information amount; Based on the macroblock division result of the picture frame to be processed and the lossless coding parameter, splitting the picture frame to be processed into a lossless coding frame and a lossy coding frame; Performing mixed coding based on the lossless coded frame and the lossy coded frame to obtain target coding information; Outputting the target coding information to the terminal through the current network channel, so that the terminal performs decoding and rendering based on the target coding information to obtain a target picture frame; The lossless coding parameters include a lossless content determination threshold and a single-frame lossless macroblock limit; The step of splitting the picture frame to be processed into a lossless coding frame and a lossy coding frame based on the macroblock division result of the picture frame to be processed and the lossless coding parameter comprises: According to a preset macroblock size and a preset segmentation direction, the picture frame to be processed is segmented to obtain a plurality of macroblocks; Based on the lossless content determination threshold and the single-frame lossless macroblock limit, the multiple macroblocks are divided into lossless coded macroblocks and lossy coded macroblocks; According to the position distribution information of the multiple macroblocks in the picture frame to be processed and the macroblock division result, the picture frame to be processed is divided into a lossless coding frame and a lossy coding frame.

2. The method for hybrid encoding and transmitting picture content according to claim 1, characterized in that: The determining, according to the state parameters of the current network channel and the preset mapping relationship, the lossless coding parameters corresponding to the picture frame to be processed includes: Estimate the available transmission capacity of the current network channel to obtain the current available network bandwidth; Allocating the currently available network bandwidth based on a preset allocation rule to obtain a lossless encoding bit rate; According to the lossless encoding bit rate and the preset mapping relationship, the lossless encoding parameters corresponding to the picture frame to be processed are determined.

3. The method for hybrid encoding and transmitting picture content according to claim 2, characterized in that: The allocating the currently available network bandwidth based on a preset allocation rule to obtain a lossless encoding bit rate includes: The set bit rate allocation coefficient is multiplied by the current available network bandwidth to obtain the lossless encoding bit rate; Or, the currently available network bandwidth is input into a set bit rate allocation function relationship to obtain the lossless encoding bit rate.

4. The method for hybrid encoding and transmitting picture content according to claim 2, characterized in that: The determining, according to the lossless coding bit rate and the preset mapping relationship, the lossless coding parameter corresponding to the picture frame to be processed includes: The lossless coding parameters corresponding to the picture frame to be processed are obtained by inputting the function mapping relation of the lossless coding bit rate setting; Or, the set parameter mapping list is traversed, and the lossless coding parameters stored in association with the lossless coding bit rate are determined as the lossless coding parameters corresponding to the picture frame to be processed.

5. The method for hybrid encoding and transmitting picture contents according to claim 2, characterized in that: The performing hybrid coding based on the lossless coding frame and the lossy coding frame to obtain target coding information includes: Performing lossless encoding on the lossless encoded frame to obtain first encoding information, and performing lossy encoding on the lossy encoded frame to obtain second encoding information; The first coding information and the second coding information are integrated to obtain target coding information.

6. The method for hybrid encoding and transmitting picture contents according to claim 5, characterized in that: After the target coding information is output to the terminal through the current network channel, the method further includes: Based on the data capacity of the first coding information, the output bit rate corresponding to the target coding information and the currently available network bandwidth, the set bit rate allocation coefficient is adjusted, and the preset mapping relationship is calibrated.

7. A device for transmitting mixed picture content coding, characterized in that: include: An acquisition unit, configured to acquire a picture frame to be processed; a parameter determination unit, configured to determine the lossless coding parameters corresponding to the to-be-processed picture frame according to the state parameters of the current network channel and a preset mapping relationship, wherein the preset mapping relationship is an association relationship between the state parameters and the lossless coding parameters, and the lossless coding parameters are used to extract lossless coding content from the to-be-processed picture frame according to the content information amount; A coding frame determining unit, configured to split the picture frame to be processed into a lossless coding frame and a lossy coding frame based on the macroblock division result of the picture frame to be processed and the lossless coding parameter; A hybrid coding unit, configured to perform hybrid coding based on the lossless coding frame and the lossy coding frame to obtain target coding information; An information output unit, configured to output the target coding information to the terminal through the current network channel, so that the terminal performs decoding and rendering based on the target coding information to obtain a target picture frame; The lossless coding parameters include a lossless content determination threshold and a single-frame lossless macroblock limit; The coding frame determination unit comprises: A picture segmentation module, configured to segment the picture frame to be processed into a plurality of macroblocks according to a preset macroblock size and a preset segmentation direction; A macroblock division module, configured to divide the plurality of macroblocks into lossless coding macroblocks and lossy coding macroblocks based on the lossless content determination threshold and the single-frame lossless macroblock limit; The picture frame splitting module is configured to split the picture frame to be processed into a lossless coding frame and a lossy coding frame according to the position distribution information of the multiple macroblocks in the picture frame to be processed and the macroblock division result.

8. A device for hybrid encoding and transmitting picture content, the device comprising: one or more processors; A storage device is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the picture content mixed coding transmission method described in any one of claims 1-6.

9. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the picture content mixed coding transmission method according to any one of claims 1 to 6 when executed by a computer processor.