Video coding method and device, electronic equipment, storage medium and program product

By encoding screen video frames layer by layer, using the encoding method information of the previous layer to skip some encoding mode, the problem of high video encoding complexity in the prior art is solved, and encoding efficiency and user experience are improved.

CN119011852APending Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310554837.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When existing video encoding technology processes screen video, the encoder complexity increases exponentially, resulting in a decrease in video encoding efficiency, affecting network traffic and bandwidth, and affecting the quality of users' visual experience.

Method used

A video encoding method is proposed. By dividing the video frames to be encoded, encoding layer by layer, each encoding layer selects the target encoding method from a different set of encoding methods, and using the encoding method information of the previous encoding layer to skip the partial encoding mode. Select to reduce coding complexity.

Benefits of technology

It effectively reduces the complexity of screen video encoding, improves encoding speed, reduces data processing, and improves the real-timeness of user experience and video transmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011852A_ABST
    Figure CN119011852A_ABST
Patent Text Reader

Abstract

The invention provides a video coding method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: a first coding layer selects a target coding mode of each to-be-coded unit from a first set and carries out coding, and an ith coding layer selects a candidate coding mode of each to-be-coded unit from a second set, and if the candidate coding mode of the to-be-coded unit in the ith coding layer is consistent with the target coding mode corresponding to the to-be-coded unit in the (i-1) th layer, confirming that the candidate coding mode is used as the target coding mode of the to-be-coded unit in the ith coding layer, and carrying out coding. By adopting the method provided by the invention, when the coding mode of the current coding layer is selected, the partial coding mode can be selected to be skipped based on the coding mode information of the previous coding layer, so that the effects of reducing the complexity of screen video coding and effectively improving the speed of screen video coding are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of image processing, multimedia, computer vision and cloud technology. Specifically, the present application relates to a video encoding method, device, electronic device, storage medium and program product. Background Art

[0002] Screen video is a video obtained by capturing the screen content of electronic devices such as computers / mobile phones. Screen video is widely used in video conferencing, online education, remote desktop and other scenarios. Screen video content also comes in many forms, including PPT presentations, word documents, etc. It has many obvious differences from traditional natural videos, such as the small number of colors in the local area of ​​screen video, a large number of sharp boundaries, a large number of flat areas, and a large number of repeated textures. Due to the inherent characteristics of screen video, the International Standards Organization has also developed a screen compression standard called SCC, which is an extended version of the HEVC standard. Compared with HEVC, the biggest improvement of the SCC standard is the addition of tools suitable for screen compression, IBC and palette, which can significantly improve compression performance.

[0003] The addition of new coding tools has caused the complexity of the encoder to grow exponentially, which has seriously affected the efficiency of video coding. Summary of the invention

[0004] In view of the above problems, the present application provides a video encoding method, device, electronic device, storage medium and program product, which can reduce the complexity of screen video encoding and effectively improve the speed of screen video encoding.

[0005] In a first aspect, an embodiment of the present application provides a video encoding method, the method comprising: dividing a video frame in a to-be-encoded video to obtain a plurality of to-be-encoded units; inputting each of the to-be-encoded units into an encoder having N encoding layers, wherein N is an integer greater than 1; the first encoding layer of the encoder determines a target encoding method for each of the to-be-encoded units from a first set, and encodes each of the to-be-encoded units using a corresponding target encoding method, obtains and outputs M encoding sub-units corresponding to each of the to-be-encoded units, wherein M is an integer greater than or equal to 1, and the first set includes at least three encoding methods; the i-th encoding layer of the encoder obtains the encoding sub-unit output by the i-1-th encoding layer as input to the i-th encoding layer; A unit to be encoded of a coding layer, the i-th coding layer determines a candidate encoding method for the unit to be encoded input to the i-th coding layer from a second set, if the candidate encoding method of the unit to be encoded is a first encoding method, and the target encoding method corresponding to the unit to be encoded in the i-1-th layer is also the first encoding method, then the first encoding method is used as the target encoding method of the unit to be encoded, and the unit to be encoded is encoded using the target encoding method to obtain M encoding sub-units corresponding to the unit to be encoded, wherein i is an integer greater than 1, and the second set includes the second encoding method and the first encoding method; and each encoding sub-unit encoded by the N-th coding layer of the encoder is obtained as the encoding result of encoding the video to be encoded.

[0006] In the second aspect, an embodiment of the present application provides a video encoding device, which includes: a module for dividing units to be encoded, which is used to divide video frames in a video to be encoded to obtain multiple units to be encoded; a coding method confirmation module, in a first coding layer, is used to determine a target coding method for each of the units to be encoded from a first set; in an i-th coding layer, it is used to determine the target coding method for the corresponding unit to be encoded in the i-th layer according to the target coding method of the unit to be encoded in the i-1th layer and the candidate coding method of the corresponding unit to be encoded in the i-th layer; an encoding module, which is used to encode the unit to be encoded according to the target coding method, and perform N-layer encoding on each unit to be encoded to obtain each encoding sub-unit as the encoding result of encoding the video to be encoded.

[0007] In one possible implementation, the coding method confirmation module includes a coding cost calculation module, a coding method selection module and a coding information acquisition module. The coding cost calculation module is used to calculate the coding cost corresponding to each coding method for encoding the unit to be encoded; the coding method selection module is used to select the coding method with the lowest coding cost as the target coding method for the unit to be encoded according to the coding cost corresponding to each coding method calculated by the coding cost calculation module. The coding information acquisition module is used to obtain coding layer information, information of each unit to be encoded in each coding layer, and target coding method information corresponding to each unit to be encoded; the coding method judgment module is used to judge whether the candidate coding method of the unit to be encoded in the current coding layer is consistent with the target coding method corresponding to the unit to be encoded in the previous coding layer, and whether the unit to be encoded meets the preset rules.

[0008] In one possible implementation, a coding cost calculation module is used for the i-th coding layer to calculate the coding cost of the unit to be encoded input to the i-th coding layer based on the first coding mode and the second coding mode, and obtains the coding costs corresponding to the first coding mode and the second coding mode respectively for the unit to be encoded; and a coding mode selection module is used to obtain the coding mode with the lowest coding cost among the coding costs corresponding to the first coding mode and the second coding mode respectively for the i-th coding layer, as a candidate coding mode.

[0009] In one possible implementation, if the candidate encoding method of the unit to be encoded is not the first encoding method or the target encoding method corresponding to the unit to be encoded in the i-1th layer is not the first encoding method, the i-th encoding layer calculates the encoding cost of the unit to be encoded based on the palette mode to obtain the encoding cost corresponding to the palette mode; the encoding method with the lowest encoding cost among the encoding costs corresponding to the palette mode and the candidate encoding methods respectively is obtained as the target encoding method of the unit to be encoded.

[0010] In one possible implementation, the first encoding mode is an intra-frame block copy mode, and the second encoding mode is an intra-frame prediction encoding mode, and the intra-frame prediction encoding mode includes a DC encoding mode, a planar encoding mode, or an image encoding mode based on directional prediction.

[0011] In one possible implementation, the coding cost calculation module is also used for the first coding layer to perform coding cost calculation on the unit to be encoded input to the first coding layer based on each coding method included in the first set, so as to obtain the coding cost corresponding to each coding method of the unit to be encoded; the coding method selection module is also used to obtain the coding method with the lowest coding cost among the coding costs corresponding to each coding method, as the target coding method of the unit to be encoded.

[0012] In one possible implementation, the first coding layer obtains a coding quality loss value and a bit consumption value for encoding a unit to be encoded that is input to the first coding layer based on each coding method included in the first set; the first coding layer performs weighted summation on the coding quality loss value and the bit consumption value corresponding to each coding method used for the unit to be encoded, and obtains the coding cost corresponding to each coding method used for the unit to be encoded.

[0013] In one possible implementation, the video encoding device also includes a video acquisition module for acquiring a screen video stream; and extracting key frames in the screen video stream as video frames to be encoded through a module for dividing units to be encoded; and dividing the video frames to be encoded to obtain multiple units to be encoded.

[0014] In one possible implementation manner, the video encoding device also includes a video stream sending module, which is used to determine a decoding method for each unit to be encoded based on the encoding method for each unit to be encoded in each encoding layer, and obtain decoding information of the video frame to be encoded based on the decoding method for each unit to be encoded in each encoding layer; the encoded video stream including the encoding result of encoding the video frame to be encoded and the decoding information are sent to a video content receiver, so that the video content receiver decodes the encoding result based on the decoding information to obtain the decoded video content.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above method.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code, and the program code can be called by a processor to execute the above method.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0018] Compared with the prior art, the video encoding method, device, electronic device, storage medium and program product provided by the present application, in which the video encoding method, when encoding multiple units to be encoded of a screen video layer by layer, obtains a target encoding method from a first set in the first encoding layer to encode the unit to be encoded; obtains a candidate encoding method from a second set in the i-th encoding layer, wherein the first set includes at least three encoding methods, and the second set includes a second encoding method and a first encoding method; when the candidate encoding method of the unit to be encoded in the i-th encoding layer is consistent with the target encoding method corresponding to the unit to be encoded in the i-1-th layer, the candidate encoding method is used as the target encoding method of the unit to be encoded in the current encoding layer and encoded; since the encoding methods in the second set are less than the first set, when the candidate encoding method is used as the target encoding method of the unit to be encoded in the current encoding layer, it is equivalent to skipping the selection of some encoding modes in the first set. This makes it possible to skip the selection of some coding methods in the current coding layer based on the coding method information corresponding to the unit to be encoded in the previous coding layer; with little impact on the loss of overall coding performance, the overall coding complexity is reduced, the coding efficiency is improved, and the amount of data processing in the coding process is reduced.

[0019] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a schematic diagram of an application scenario of a video encoding method provided in an embodiment of the present application.

[0022] Figure 2 It is a flowchart of a video encoding method provided in an embodiment of the present application.

[0023] Figure 3 The embodiment of this application provides Figure 2 Schematic diagram of the process of step S210.

[0024] Figure 4 The embodiment of this application provides Figure 2 Schematic diagram of the process of step S230.

[0025] Figure 5 It is a schematic diagram of corresponding units to be encoded in adjacent coding layers provided in an embodiment of the present application.

[0026] Figure 6 The embodiment of this application provides Figure 2 Schematic diagram of the process of step S240.

[0027] Figure 7 a is a schematic diagram of a palette based on a palette mode provided in an embodiment of the present application.

[0028] Figure 7 b is an index map based on the palette mode provided in an embodiment of the present application.

[0029] Figure 8 It is a signaling interaction diagram of a video sharing application scenario shown in an embodiment of the present application.

[0030] Fig. 9 It is a schematic diagram of a video encoding device provided in an embodiment of the present application.

[0031] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0033] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0034] In order to better understand and illustrate the solutions provided by the embodiments of the present application, the terms and related technologies involved in the present application are explained below:

[0035] Video is formed based on the relationship between images. In the process of video compression, each video frame represents an image. When encoding the video, the video frames can be divided into I frames, P frames and B frames. In this solution, video encoding is mainly performed on I frames.

[0036] I frame, i.e. key frame, is an intra-frame prediction frame. It uses intra-frame information for prediction. When decoding, only the data of this frame is needed to complete the decoding of the image corresponding to the frame.

[0037] Video encoding: refers to converting a file of a certain video format into a file of another video format according to a certain agreed video code stream encoding rule. Since the main task of video encoding is to reduce the storage space of video files, video encoding is also called video compression encoding or video compression, which simply means removing redundant information from video data. In the embodiment of the present application, the specific encoding tool involved in the video encoding can be referred to as an encoding method. The encoding methods include traditional encoding mode (abbreviated as IPM encoding mode), Intra block copy (IBC), Palette mode (PLT), Adaptive color transform (ACT) and Adaptive motion vector resolution, etc.

[0038] Intra block copy (IBC): A coding method whereby when IBC is used to encode a CU, the PU in the CU searches for the most similar block in the reconstructed part of the current image. IBC can be considered as "motion compensation" in the current image.

[0039] Palette mode (PLT): a coding method. When the palette mode is used to encode the CU, several grayscale values ​​that appear most frequently in the image are selected as the basic colors (base color), and index values ​​are assigned to them respectively to establish a palette. Then, each pixel value of the original image is replaced with the corresponding index value to form an index map (index map), and finally the palette and index map are predictively coded and entropy coded. By transmitting the entropy-coded palette and index map to the decoding end, the composite image can be reconstructed using the palette and index map. Among them, the pixel value can be represented by the RGB (R represents red, G represents green, and B represents blue) mode, and can also be represented by the YUV ("Y" represents brightness (Luminance), "U" represents chrominance (Chrominance), and "V" represents concentration (Chroma)) mode, which is not limited in the embodiments of the present application.

[0040] Adaptive color transform (ACT): Since most screen content uses RGB color space, removing redundancy between different color channels can greatly improve coding efficiency. In ACT, CU can adaptively convert color space. In other words, a block in RGB space can be encoded directly or converted to YCoCg (Y represents brightness value, Co represents chromaticity orange, and Cg represents chromaticity green) space before encoding.

[0041] Adaptive motion vector resolution: Unlike the continuous motion of the content captured by the camera, the motion in the screen content is often discrete and moves by pixel. Therefore, it is not necessary to perform pixel-by-pixel motion compensation for most screen motion.

[0042] Encoder: A program (or hardware) that encodes video or audio according to a certain agreed video bitstream encoding rule.

[0043] Coding cost: can be characterized by rate-distortion cost (RD Cost for short), which is mainly used to indicate the value obtained by weighting the number of bits required to encode the current coding unit using the current coding method and the coding quality loss (such as the residual of the coding unit and the prediction unit). The smaller the value, the more effective the current coding method is for the current coding unit.

[0044] When encoding a video, it is necessary to traverse the above-mentioned multiple encoding methods and select the encoding method with the lowest encoding cost for encoding. With the addition of new encoding methods, while improving compression performance, it will inevitably lead to an exponential increase in the complexity of the corresponding encoder, bringing greater challenges to network traffic and bandwidth, and affecting the user's visual experience quality. Among them, the encoding complexity of the palette mode is particularly high, and not all encoding units are suitable for encoding using the palette mode.

[0045] Therefore, in order to solve the above problems, that is, to reduce the complexity of the video encoding process existing in the related art. This application proposes a video encoding method, which mainly can quickly skip the calculation of the encoding cost of some encoding modes in the current encoding layer based on the information of the encoded encoding method of the unit to be encoded in the previous encoding layer when multiple encoding layers encode the unit to be encoded layer by layer, thereby effectively reducing the complexity of screen video encoding and improving the speed of screen video encoding, thereby reducing the power consumption of mobile phones and reducing the delay of screen sharing, and improving user experience.

[0046] The following describes an exemplary application of the video encoding method provided by an embodiment of the present invention for executing the above-mentioned video encoding method. The video encoding method provided by an embodiment of the present invention can be applied to the following: Figure 1 In the application environment shown.

[0047] Figure 1 According to a schematic diagram of an application scenario shown in an embodiment of the present application, Figure 1As shown, the application scenario includes a first terminal 10, a second terminal 20 and a server 30, and both the first terminal 10 and the second terminal 20 are communicatively connected to the server 30 via a network.

[0048] The first terminal 10 and the second terminal 20 may specifically be a mobile phone, a computer, a tablet computer or a vehicle-mounted terminal, etc. The first terminal 10 and the second terminal 20 may also be other electronic devices used for video recording or video playback.

[0049] The network can be a wide area network or a local area network, or a combination of both.

[0050] Server 30 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0051] The first terminal 10 can generate a video to be encoded and encode it before sharing it with the second terminal 20 through the server 30. The first terminal 10 can also upload the video to be encoded to the server 30 so that the server 30 can encode the video to be encoded after receiving it and then transmit it to the second terminal 20. The embodiment of the present application does not limit this.

[0052] The first terminal 10 and the second terminal 20 both include a codec for encoding and decoding a video to be decoded.

[0053] Taking the example of the first terminal 10 generating a video stream to be encoded and encoding it and then sharing it to the second terminal 20 through the server 30, the real-time performance of video sharing between the first terminal 10 and the second terminal 20 is related to the encoding and decoding efficiency of the first terminal 10 and the second terminal 20; specifically, the higher the encoding and decoding efficiency of the first terminal 10 and the second terminal 20, the stronger the real-time performance of video sharing between the first terminal 10 and the second terminal 20; the lower the encoding and decoding efficiency of the first terminal 10 and the second terminal 20, the weaker the real-time performance of video sharing between the first terminal 10 and the second terminal 20.

[0054] In the process of the first terminal 10 sharing the video with the second terminal 20, the video frames in the to-be-encoded video are divided into a plurality of to-be-encoded units, and each to-be-encoded unit is input into an encoder having N encoding layers, wherein N is an integer greater than 1; in the first encoding layer, a target encoding method corresponding to each to-be-encoded unit is selected from the encoding methods in the first set, and the first set includes at least three encoding methods; in the i-th encoding layer, the encoding sub-unit output by the i-1-th encoding layer is obtained as the to-be-encoded unit input to the i-th encoding layer, and each to-be-encoded unit is selected from the encoding methods in the second set. The candidate encoding method corresponding to the element, the second set includes the second encoding method and the first encoding method; when the candidate encoding method of the unit to be encoded and the target encoding method corresponding to the unit to be encoded in the i-1th layer are both the first encoding method, the first encoding method is used as the target encoding method of the unit to be encoded, and the unit to be encoded is encoded using the target encoding method, and finally, each encoding sub-unit obtained by encoding the Nth encoding layer of the encoder is obtained as the encoding result of encoding the video to be encoded, so as to obtain the encoding result corresponding to each video frame in the video to be encoded (that is, the encoded video obtained by encoding the video to be encoded). Through the video encoding method provided by the present application, when the first terminal 10 encodes the video to be encoded, it is possible to quickly skip the selection and calculation process of part of the encoding method based on the encoded information of the unit to be encoded, thereby reducing the complexity of the first terminal 10 encoding the video to be encoded, and improving the encoding efficiency of the first terminal 10 for the video to be encoded, thereby realizing the sharing of the video stream to be encoded generated by the first terminal 10 to the second terminal 20, and at the same time, improving the real-time performance of video transmission between the first terminal 10 and the second terminal 20.

[0055] It is understandable that the video encoding method involved in the embodiments of the present application can be applied to any application scenario that requires video encoding and to various products related to video encoding, such as video transmission, video sharing and other scenarios. Among them, the video sharing scenario may include but is not limited to application scenarios such as video calls, video websites, remote conferences, live broadcasts, etc. For example, it can be applied to screen sharing during video calls in WeChat, that is, sharing the content on the screen of the local mobile phone to the other party, such as documents, web pages, etc. In these application scenarios, the acquired video can be encoded based on the video encoding method provided in the embodiments of the present application, and the embodiments of the present application do not limit the method for acquiring the video. After acquiring the encoded video, other processing can also be performed on the encoded video, for example, the encoded video is shared, etc.

[0056] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0057] Figure 2A flow chart of a video encoding method provided by an embodiment of the present application is given. The method comprises:

[0058] S210, dividing a video frame in the video to be encoded to obtain a plurality of units to be encoded;

[0059] The video to be encoded may be a video of a movie or TV series, a video of a live broadcast or a conversation, or a screen video.

[0060] There are many ways to obtain the video to be encoded. The screen video stream can be obtained by image acquisition through an image acquisition device, or by downloading the screen video stream through the network at a terminal, or by screen recording of a video playback device. The embodiments of the present application do not limit this.

[0061] The sizes of the multiple units to be encoded obtained by dividing the video frame in the video to be encoded may be the same or different; wherein the size of the unit to be encoded may be represented by the number of pixels contained in the unit to be encoded.

[0062] The video frame represents a codable video frame that can be intra-coded in the video to be coded. If each frame in the video to be coded is a codable video frame that can be intra-coded, the video frame is each frame in the video stream to be coded. Otherwise, the codable video frame that can be intra-coded in the video stream to be coded can be determined, and the determined codable video frame is determined as the video frame.

[0063] There are many ways to determine the codable video frames. The content of each frame can be identified and the codable video frames can be determined based on the identified frame content. The features of each frame can also be extracted and the codable video frames can be determined based on the extracted features. The specific method is not specifically limited in the embodiments of the present application.

[0064] In some possible implementations, such as Figure 3 , Figure 3 The embodiment of the present application provides Figure 2 Schematic diagram of the process of step S210, step S210 includes:

[0065] S211, obtaining screen video stream;

[0066] Among them, screen video refers to the video obtained by capturing the screen content of electronic devices such as computers / mobile phones. Compared with traditional natural videos, screen video has many obvious differences. For example, the local area of ​​the screen video has fewer colors, a large number of sharp boundaries, a large number of flat areas, and a large number of repeated textures. For local areas with fewer colors in the screen video, such as text in the screen video, using the palette mode to establish a color palette for the pixels in the local area can greatly improve the performance of screen video encoding.

[0067] S212, extracting key frames in the screen video stream as video frames to be encoded;

[0068] Among them, the extraction of key frames can be to classify each video frame in the screen video stream according to the frame content in each video frame, and divide each video frame into I frame, B frame and P frame, I frame is the key frame, and I frame is used as the video frame to be encoded; it can also be by extracting the deep features of the video frame, clustering them, and using clarity screening in each type of video frame to take out the clearest video frame as the key frame extracted for the first time; then use the point density method to perform secondary optimization on the key frame extracted for the first time to obtain the final extracted key frame, and the embodiment of the present application does not limit this.

[0069] S213, dividing the video frame to be encoded to obtain a plurality of units to be encoded.

[0070] Among them, when dividing each video frame to be encoded, the video frame to be encoded can be first divided into multiple code tree units (CTUs), and each code tree unit can be divided into multiple units to be encoded. Exemplarily, the size of each unit to be encoded can be n×n, where n is the number of pixels, such as 8×8, 16×16 or 32×32; and each unit to be encoded can be further divided into multiple sub-units to be encoded based on the method of obtaining the above-mentioned units to be encoded, forming multiple levels of units to be encoded. Therefore, each unit to be encoded in each level can be encoded based on the method of encoding each unit to be encoded proposed in the embodiment of the present application.

[0071] It can be understood that, for each video frame to be encoded, the sizes of multiple units to be encoded in the video frame to be encoded may be the same or different, and the embodiment of the present application does not impose any limitation on this.

[0072] S220, inputting each unit to be encoded into an encoder having N encoding layers, where N is an integer greater than 1;

[0073] Among them, the N encoding layers in the encoder are connected in series; the N encoding layers in the encoder can encode each unit to be decoded layer by layer, so that the features of each unit to be decoded can be extracted layer by layer, and then the dimension of the data can be reduced layer by layer, and high-dimensional data can be converted into low-dimensional data.

[0074] It can be understood that the number of coding layers in the encoder is positively correlated with the complexity of the unit to be encoded; the more complex the unit to be encoded, the more coding layers in the encoder; specifically, the complexity of the unit to be encoded can be judged according to the number of features or the number of feature types of the unit to be encoded, and the more the number of features or the number of feature types, the higher the complexity of the unit to be encoded; the number of feature types and the number of features of the unit to be encoded can also be summed, and the higher the sum of the number of feature types and the number of features, the higher the complexity of the unit to be encoded. The summing method of the number of feature types and the number of features can be a simple addition summation or a weighted summation, which is not limited here.

[0075] S230, the first encoding layer of the encoder determines a target encoding mode for each unit to be encoded from the first set, and encodes each unit to be encoded using the corresponding target encoding mode, obtains M encoding sub-units corresponding to each unit to be encoded, and outputs them, where M is an integer greater than or equal to 1, and the first set includes at least three encoding modes;

[0076] The first set is a set of encoding methods that can be used to encode the unit to be encoded.

[0077] Exemplarily, the at least three encoding methods may be at least three encoding methods of IPM, IBC, PLT and ACT, etc. It should be understood that in addition to the encoding methods listed above, there may be other encoding methods, which are not described one by one in the embodiments of the present application. All or part of the encoding method.

[0078] In this embodiment, determining the target encoding method corresponding to the unit to be encoded from the first set can be that when each encoding method in the first set encodes the unit to be encoded, the encoding quality loss value or the bit consumption value generated by the encoding is calculated, and the encoding method with the smallest encoding quality loss value or the smallest bit consumption value is selected as the target encoding method corresponding to the unit to be encoded; it can also be that the encoding quality loss value or the bit consumption value is weightedly summed to obtain the encoding cost of the encoding method, and the encoding method with the smallest encoding cost is selected as the target encoding method.

[0079] It can be understood that after each unit to be encoded is encoded, M encoding sub-units can be output, and the size of each encoding sub-unit is less than or equal to the size of the corresponding unit to be encoded.

[0080] In some possible implementations, such as Figure 4 , Figure 4 The embodiment of the present application provides Figure 2 Schematic diagram of the process of step S230, step S230 includes:

[0081] S231, the first coding layer calculates the coding cost of the unit to be coded input to the first coding layer based on each coding mode included in the first set, and obtains the coding cost corresponding to each coding mode used by the unit to be coded;

[0082] Among them, the coding cost can be pre-coded for each unit to be coded based on each coding method in the first set, and the coding cost required for each coding method in the pre-coding process is calculated, so as to obtain the coding cost corresponding to each coding method of each unit to be coded; each unit to be coded can also be divided to obtain at least one sub-unit to be coded, and one sub-unit to be coded is selected from each unit to be coded, and each sub-unit to be coded selected based on each coding method in the first set is pre-coded, and the coding cost required for each coding method in the pre-coding process is calculated to obtain the coding cost corresponding to each coding method of each sub-unit to be coded, and then the coding cost of the unit to be coded corresponding to the sub-unit to be coded is obtained.

[0083] It can be understood that for each unit to be encoded, the size of the sub-unit to be encoded selected can be arbitrary; or a sub-unit to be encoded with a fixed size ratio can be selected based on the size of the unit to be encoded. For example, when the sizes of the units to be encoded are 16×16 and 32×32, respectively, the corresponding sub-units to be encoded can be 8×8 and 16×16, respectively.

[0084] Specifically, in some embodiments, the first coding layer calculates the coding cost of the unit to be coded input to the first coding layer based on each coding mode included in the first set, and obtains the coding cost corresponding to each coding mode of the unit to be coded, which can be specifically obtained by:

[0085] The first coding layer obtains a coding quality loss value and a bit consumption value of encoding a to-be-encoded unit input to the first coding layer based on each coding mode included in the first set;

[0086] The first coding layer performs weighted summation on the coding quality loss value and the bit consumption value corresponding to each coding method adopted by the coding unit to be coded, and obtains the coding cost corresponding to each coding method adopted by the coding unit to be coded.

[0087] The encoding quality loss value is calculated by using the sum of square differences (SSE) between the reconstructed pixels after encoding and the original pixels; the bit consumption is the number of bits required for encoding using the encoding method.

[0088] It can be understood that the weights corresponding to the quality loss value and the bit consumption value can be fixed values ​​pre-set in the encoder, or can be set according to actual needs when calculating the encoding cost. The embodiment of the present application does not limit the setting method of the weight.

[0089] S232: Obtain a coding method with the lowest coding cost among the coding costs corresponding to each coding method, as the target coding method for the unit to be coded.

[0090] Among them, selecting the encoding method with the lowest encoding cost can be obtained by comparing multiple encoding methods in pairs in the process of calculating the encoding cost, retaining the encoding method with lower encoding cost through each comparison, and finally obtaining the encoding method with the lowest encoding cost as the target encoding method of the unit to be encoded; or after all the encoding costs are calculated, the encoding methods are sorted according to the encoding costs, so as to obtain the encoding method with the lowest encoding cost as the target encoding method of the unit to be encoded.

[0091] S240, the i-th coding layer of the encoder obtains the coding sub-unit output by the i-1-th coding layer as the unit to be encoded input to the i-th coding layer, and the i-th coding layer determines a candidate coding mode for the unit to be encoded input to the i-th coding layer from the second set; if the candidate coding mode of the unit to be encoded is the first coding mode, and the target coding mode corresponding to the unit to be encoded in the i-1-th layer is also the first coding mode, then the first coding mode is used as the target coding mode for the unit to be encoded, and the unit to be encoded is encoded using the target coding mode to obtain M coding sub-units corresponding to the unit to be encoded, wherein i is an integer greater than 1, and the second set includes the second coding mode and the first coding mode.

[0092] The second set is a set of encoding methods that can be used to encode the unit to be encoded.

[0093] In this embodiment, determining the candidate encoding method corresponding to the unit to be encoded from the second set can be that when each encoding method in the second set encodes the unit to be encoded, the encoding quality loss value or the bit consumption value generated by the encoding is calculated, and the encoding method with the smallest encoding quality loss value or the smallest bit consumption value is selected as the candidate encoding method corresponding to the unit to be encoded; it can also be that the encoding quality loss value or the bit consumption value is weightedly summed to obtain the encoding cost of the encoding method, and the encoding method with the smallest encoding cost is selected as the candidate encoding method.

[0094] It can be understood that, since the unit to be encoded of the i-th coding layer is obtained by taking the coding sub-unit output by the i-1-th coding layer as input, the size of the unit to be encoded of the i-1-th coding layer is greater than or equal to the size of the unit to be encoded of the i-th coding layer, and the unit to be encoded of the i-1-th coding layer contains the unit to be encoded of the i-th coding layer; specifically, when M is equal to 1, the size of the unit to be encoded of the i-1-th coding layer is equal to the size of the unit to be encoded of the i-th coding layer; when M is greater than 1, the size of the unit to be encoded of the i-1-th coding layer is greater than the size of the unit to be encoded of the i-th coding layer.

[0095] For example, Figure 5 , Figure 5 A schematic diagram of the corresponding units to be encoded in adjacent coding layers is given. Figure 5 The size of the to-be-coded unit CU of the i-th coding layer is smaller than the to-be-coded unit parent CU of the i-1-th coding layer, and the to-be-coded unit parent CU of the i-1-th coding layer includes the to-be-coded unit CU of the i-th coding layer.

[0096] It can be understood that when the candidate encoding method of the unit to be encoded is the first encoding method, and the target encoding method corresponding to the unit to be encoded in the i-1th layer is also the first encoding method, the first encoding method is used as the target encoding method of the unit to be encoded; in other words, at this time, there is no need to calculate the encoding cost of the unit to be encoded based on other encoding methods in the first set, thereby reducing the complexity of screen video encoding and effectively improving the speed of screen video encoding. Other encoding methods in the first set refer to encoding methods that exist in the first set and do not exist in the second set.

[0097] For example, if the N coding layers of the encoder encode only one unit to be encoded layer by layer, and the unit to be encoded outputs only one sub-unit to be encoded after each encoding, if the video encoding method provided by the present application is not adopted, that is, it is necessary to traverse all encoding methods at each coding layer to obtain the encoding method with the lowest encoding cost, then it is necessary to perform m N The calculation of the encoding cost is performed by the video encoding method provided by the embodiment of the present application. If there are N / 2 encoding layers, the candidate encoding method and the target encoding method of the previous encoding layer are both the first encoding method. At this time, the number of encoding cost calculations required is m. (N-N / 2) ×n N / 2 ; Where m is the number of encoding methods included in the first set, n is the number of encoding methods included in the second set, m>n; the difference in the number of encoding cost calculations under two different encoding methods is: m N / 2 ×(m N / 2 -n N / 2), it can be clearly seen that by adopting the video encoding method provided in the embodiment of the present application, the number of calculations of the encoding cost is reduced exponentially. Therefore, by adopting the video encoding method provided in the embodiment of the present application, the complexity of screen video encoding can be effectively reduced, and the speed of screen video encoding can be effectively improved.

[0098] It can be understood that if the candidate encoding method of the unit to be encoded in the i-th encoding layer is the second encoding method, and the target encoding method corresponding to the unit to be encoded in the i-1-th encoding layer is also the second encoding method, the unit to be encoded can be judged by a preset rule. When the unit to be encoded meets the preset rule, the second encoding method is used as the target encoding method of the unit to be encoded, and the unit to be encoded is encoded using the target encoding method, thereby skipping the calculation of the encoding cost of other encoding methods in the first set. Among them, the preset rule can be that the color richness of the unit to be encoded meets the preset color richness threshold, or that the directional feature of the unit to be encoded meets the preset directional feature condition. No specific limitation is made to the preset rule.

[0099] Specifically, in some possible implementation modes, the first encoding mode is an intra block copy mode (IBC), and the second encoding mode is an intra prediction encoding mode, and the intra prediction encoding mode includes a DC encoding mode, a planar encoding mode, or an image encoding mode based on directional prediction.

[0100] The intra-frame block copy mode (IBC) refers to performing motion search in the reconstructed area based on the current unit to be coded, obtaining the best block vector corresponding to the current unit to be coded, indexing the current unit to be coded based on the block vector, and then encoding. In order to reduce memory consumption and decoder complexity, the reconstructed area is limited to the area corresponding to the unit to be coded in the i-1th coding layer and the area to the left of the area, and the block vector is used to indicate the displacement from the current coding unit to the reference coding unit.

[0101] It can be understood that when the candidate coding mode of the i-th coding layer is the intra block copy mode (IBC), it means that in the reconstructed area corresponding to the unit to be coded in the i-th coding layer, the coding cost of IBC is less than the coding cost of the intra prediction coding mode; when the target coding mode corresponding to the unit to be coded in the i-1th coding layer is also the intra block copy mode (IBC), it means that in the reconstructed area corresponding to the unit to be coded in the i-1th coding layer, the coding cost of IBC is less than the coding cost of the intra prediction coding mode and less than the coding cost of the palette mode. Since the unit to be coded in the i-th coding layer is an encoding sub-unit encoded by the unit to be coded in the i-1th coding layer as input, the reconstructed area corresponding to the unit to be coded in the i-th coding layer is included in the reconstructed area corresponding to the unit to be coded in the i-1th coding layer; therefore, in the reconstructed area corresponding to the unit to be coded in the i-th coding layer, the coding cost of IBC is also less than the coding cost of the palette mode, so the intra block copy mode (IBC) can be used as the target coding mode for the unit to be coded in the i-th coding layer.

[0102] That is, when the candidate coding mode of the i-th coding layer is the intra block copy mode (IBC), and the target coding mode corresponding to the unit to be encoded in the i-1-th coding layer is also the intra block copy mode (IBC), the intra block copy mode (IBC) is used as the target coding mode of the unit to be encoded in the i-th coding layer.

[0103] It can be understood that when the candidate encoding mode of the i-th encoding layer is an intra-frame prediction encoding mode, and the target encoding mode corresponding to the unit to be encoded in the i-1-th encoding layer is also an intra-frame prediction encoding mode, such as a DC encoding mode, at this time, the preset rule can be that the color richness of the unit to be encoded satisfies the preset color richness threshold. That is, the pixel information of the unit to be encoded in the current encoding layer is obtained, and the color richness of the unit to be encoded in the current encoding layer is determined based on the pixel information; when the color richness is greater than the preset color richness threshold, the intra-frame prediction encoding mode is used as the target encoding mode of the unit to be encoded, and the unit to be encoded is encoded using the target encoding mode; thereby skipping the calculation of the encoding cost of other encoding modes in the first set. Among them, the pixel information may include pixel value information, pixel color type information, and pixel density information.

[0104] In other words, when the candidate encoding method of the unit to be encoded in the i-th encoding layer is consistent with the target encoding method corresponding to the unit to be encoded in the i-1-th encoding layer, the unit to be encoded is judged according to a preset rule. When the unit to be encoded meets the preset rule, the candidate encoding method is adopted as the target encoding method of the unit to be encoded in the i-th encoding layer, and the unit to be encoded is encoded using the target encoding method, thereby skipping the calculation of the encoding cost of some encoding methods, reducing the overall encoding complexity, improving the encoding efficiency, and reducing the amount of data processing in the encoding process.

[0105] In some embodiments, the i-th coding layer determines a candidate coding mode for the to-be-coded unit input to the i-th coding layer from the second set, specifically by:

[0106] The i-th coding layer calculates the coding cost of the unit to be coded input to the i-th coding layer based on the first coding mode and the second coding mode, and obtains the coding cost corresponding to the first coding mode and the second coding mode respectively for the unit to be coded;

[0107] Among them, the process of calculating the coding cost for the units to be encoded input to the i-th coding layer based on the first coding method and the second coding method is the same or similar to the process of calculating the coding cost for each unit to be encoded in the first coding layer based on each coding method in the first set in the aforementioned step S231. Therefore, its specific description can refer to the specific description of step S231 in the previous text, and will not be repeated here one by one.

[0108] A coding method with the lowest coding cost among the coding costs corresponding to the first coding method and the second coding method at the i-th coding layer is obtained as a candidate coding method.

[0109] Among them, the encoding method with the lowest encoding cost is obtained from the first encoding method and the second encoding method as a candidate encoding method. The process is the same or similar to that of obtaining the encoding method with the lowest encoding cost in the first set in the aforementioned step S232 as the target encoding method. Therefore, its specific description can refer to the specific description of step S232 in the previous text, and will not be repeated here.

[0110] In some embodiments, the first set includes a palette mode and the second set includes a first encoding mode and a second encoding mode, such as Figure 6 , Figure 6 Given Figure 2 The flowchart of step S240 is shown in FIG. 1 , wherein step S240 further includes:

[0111] S241. If the candidate encoding method of the unit to be encoded is not the first encoding method or the target encoding method corresponding to the unit to be encoded in the i-1th layer is not the first encoding method, the i-th encoding layer calculates the encoding cost of the unit to be encoded based on the palette mode to obtain the encoding cost corresponding to the palette mode.

[0112] Among them, when calculating the encoding cost of the unit to be encoded based on the palette mode, it is necessary to determine the number of pixels cu Size in the unit to be encoded according to the size of the unit to be encoded, and determine the type diff Count of pixel values ​​contained in the unit to be encoded according to the pixel value of each pixel in the unit to be encoded, so as to obtain the palette corresponding to the unit to be encoded and the palette index value of each pixel based on the palette mode, and then perform predictive encoding and entropy encoding on the unit to be encoded based on the palette and the palette index value, and then calculate the encoding cost of the predictive encoding and entropy encoding processes to obtain the encoding cost of the palette mode.

[0113] like Figure 7 , Figure 7 A palette diagram and index diagram based on the palette mode are given, taking a sub-unit of size 4×4 in the unit to be encoded as an example, as shown in FIG. Figure 7 a. The number of pixels in the subunit is 4×4=16, that is, cuSize=16. By clustering the pixel values ​​of each pixel in the subunit, it can be determined that there are 4 pixel values ​​in the subunit, that is, 4 basic colors, and diffCount=4.

[0114] Among them, based on the distribution of these four basic colors in the subunit, the following can be established: Figure 7 By replacing the four basic color pixel values ​​in the subunit with the corresponding index values, the color palette can be formed. Figure 7 b shows the index map. Figure 7 As shown in b, taking each pixel value as an index, we can get: the pixel value of index 0 is (Y=240, U=80, V=80), the pixel value of index 1 is (Y=100, U=50, V=20), the pixel value of index 2 is (Y=200, U=200, V=250), and the pixel value of index 3 is (Y=250, U=250, V=150). Based on the palette of the current unit to be encoded and the palette index value of each pixel, predictive coding and entropy coding are performed, and then the encoding cost of the predictive coding and entropy coding process is calculated to obtain the encoding cost of the palette mode.

[0115] Among them, the process of calculating the coding cost of the prediction coding and entropy coding process to obtain the coding cost of the palette mode is the same or similar to the process of calculating the coding cost of each unit to be coded in the first coding layer based on each coding method in the first set in step S231 in the previous text. Therefore, its specific description can refer to the specific description of step S231 in the previous text, and will not be repeated here. Obviously, the coding process of the palette mode is more complicated than the coding process of other coding methods. Therefore, if the palette mode can be skipped, the complexity of the coding process can be greatly reduced.

[0116] It can be understood that, in this embodiment, the first set includes all encoding modes of the second set, and the only difference between the first set and the second set is that the first set has an additional palette mode.

[0117] When the candidate encoding method of the unit to be encoded is not the first encoding method or the target encoding method corresponding to the unit to be encoded in the i-1th layer is not the first encoding method, since the encoding cost has been calculated based on the various encoding methods in the second set when determining the candidate encoding method, it is only necessary to calculate the encoding cost of the unit to be encoded based on the palette mode to obtain the encoding cost corresponding to the palette mode, and there is no need to repeat the calculation for other encoding methods in the first set.

[0118] It can be understood that, in some embodiments, when the candidate encoding method of the unit to be encoded and the target encoding method corresponding to the unit to be encoded in the i-1th layer are both the second encoding method, if the unit to be encoded meets the preset rules, at this time, the candidate encoding method is the target encoding method, and there is no need to calculate the encoding cost of the unit to be encoded based on the palette mode; if the unit to be encoded does not meet the preset rules, the unit to be encoded in the i-th encoding layer also needs to calculate the encoding cost based on the palette mode to obtain the encoding cost corresponding to the palette mode.

[0119] That is, when the candidate encoding method of the unit to be encoded in the i-th layer is inconsistent with the target encoding method corresponding to the unit to be encoded in the i-1th layer, or the candidate encoding method of the unit to be encoded in the i-th layer and the target encoding method corresponding to the unit to be encoded in the i-1th layer are both the second encoding method and the unit to be encoded does not meet the preset rules, the i-th encoding layer needs to calculate the encoding cost of the unit to be encoded based on the palette mode to obtain the encoding cost corresponding to the palette mode.

[0120] S242 , obtaining a coding method with the lowest coding cost among the coding costs corresponding to the palette mode and the candidate coding methods, as a target coding method for the unit to be coded.

[0121] Among them, the encoding method with the lowest encoding cost is obtained from the palette mode and the candidate encoding methods as the candidate encoding method, which is the same or similar to the process of obtaining the encoding method with the lowest encoding cost in the first set in the aforementioned step S232 as the target encoding method. Therefore, its specific description can refer to the specific description of step S232 in the previous text, and will not be repeated here.

[0122] It can be understood that the encoding method with the lowest encoding cost among the palette mode and the candidate encoding methods is the encoding method with the lowest encoding cost among all the encoding methods in the first set and the second set.

[0123] S250 , obtaining each encoding sub-unit obtained by encoding the Nth encoding layer of the encoder as an encoding result for encoding the video to be encoded.

[0124] The encoding result is all the encoding sub-units obtained by encoding each unit to be encoded at the Nth encoding layer of the encoder, that is, each encoded I frame in the video stream to be encoded.

[0125] It can be understood that when there are other B frames and P frames in the video stream to be encoded, after obtaining the encoded I frame in the video stream to be encoded, each B frame and P frame can be encoded based on the relationship between each B frame, P frame and each I frame using a corresponding inter-frame coding method to obtain the encoded video stream.

[0126] In some implementations, a decoding method for each unit to be encoded may be determined based on an encoding method for each unit to be encoded in each encoding layer, and decoding information of a video frame to be encoded may be obtained based on a decoding method for each unit to be encoded in each encoding layer, after obtaining an encoding result;

[0127] The encoded video stream including the encoding result of the video frame to be encoded and the decoding information are sent to the video content receiver, so that the video content receiver decodes the encoding result based on the decoding information to obtain the decoded video content.

[0128] The above decoding information indicates the decoding method corresponding to the encoded video stream, and the above decoding indication information is determined based on the target encoding method corresponding to each unit to be encoded in the video stream to be encoded. The above decoding information may also include the encoding method for encoding each B frame and P frame in the video stream to be encoded. Thus, the receiving party can decode the received video stream based on the decoding indication information and obtain the decoded video content.

[0129] It can be understood that, in the video encoding method provided in the embodiment of the present application, the video stream to be encoded is encoded to obtain the encoded video stream, and then the encoded video stream is sent to the receiving party for decoding; since the complexity of encoding the video stream to be encoded is reduced during video encoding, the encoding speed of the video stream to be encoded is increased, thereby shortening the overall process of video encoding and decoding, rapidly improving the real-time performance of video stream sharing, and then improving the work efficiency of the corresponding users.

[0130] A video encoding method provided by the present application, when using an encoder to perform multi-layer encoding on each unit to be encoded, except that in the first encoding layer, each encoding method in the first set needs to be traversed to obtain a target encoding method with the minimum encoding cost, each subsequent encoding layer can first skip the calculation of the encoding cost of some encoding methods in the first set, which is the calculation of the encoding cost of the palette mode in the present application, and select the candidate encoding method with the minimum encoding cost from the second set, and then based on the encoding method information already selected in the previous encoding layer, confirm whether the palette encoding mode can be skipped in the current encoding layer; that is, by judging whether the candidate encoding method of the unit to be encoded in the current encoding layer is consistent with the target encoding method corresponding to the unit to be encoded in the previous encoding layer, it is confirmed whether the unit to be encoded in the current encoding layer can skip the palette mode, thereby reducing the overall encoding complexity, improving the encoding efficiency, and reducing the amount of data processing in the encoding process with less impact on the overall encoding performance.

[0131] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in combination with application scenarios.

[0132] like Figure 8 , Figure 8 This is a signaling interaction diagram of a video sharing application scenario shown in an embodiment of the present application. Taking the video sharing application scenario as a remote conference as an example, the signaling interaction process of the video encoding method provided in the embodiment of the present application is explained.

[0133] The application scenario includes a sending terminal 40 and a receiving terminal 50. During a remote conference, the video stream to be encoded generated by the sending terminal 40 needs to be shared with the receiving terminal 50, which may specifically include the following steps S400 to S490.

[0134] Step S400: triggering a screen recording plug-in through the user interface of the sending terminal 40, and recording the screen of the sending terminal 40 through the screen recording plug-in to obtain a screen video.

[0135] Step S410: the sending terminal 40 divides the video frames of the screen video according to the frame content of each frame of the screen video, and determines the I frame, B frame and P frame in the screen video, and uses the I frame as the frame to be encoded.

[0136] Step S420: For each frame to be encoded, the transmitting terminal 40 divides the frame to be encoded into a plurality of units to be encoded.

[0137] The multiple units to be encoded may be of the same size or of different sizes, such as 8×8, 16×16, or 32×32.

[0138] Step S430: Encode each unit to be encoded by the encoder in the transmitting terminal 40.

[0139] The encoder has multiple encoding layers, and the multiple encoding layers encode each unit to be encoded layer by layer.

[0140] Step S440: In the first coding layer, a target coding mode corresponding to each unit to be coded is determined from the first set, and the unit to be coded is coded using the target coding mode.

[0141] The target coding mode refers to the coding mode with the lowest coding cost. The first set includes IPM coding mode (traditional intra-frame coding mode, including DC, planar mode, and direction-based prediction mode), IBC coding mode (intra-frame block copy coding mode) and Palette coding mode (palette mode).

[0142] Specifically, for each unit to be encoded, the IPM encoding mode, the IBC encoding mode and the Palette encoding mode are traversed in turn to obtain the encoding cost corresponding to each encoding mode, and the encoding mode with the lowest encoding cost is selected as the target encoding mode.

[0143] The coding cost rdcost of each mode is calculated by weighting the coding quality loss D and the bit consumption R, that is, rdcost = D + a*R, where a is a weight parameter preset in the encoder. The coding quality loss is calculated by the sum of square differences (SSE) between the reconstructed and original pixels after coding, and the bit consumption is the number of bits required for coding using the prediction method.

[0144] Step S450: In the i-th coding layer, a candidate coding mode for the unit to be coded in the i-th coding layer is selected from the second set, a target coding mode for the unit to be coded in the i-th coding layer is determined based on the candidate coding mode, and coding is performed using the target coding mode.

[0145] The second set includes the IPM coding mode and the IBC coding mode. The candidate coding mode is the coding mode with the lowest coding cost in the second set.

[0146] It is understandable that in order to select a candidate encoding method, it is necessary to traverse the encoding methods in the second set, obtain the encoding cost of each encoding method, and select the encoding method with the lowest encoding cost as the candidate encoding method.

[0147] Specifically, for each to-be-encoded unit in the i-th layer, the encoding costs of the IPM encoding mode and the IBC mode are calculated, and the mode with the smallest encoding cost is selected as the candidate encoding mode for the current encoding unit.

[0148] If the candidate encoding mode of the unit to be encoded in the i-th encoding layer is the IBC mode, and at the same time, the unit to be encoded in the i-1th encoding layer corresponding to the unit to be encoded (that is, the encoding size of the unit to be encoded in the i-1th encoding layer is larger than the encoding size of the unit to be encoded in the i-th encoding layer, and the unit to be encoded in the i-1th encoding layer contains the unit to be encoded in the i-th encoding layer, such as Figure 5 The IBC mode is also selected, that is, the following conditions 1 and 2 are met at the same time:

[0149] Condition 1.rdcost(IBC) <rdcost(IPM)

[0150] Condition 2.bestmode(parent CU)==IBC

[0151] Then, the coding cost calculation process of the palette mode is skipped for the unit to be coded in the current coding layer, and the IBC mode is directly selected as the optimal coding mode for the unit to be coded.

[0152] If the above conditions 1 or 2 cannot be met, it is necessary to calculate the encoding cost rdcost(palette) of the palette mode, and select the mode with the smallest encoding cost from all traversed modes as the optimal encoding mode for the current block, that is, compare rdcost(IPM), rdcost(IBC) and rdcost(palette), select the mode with the smallest rdcost, and perform the encoding process based on this mode.

[0153] It can be understood that when multiple coding layers encode the coding units layer by layer, only the coding layer whose candidate coding method is inconsistent with the target coding method of the previous layer needs to calculate the coding cost of the palette mode, thereby reducing the number of coding layers that need to calculate the coding cost of the palette mode and improving the coding efficiency of the encoder.

[0154] Step S460: The sending terminal 40 obtains the encoding sub-units obtained by the Nth encoding layer encoding of the encoder as the encoding result for encoding each I frame in the encoded video stream, and encodes each B frame and each P frame in the encoded video stream according to the relationship between each B frame, P frame and each I frame, respectively, to obtain the encoded video stream.

[0155] Step S470: the sending terminal 40 sends the encoded video stream, the target encoding method for encoding each unit to be encoded in the encoded video stream, and the encoding method for encoding each B frame and each P frame to the server 30, and sends it to the receiving terminal 50 through the server 30.

[0156] The target encoding method for encoding each unit to be encoded in the video stream to be encoded and the encoding method for encoding each B frame and each P frame may be collectively referred to as decoding information.

[0157] Step S480 : the receiving terminal 50 receives the encoded video stream and the decoding information, decodes the encoded video stream based on the decoding information, and displays the decoded video stream on the user interface of the receiving terminal 50 .

[0158] By adopting the video encoding method provided by the present solution, when the sending terminal 40 encodes the screen video layer by layer, the information of the already encoded mode of the corresponding unit to be encoded in the previous encoding layer is used to quickly skip the palette mode encoding of the unit to be encoded in the current encoding layer, thereby effectively reducing the complexity of screen video encoding by the sending terminal 40, thereby reducing the power consumption of the sending terminal 40 and reducing the delay in screen sharing between the sending terminal 40 and the receiving terminal 50, thereby improving the user experience.

[0159] Based on the same principle as the video encoding method provided in the embodiment of the present application, a video encoding device 90 is also provided in the embodiment of the present application. Fig. 9 FIG. 2 is a schematic diagram of a video encoding device proposed in an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0160] The to-be-encoded unit division module 910 is used to divide the video frames in the to-be-encoded video into a plurality of to-be-encoded units;

[0161] It can be understood that the module 910 for dividing the units to be encoded can also be used to identify the content of each frame in the video to be encoded, and then identify the video frames that can be intra-coded according to the content of each frame.

[0162] The coding mode confirmation module 920 is used, in the first coding layer, to determine the target coding mode for each unit to be encoded from the first set; in the i-th coding layer, to determine the target coding mode for the corresponding unit to be encoded in the i-th layer based on the target coding mode of the unit to be encoded in the i-1th layer and the candidate coding mode for the corresponding unit to be encoded in the i-th layer.

[0163] The encoding module 930 is used to encode the units to be encoded according to the target encoding method, and perform N-layer encoding on each of the units to be encoded to obtain encoding sub-units as encoding results of the video to be encoded.

[0164] It is understandable that the encoding module 930 may also encode each B frame, P frame based on the relationship between each B frame, P frame and I frame by adopting a corresponding inter-frame encoding method, so as to obtain an encoded video stream.

[0165] In some implementations, the video encoding device 90 may further include a video acquisition module, and the video acquisition module is used to acquire screen video.

[0166] In some implementations, in the first coding layer, determining a target coding mode for each unit to be coded from the first set may be specifically performed by:

[0167] The coding method confirmation module 920 calculates the coding cost of the unit to be encoded input to the first coding layer based on the coding methods included in the first set, and obtains the coding cost corresponding to each coding method of the unit to be encoded; and obtains the coding method with the lowest coding cost among the coding costs corresponding to each coding method as the target coding method of the unit to be encoded.

[0168] It can be understood that the encoding mode confirmation module 920 includes an encoding cost calculation module, an encoding mode selection module, an encoding information acquisition module and an encoding mode determination module.

[0169] The coding cost calculation module is used to calculate the coding cost corresponding to each coding method for encoding the coding unit; specifically, the coding cost calculation module is used to calculate the coding quality loss value and the bit consumption value when each coding method is used to encode the coding unit, and to perform weighted summation on the coding quality loss value and the bit consumption value to obtain the coding cost corresponding to each coding method for the coding unit.

[0170] The coding mode selection module is used to select the coding mode with the lowest coding cost as the target coding mode of the unit to be coded according to the coding cost corresponding to each coding mode calculated by the coding cost calculation module.

[0171] The coding information acquisition module is used to obtain coding layer information, information of each unit to be coded in each coding layer, and target coding mode information corresponding to each unit to be coded.

[0172] The coding mode judgment module is used to judge whether the candidate coding mode of the unit to be coded in the current coding layer is consistent with the target coding mode corresponding to the unit to be coded in the previous coding layer, and whether the unit to be coded meets the preset rules.

[0173] In some embodiments, in the i-th coding layer, according to the target coding mode of the unit to be coded in the i-1th layer and the candidate coding mode of the corresponding unit to be coded in the i-th layer, the target coding mode corresponding to the unit to be coded in the i-th layer is determined, which can be specifically used for:

[0174] If the target encoding mode of the unit to be encoded in the i-1th layer is consistent with the candidate encoding mode of the corresponding unit to be encoded in the i-th layer, the candidate encoding mode is used as the target encoding mode of the corresponding unit to be encoded in the i-th layer;

[0175] If the target encoding method of the unit to be encoded in the i-1th layer and the candidate encoding method of the corresponding unit to be encoded in the i-th layer are inconsistent, the encoding costs of the candidate encoding methods and the palette mode are calculated respectively, and the encoding method with the lowest encoding cost is selected as the target encoding method of the corresponding unit to be encoded in the i-th layer.

[0176] In some implementations, the above device further includes a video stream sending module.

[0177] The video stream sending module is used to determine the decoding method for each unit to be encoded based on the encoding method for each unit to be encoded in each encoding layer, and obtain the decoding information of the video frame to be encoded based on the decoding method for each unit to be encoded in each encoding layer; the encoded video stream including the encoding result of the video frame to be encoded and the decoding information are sent to the video content receiver, so that the video content receiver decodes the encoding result based on the decoding information to obtain the decoded video content.

[0178] The above decoding information indicates a decoding method corresponding to the encoded video stream, and the above decoding information is determined based on a target encoding method corresponding to each unit to be encoded in the video stream to be encoded.

[0179] It is understandable that if the encoding module also encodes the B frame and the P frame, the video stream sending module also needs to send each encoded B frame and P frame and the decoding information corresponding to each B frame and P frame.

[0180] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0181] See also Fig.10 Based on the video encoding method provided in the above embodiment, the embodiment of the present application also provides another electronic device 100 including a processor 102 that can execute the above method. The electronic device 100 can be a server or a terminal device, and the terminal device can be a smart phone, a tablet computer, a computer, a portable computer or other devices.

[0182] The electronic device 100 further includes a memory 104 . The memory 104 stores a program that can execute the contents of the aforementioned embodiments, and the processor 102 can execute the program stored in the memory 104 .

[0183] Among them, the processor 102 may include one or more cores for processing data and a message matrix unit. The processor 102 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 104, and calling data stored in the memory 104. Optionally, the processor 102 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 102 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 102, but may be implemented separately through a communication chip.

[0184] The memory 104 may include a random access memory (RAM) or a read-only memory (ROM). The memory 104 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the first sample audio, the second sample audio, and the target voiceprint feature) acquired by the electronic device 100 during use.

[0185] The electronic device 100 may also include a network module and a screen. The network module is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices, such as communicating with an audio playback device. The network module may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, and the like. The network module may communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices via wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The screen may display interface content and perform data interaction, such as entering the relationship between events to be predicted, and displaying the results of relationship prediction of events to be predicted using event relationship prediction models.

[0186] In some embodiments, the electronic device 100 may further include: a peripheral interface 106 and at least one peripheral device. The processor 102, the memory 104 and the peripheral interface 106 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral interface via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency component 108, a positioning component 112, a camera 114, an audio component 116, a display screen 118 and a power supply 122.

[0187] The peripheral interface 106 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 102 and the memory 104. In some embodiments, the processor 102, the memory 104, and the peripheral interface 106 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 102, the memory 104, and the peripheral interface 106 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0188] The radio frequency component 108 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency component 108 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency component 108 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency component 108 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency component 108 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency component 108 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0189] The positioning component 112 is used to locate the current geographic location of the electronic device to implement navigation or LBS (Location Based Service). The positioning component 112 can be a positioning component based on the US GPS (Global Positioning System), Beidou system or Galileo system.

[0190] The camera 114 is used to capture images or videos. Optionally, the camera 114 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the electronic device 100, and the rear camera is arranged on the back of the electronic device 100. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera 114 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0191] The audio component 116 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 102 for processing, or input them into the RF component 108 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged at different parts of the electronic device 100. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 102 or the RF component 108 into sound waves. The speaker can be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio component 114 may also include a headphone jack.

[0192] The display screen 118 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 118 is a touch display screen, the display screen 118 also has the ability to collect touch signals on the surface or above the surface of the display screen 118. The touch signal can be input to the processor 102 as a control signal for processing. At this time, the display screen 118 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 118 can be one, and the front panel of the electronic device 100 is set; in other embodiments, the display screen 118 can be at least two, which are respectively set on different surfaces of the electronic device 100 or are folded; in some other embodiments, the display screen 118 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 100. Even, the display screen 118 can also be set as a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 118 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, machine light-emitting diode).

[0193] The power supply 122 is used to power various components in the electronic device 100. The power supply 122 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 122 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0194] The embodiment of the present application also provides a structural block diagram of a computer-readable storage medium. The computer-readable medium stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0195] The computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0196] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the method described in the above various optional implementations.

[0197] In summary, the present application provides a video encoding method, device, electronic device, storage medium and program product, which divides the video frames in the video to be encoded into multiple units to be encoded, and encodes each unit to be encoded layer by layer; in the first encoding layer, a target encoding method corresponding to each unit to be encoded is obtained by traversing multiple encoding methods; in the i-th encoding layer, a candidate encoding method corresponding to each unit to be encoded is obtained by traversing multiple encoding methods; when the candidate encoding method of the unit to be encoded is consistent with the target encoding method corresponding to the unit to be encoded in the i-1th layer, the first encoding method is used as the target encoding method of the unit to be encoded, and the unit to be encoded is encoded using the target encoding method, so that the selection and calculation process of part of the encoding method can be quickly skipped based on the encoded information of the unit to be encoded, thereby reducing the complexity of encoding the video to be encoded and improving the encoding efficiency of the video to be encoded.

[0198] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technical personnel in the field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A video encoding method, characterized in that: The method comprises: Dividing a video frame in a video to be encoded to obtain a plurality of units to be encoded; Input each of the units to be encoded into an encoder having N encoding layers, where N is an integer greater than 1; The first encoding layer of the encoder determines a target encoding mode for each of the units to be encoded from the first set, and encodes each of the units to be encoded using the corresponding target encoding mode, obtains and outputs M encoding sub-units corresponding to each of the units to be encoded, where M is an integer greater than or equal to 1, and the first set includes at least three encoding modes; The i-th encoding layer of the encoder obtains the encoding sub-unit output by the i-1-th encoding layer as the unit to be encoded input to the i-th encoding layer, and the i-th encoding layer determines a candidate encoding mode for the unit to be encoded input to the i-th encoding layer from the second set. If the candidate encoding mode of the unit to be encoded is the first encoding mode, and the target encoding mode corresponding to the unit to be encoded in the i-1-th layer is also the first encoding mode, the first encoding mode is used as the target encoding mode of the unit to be encoded, and the unit to be encoded is encoded by using the target encoding mode to obtain M encoding sub-units corresponding to the unit to be encoded, wherein i is an integer greater than 1, and the second set includes the second encoding mode and the first encoding mode; Each encoding sub-unit obtained by encoding the Nth encoding layer of the encoder is obtained as an encoding result of encoding the video to be encoded.

2. The method according to claim 1, characterized in that The i-th coding layer determines, from the second set, a candidate coding mode for a unit to be coded input to the i-th coding layer, including: The i-th coding layer calculates the coding cost of the unit to be coded input to the i-th coding layer based on the first coding mode and the second coding mode, and obtains the coding cost corresponding to the first coding mode and the second coding mode respectively for the unit to be coded; A coding mode with the lowest coding cost among the coding costs respectively corresponding to the first coding mode and the second coding mode at the i-th coding layer is obtained as a candidate coding mode.

3. The method according to claim 2, characterized in that The first set includes a palette mode and the first encoding mode and the second encoding mode included in the second set, and the method further includes: If the candidate encoding mode of the unit to be encoded is not the first encoding mode or the target encoding mode corresponding to the unit to be encoded in the i-1th layer is not the first encoding mode, the i-th encoding layer calculates the encoding cost of the unit to be encoded based on the palette mode to obtain the encoding cost corresponding to the palette mode; A coding method with the lowest coding cost among the coding costs respectively corresponding to the palette mode and the candidate coding methods is obtained as the target coding method of the unit to be encoded.

4. The method according to claim 1, characterized in that: The first encoding mode is an intra-frame block copy mode, and the second encoding mode is an intra-frame prediction encoding mode. The intra-frame prediction encoding mode includes a DC encoding mode, a planar encoding mode, or an image encoding mode based on directional prediction.

5. The method according to claim 1, characterized in that The first encoding layer of the encoder determines a target encoding mode for each of the to-be-encoded units from the first set, including: The first coding layer calculates the coding cost of the unit to be coded input to the first coding layer based on each coding mode included in the first set, and obtains the coding cost corresponding to each coding mode used by the unit to be coded; A coding method with the lowest coding cost among the coding costs corresponding to each coding method is obtained as the target coding method of the unit to be coded.

6. The method according to claim 5, characterized in that The first coding layer calculates the coding cost of the unit to be coded input to the first coding layer based on each coding mode included in the first set, and obtains the coding cost corresponding to each coding mode used by the unit to be coded, including: The first coding layer obtains a coding quality loss value and a bit consumption value for encoding a to-be-encoded unit input to the first coding layer based on each coding mode included in the first set; The first coding layer performs weighted summation on the coding quality loss value and the bit consumption value corresponding to each coding method adopted by the unit to be coded, so as to obtain the coding cost corresponding to each coding method adopted by the unit to be coded.

7. The method according to claim 1, characterized in that The acquisition involves dividing the video frames in the video to be encoded to obtain a plurality of units to be encoded, including: Get screen video stream; Extracting key frames from the screen video stream as video frames to be encoded; The to-be-encoded video frame is divided into multiple to-be-encoded units.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Determining a decoding method for each unit to be encoded based on an encoding method for each unit to be encoded in each encoding layer, and obtaining decoding information of the video frame to be encoded based on the decoding method for each unit to be encoded in each encoding layer; The encoded video stream including the encoding result of the video frame to be encoded and the decoding information are sent to a video content receiver, so that the video content receiver decodes the encoding result based on the decoding information to obtain decoded video content.

9. A video encoding device, characterized in that: The device comprises: A module for dividing units to be encoded, used for dividing video frames in a video to be encoded to obtain a plurality of units to be encoded; A coding mode confirmation module, in the first coding layer, is used to determine a target coding mode for each of the units to be coded from the first set; in the i-th coding layer, is used to determine a target coding mode for the units to be coded in the i-th layer according to the target coding modes for the units to be coded in the i-th layer and the candidate coding modes for the units to be coded in the i-th layer; The encoding module is used to encode the units to be encoded according to the target encoding method, and perform N-layer encoding on each of the units to be encoded to obtain each encoding sub-unit as an encoding result of encoding the video to be encoded.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.