Method and apparatus for processing video image

Abstract

The embodiment of the application discloses a video image processing method and device, and the method comprises the following steps: controlling second encoding for generating a second code stream according to the encoding information of the first encoding and / or the first reconstructed image, and the second code stream is encoded by using a full intra prediction mode, and the second code stream is taken as a random access stream, so that the decoding quality of the accessed video content can be improved, the blocking effect can be reduced, and part of the pseudo effect can be eliminated, and the method belongs to the field of video coding and decoding.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and in particular to a method and apparatus for processing video images. Background Technology

[0002] Video, as an efficient means of information transmission, is widely used in the internet, television broadcasting, and various emerging media applications. With the rapid development of video encoding and decoding technologies, communication technologies, and electronic devices, more and more application scenarios have high requirements for video playback latency. Examples include video conferencing, interactive entertainment, live sports broadcasts, and other on-demand or live streaming applications.

[0003] In applications such as video conferencing, interactive entertainment, and live sports broadcasts, multiple cameras are typically deployed at different locations to capture the same scene from various angles, producing a set of video signals to provide users with a multi-angle viewing experience. Users can then choose a specific angle to view the recorded video using appropriate interactive methods. Multi-camera shooting provides users with a multi-angle, three-dimensional visual experience. More broadly, in other scenarios with high latency requirements, such as cloud virtual reality (Cloud VR) games and low-latency live streaming applications, the decoding and playback latency of video content is a key indicator affecting user experience, ensuring seamless switching between different video content.

[0004] The common method for switching video content involves dividing the encoded video into segments at fixed time intervals, with each segment starting with an I-frame. When switching video content is required, the decoding or playback continues from the segment with the most recent time point of the new content, and decoding and playback begin from the I-frame of the new content segment. This switching method has a relatively long latency from receiving the switching command to completing the content switch, which cannot meet the low latency requirements of some application scenarios. Summary of the Invention

[0005] This application provides a video image processing method and apparatus. By controlling the second encoding used to generate a second bitstream according to the encoding information of the first encoding, the second bitstream adopts full intra-frame prediction mode encoding. The second bitstream is used as a random access bitstream, which can improve the decoding quality of the accessed video content, reduce block artifacts, and eliminate some artifact effects while meeting the requirements of low latency access to video content.

[0006] In a first aspect, embodiments of this application provide a video image processing method, which may include: acquiring an image to be encoded; performing a first encoding on the image to be encoded to generate a first bitstream; and performing a second encoding in full-frame intra-prediction mode on the image to be encoded or a first reconstructed image based on the encoding information of the first encoding to generate a second bitstream. The first reconstructed image is either the first bitstream or a reconstructed image from the first encoding process.

[0007] The first coding and the second coding are two different coding methods. The first coding allows inter-frame prediction mode, while the second coding is full intra-frame prediction mode. When these two coding methods are used to encode images or reconstructed images of the same video content, the first coding is used to generate the first bitstream, and the second coding is used to generate the second bitstream. In some cases, the coding information of the first and second coding can be different. For example, the partitioning method of the first coding and the second coding can be different, and the quantization parameters of the first coding and the second coding can be different, etc. In some cases, the coding information of the first coding and the second coding can be the same. For example, the partitioning method of the first coding and the second coding can be the same, and the quantization parameters of the first coding and the second coding can be the same, etc.

[0008] At the decoding end, the first and second bitstreams serve different purposes. When the decoding end switches the displayed video content, it can first decode the second bitstream corresponding to the current moment of the video content to be displayed. Then, it can use the frames obtained from decoding the second bitstream at the corresponding moment as reference frames for decoding subsequent frames of the first bitstream. This allows for timely decoding of the first bitstream without waiting for the next I-frame, significantly reducing decoding latency. In the solution provided in the first aspect above, the second encoding is performed based on the encoding information of the first encoding to obtain the second bitstream. This ensures that the quality of the second bitstream is comparable to that of the first bitstream. During decoding, using the decoding result of the second bitstream as a reference frame for the first bitstream allows for smooth transitions without noticeable blockiness or artifacts during content switching, which could negatively impact the user experience.

[0009] Therefore, in the implementation of the first aspect described above, based on the encoding information of the first encoding, the image to be encoded or the first reconstructed image is subjected to the second encoding in the full-frame prediction mode, so as to achieve that the reconstructed image of the first bitstream and the reconstructed image of the corresponding second bitstream have the same or equivalent quality. Thus, while satisfying the low-latency access to video content, it is beneficial to reduce the block effect and some artifact effect caused by encoding and decoding inconsistency, and improve the decoding quality of the accessed video content.

[0010] In one possible design, the encoding information of the first code includes one or more of the following: the division method of the first code, the quantization parameters of the first code, and the encoding distortion information of the first code.

[0011] The partitioning method of the first code may include the TU partitioning method, the PU partitioning method, or the CU partitioning method. Using the CU partitioning method of the first code, the second code may adopt the same CU partitioning method as the first code, or the CUs of the second code may not cross the CU boundaries of the first code.

[0012] Taking the encoding information of the first code as including the quantization parameters of the first code as an example, the quantization parameters of the second code can be based on the quantization parameters of the first code plus a quantization parameter offset.

[0013] In one possible design, the second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode, based on the encoding information of the first encoding, includes at least one of the following: performing the second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode using the same partitioning method as the first encoding; or performing the second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode using the same quantization parameters as the first encoding; or determining the quantization parameters of the second encoding based on the quantization parameters and quantization parameter offset of the first encoding, and performing the second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode based on the quantization parameters of the second encoding; or determining the quantization parameters of the second encoding based on the encoding distortion information of the first encoding, and performing the second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode based on the quantization parameters of the second encoding.

[0014] In one possible design, performing a second encoding of the intra-frame prediction mode on the image to be encoded or the first reconstructed image based on the encoding information of the first encoding includes: determining quantization parameters for the second encoding based on the encoding information of the first encoding and the feature information of the image to be encoded; and performing the second encoding of the intra-frame prediction mode on the image to be encoded or the first reconstructed image based on the quantization parameters of the second encoding.

[0015] In one possible design, the feature information of the image to be encoded includes one or more of the following: the content complexity of the image to be encoded, the color classification information of the image to be encoded, the contrast information of the image to be encoded, and the content segmentation information of the image to be encoded.

[0016] The feature information of the image to be encoded can be obtained by performing feature analysis on the image to be encoded.

[0017] In one possible design, performing a second encoding of the intra-frame prediction mode on the image to be encoded or the first reconstructed image based on the encoding information of the first encoding includes: determining at least one of a first partitioning method or a first encoding parameter for the second encoding of the image to be encoded or the first reconstructed image based on the encoding information of the first encoding and the first reconstructed image; and performing a second encoding of the intra-frame prediction mode on the image to be encoded or the first reconstructed image based on at least one of the first partitioning method or the first encoding parameter.

[0018] In the first bitstream, the interval between two adjacent intra-prediction modes is greater than the interval between two adjacent intra-prediction modes in the second bitstream.

[0019] The first stream can also be called the long GOP stream, and the second stream can also be called the random access stream. When video content needs to be accessed, the second stream is decoded first. The decoding result of the second stream is used as a reference frame to decode the first stream, so as to achieve fast access to video content.

[0020] Therefore, in the above possible designs, based on the encoding information of the first code and the first reconstructed image, at least one of the first partitioning method or the first encoding parameter is determined for encoding the second bitstream, so as to achieve that the reconstructed image of the first bitstream and the reconstructed image of the corresponding second bitstream have the same or equivalent quality. This is beneficial to reduce the block effect and some artifact effect caused by encoding and decoding inconsistency, and improve the decoding quality of the accessed video content while satisfying the low latency access video content.

[0021] In one possible design, the difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, and the second reconstructed image is a reconstructed image in the second bitstream or the second encoding process.

[0022] By controlling the difference between the first reconstructed image and the second reconstructed image to be less than the difference threshold or the similarity between the first reconstructed image and the second reconstructed image to be higher than the similarity threshold, the quality of the reconstructed image of the first bitstream and the reconstructed image of the corresponding second bitstream are the same or equivalent. This helps to reduce the block effect and some artifact effects caused by encoding and decoding inconsistencies while meeting the requirements for low-latency access to video content, thereby improving the decoding quality of the accessed video content.

[0023] In one possible design, determining at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the image to be encoded or the first reconstructed image, based on the encoding information of the first encoding and the first reconstructed image, includes: determining multiple second partitioning methods based on the encoding information of the first encoding and the first reconstructed image, and selecting one of the multiple second partitioning methods as the first partitioning method; and / or, determining multiple second encoding parameters based on the encoding information of the first encoding and the first reconstructed image, and selecting one of the multiple second encoding parameters as the first encoding parameter.

[0024] Among them, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and multiple third reconstructed images. The multiple third reconstructed images include the second reconstructed image. The multiple third reconstructed images are reconstructed images obtained by performing multiple second encoding processes on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters. Alternatively, the multiple third reconstructed images are reconstructed images of multiple third bitstreams. The multiple third bitstreams are obtained by performing multiple second encoding processes on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters.

[0025] Multiple third bitstreams are obtained by performing multiple second encodings on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters. By comparing the similarity between each of the multiple third reconstructed images and the first reconstructed image, the one with the highest similarity is selected as the second reconstructed image, and its corresponding third bitstream is used as the second bitstream. In this way, by maximizing the similarity between the first reconstructed image and the second reconstructed image, the blocking artifacts and some artifacts caused by encoding and decoding inconsistencies can be reduced.

[0026] This is understandable. Minimizing the difference between the first and second reconstructed images can reduce block artifacts and some artifacts caused by encoding / decoding inconsistencies.

[0027] In one possible design, the above method may further include: obtaining the prediction mode of the first encoding. When the prediction mode of the first encoding is inter-frame prediction, the steps of obtaining the encoding information of the first encoding and performing a second encoding of the image to be encoded or the first reconstructed image in a full intra-frame prediction mode based on the encoding information of the first encoding are performed to generate a second bitstream. When the prediction mode of the first encoding is intra-frame prediction, the first bitstream is used as the second bitstream.

[0028] By determining whether the prediction mode of the first encoding is intra-frame prediction, if the prediction mode of the first encoding is intra-frame prediction, the first bitstream can be directly used as the second bitstream, which can improve the efficiency of encoding to generate the first bitstream and the second bitstream.

[0029] In one possible design, the image to be encoded is the source video image.

[0030] By encoding the first and second bitstreams at the frame level, the quality of the first and second bitstreams can be kept consistent at the frame level. This helps to reduce block artifacts and some artifacts caused by inconsistencies in encoding and decoding, thereby improving the decoding quality of the accessed video content.

[0031] In one possible design, the image to be encoded is an image block obtained by dividing the source video image.

[0032] By encoding the first and second bitstreams at the image block level, the two bitstreams can be output synchronously at the image block level, so as to obtain the random access frames of the second bitstream used for accessing video content faster and reduce access latency.

[0033] In one possible design, the first encoding parameter may include a first quantization parameter or a first code rate. The second encoding parameter may include a second quantization parameter or a second code rate.

[0034] Secondly, embodiments of this application provide a video image processing method, which may include: acquiring at least one first image to be encoded and a second image to be encoded, wherein the second image to be encoded is a video image preceding the at least one first image to be encoded; performing a first encoding on the at least one first image to be encoded to generate a first bitstream; determining at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the second image to be encoded based on the at least one first reconstructed image, wherein the at least one first reconstructed image is either the first bitstream or a reconstructed image in the first encoding process; and performing a second encoding in full-frame intra-prediction mode on the second image to be encoded based on at least one of the first partitioning method or the first encoding parameter to generate a second bitstream.

[0035] The first stream can also be called the long GOP stream, and the second stream can also be called the random access stream. When video content needs to be accessed, the second stream is decoded first. The decoding result of the second stream is used as a reference frame to decode the first stream, so as to achieve fast access to video content.

[0036] The encoding of the second bitstream is adjusted based on the encoding result of the first bitstream to achieve the same or equivalent quality between the reconstructed image of the first bitstream and the corresponding reconstructed image of the second bitstream. This improves the decoding quality of the accessed video content, reduces block artifacts, and eliminates some artifacts while meeting the requirements for low-latency access to video content.

[0037] Among these methods, simulating the behavior of the decoding end during the encoding process helps to reduce block artifacts and artifacts caused by inconsistencies between encoding and decoding.

[0038] Optionally, the second image to be encoded can be a video image that precedes at least one frame or is separated by one or more frames from the first image to be encoded.

[0039] In one possible design, there is at least one first image to be encoded and at least one first reconstructed image. The difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold. The second reconstructed image is obtained by decoding the first bitstream using the third reconstructed image as a reference image. The third reconstructed image is the second bitstream or the reconstructed image in the second encoding process.

[0040] In one possible design, there are multiple first images to be encoded, multiple first reconstructed images, and the difference between multiple first reconstructed images and multiple second reconstructed images is less than a difference threshold, or the similarity between multiple first reconstructed images and multiple second reconstructed images is higher than a similarity threshold. The multiple second reconstructed images are obtained by decoding the first bitstream using a third reconstructed image as a reference image. The third reconstructed image is a reconstructed image in the second bitstream or the second encoding process.

[0041] The differences between multiple first reconstructed images and multiple second reconstructed images can be the weighted sum of the differences between each of the multiple first reconstructed images and its corresponding second reconstructed image. The similarity between multiple first reconstructed images and multiple second reconstructed images can be the weighted sum of the similarities between each of the multiple first reconstructed images and its corresponding second reconstructed image.

[0042] The second reconstructed image corresponding to one of the multiple first reconstructed images refers to the reconstructed image with the same video content.

[0043] In one possible design, there is at least one first image to be encoded and at least one first reconstructed image. Based on the at least one first reconstructed image, determining at least one of the first partitioning method or first encoding parameter for second encoding of the second image to be encoded includes: selecting a second partitioning method as the first partitioning method from a plurality of second partitioning methods based on the first reconstructed image; and / or selecting a second encoding parameter as the first encoding parameter from a plurality of second encoding parameters based on the first reconstructed image.

[0044] Among them, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the multiple fourth reconstructed images. The multiple fourth reconstructed images include the second reconstructed image. The multiple fourth reconstructed images are obtained by decoding the first bitstream using multiple fifth reconstructed images as reference images. The multiple fifth reconstructed images are reconstructed images of multiple third bitstreams. The multiple third bitstreams are obtained by performing multiple second encodings on the second image to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters. Alternatively, the multiple fifth reconstructed images are reconstructed images during the process of performing multiple second encodings on the second image to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters.

[0045] In one possible design, at least one first image to be encoded is multiple first images to be encoded. Based on at least one first reconstructed image, determining at least one of a first partitioning method or a first encoding parameter for second encoding of the second image to be encoded includes: selecting a second partitioning method as the first partitioning method from multiple second partitioning methods based on multiple first reconstructed images; and / or selecting a second encoding parameter as the first encoding parameter from multiple second encoding parameters.

[0046] For example, the second image to be encoded can be subjected to multiple second encodings based on multiple second partitioning methods and / or multiple second encoding parameters to generate multiple third bitstreams. Multiple fifth reconstructed images are either multiple third bitstreams or reconstructed images from multiple second encoding processes. Multiple fifth reconstructed images can be used as reference images to decode the first bitstream to obtain multiple sets of fourth reconstructed images. Each set of fourth reconstructed images can include the fourth reconstructed images corresponding to each of the multiple first images to be encoded. By comparing the similarity between each set of fourth reconstructed images and the multiple first reconstructed images, the set with the highest similarity is selected as the second reconstructed image corresponding to each of the multiple first images to be encoded. The third bitstream corresponding to the set of fourth reconstructed images with the highest similarity is used as the second bitstream. A third bitstream corresponding to a set of fourth reconstructed images means that the set of fourth reconstructed images can be obtained by using the reconstructed image of this third bitstream or the reconstructed image from the encoding process used to generate this third bitstream as a reference image to decode the first bitstream.

[0047] In one possible design, before performing the first encoding on at least one first image to be encoded, the method further includes: performing the first encoding on a second image to be encoded to generate a fourth bitstream; obtaining the prediction mode of the first encoding; if the prediction mode of the first encoding is inter-frame prediction, then performing the step of determining at least one of a first partitioning method or a first encoding parameter for performing the second encoding on the second image to be encoded based on at least one first reconstructed image; if the prediction mode of the first encoding is intra-frame prediction, using the fourth bitstream as the second bitstream.

[0048] In one possible design, at least one first image to be encoded is at least one first source video image, and the second image to be encoded is a second source video image.

[0049] In one possible design, the first encoding parameter includes a first quantization parameter or a first code rate. The second encoding parameter includes a second quantization parameter or a second code rate.

[0050] Based on the first aspect or any possible design of the first aspect, or the second aspect or any possible design of the second aspect, identification information of the bitstream characteristics may be carried during the encoding process. This identification information is used by the decoding end to distinguish between the first bitstream and the second bitstream. The identification information of the first bitstream and the second bitstream may be carried in any one of the following: parameter set, auxiliary enhancement information, encapsulation layer, file format, file description information, or custom message.

[0051] In one possible design, the first and second bitstreams use the same parameter set, which may carry first identification information. This first identification information indicates whether the current bitstream is the first or the second bitstream, or whether the first bitstream with the same video content precedes the second bitstream, or whether the first bitstream with the same video content follows the second bitstream. For example, this first identification information could be a stream identifier (stream_id) from a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS).

[0052] In this way, the first and second bitstreams can be encapsulated together.

[0053] In one possible design, the first and second bitstreams use different parameter sets, which may carry second identification information. This second identification information is used to indicate whether the current bitstream is the first or the second bitstream. For example, this second identification information could be a stream identifier (stream_id) from the VPS, SPS, or PPS of different bitstreams.

[0054] In this way, the first and second bitstreams can be encapsulated together or encapsulated independently.

[0055] In one possible design, the fragment header information of the first bitstream can carry third identification information. This third identification information is used to indicate that the current bitstream is the first bitstream. The fragment header information of the first bitstream can carry fourth identification information. This fourth identification information is used to indicate that the current bitstream is the second bitstream. For example, the third or fourth identification information can be the stream identifier (stream_id) in the fragment header information.

[0056] In this way, the first and second bitstreams can be encapsulated together or encapsulated independently.

[0057] In one possible design, the auxiliary enhancement information of the first bitstream can carry fifth identification information. This fifth identification information is used to indicate that the current bitstream is the first bitstream. The auxiliary enhancement information of the second bitstream can carry sixth identification information. This sixth identification information is used to indicate that the current bitstream is the second bitstream. For example, the fifth or sixth identification information can be the stream identifier (stream_id) in SEI.

[0058] In this way, the first and second bitstreams can be encapsulated together or encapsulated independently.

[0059] In one possible design, the first and second bitstreams are encapsulated independently. The encapsulation layer of the first bitstream carries a seventh identification information, which indicates that the current bitstream is the first bitstream. The encapsulation layer of the second bitstream carries an eighth identification information, which indicates that the current bitstream is the second bitstream. For example, the first bitstream is encapsulated in a first media stream sequence (track), and the second bitstream is encapsulated in a second media sequence (track), where the seventh or eighth identification information can be a media stream sequence class (track_class).

[0060] In one possible design, the first and second bitstreams are encapsulated and transmitted independently. The file format of the first bitstream carries a ninth identifier, which indicates that the current bitstream is the first bitstream. The file format of the second bitstream carries a tenth identifier, which indicates that the current bitstream is the second bitstream.

[0061] In one possible design, the first and second bitstreams are encapsulated and transmitted independently. The file description information of the first bitstream carries an eleventh identifier, which indicates that the current bitstream is the first bitstream. The file description information of the second bitstream carries a twelfth identifier, which indicates that the current bitstream is the second bitstream.

[0062] In one possible design, the first and second bitstreams are transmitted in a custom message mode. The custom message containing the first bitstream carries a thirteenth identification information, which indicates that the current bitstream is the first bitstream. The custom message containing the second bitstream carries a fourteenth identification information, which indicates that the current bitstream is the second bitstream. For example, the custom message is a TLV message. The thirteenth or fourteenth identification information can be type information in the TLV.

[0063] Thirdly, this application provides a video image processing apparatus. This apparatus can be an electronic device or a server, such as a chip or system-on-a-chip in the electronic device or server, or a functional module in the electronic device or server for implementing the first aspect or any possible implementation thereof. For example, the video image processing apparatus includes: an acquisition module for acquiring an image to be encoded; a first encoding module for performing a first encoding on the image to be encoded to generate a first bitstream; and a second encoding module for performing a second encoding of the image to be encoded or a first reconstructed image using intra-frame prediction mode based on the encoding information from the first encoding to generate a second bitstream, wherein the first reconstructed image is the first bitstream or a reconstructed image from the first encoding process.

[0064] In one possible design, the encoding information of the first code includes one or more of the following: the division method of the first code, the quantization parameters of the first code, and the encoding distortion information of the first code.

[0065] In one possible design, the second encoding module is used to perform at least one of the following: performing a second encoding of the image to be encoded or the first reconstructed image in a full intra-frame prediction mode using the same partitioning method as the first encoding; or performing a second encoding of the image to be encoded or the first reconstructed image in a full intra-frame prediction mode using the same quantization parameters as the first encoding; or determining the quantization parameters of the second encoding based on the quantization parameters and quantization parameter offset of the first encoding, and performing a second encoding of the image to be encoded or the first reconstructed image in a full intra-frame prediction mode based on the quantization parameters of the second encoding; or determining the quantization parameters of the second encoding based on the encoding distortion information of the first encoding, and performing a second encoding of the image to be encoded or the first reconstructed image in a full intra-frame prediction mode based on the quantization parameters of the second encoding.

[0066] In one possible design, the second encoding module is used to: determine the quantization parameters of the second encoding based on the encoding information of the first encoding and the feature information of the image to be encoded; and perform second encoding of the image to be encoded or the first reconstructed image in the intra-frame prediction mode based on the quantization parameters of the second encoding.

[0067] In one possible design, the feature information of the image to be encoded includes one or more of the following: the content complexity of the image to be encoded, the color classification information of the image to be encoded, the contrast information of the image to be encoded, and the content segmentation information of the image to be encoded.

[0068] In one possible design, the second encoding module is used to determine, based on the encoding information of the first encoding and the first reconstructed image, at least one of a first partitioning method or a first encoding parameter to be used for the second encoding of the image to be encoded or the first reconstructed image. The second encoding module is also used to perform a second encoding of the image to be encoded or the first reconstructed image using a full intra-frame prediction mode based on at least one of the first partitioning method or the first encoding parameter.

[0069] In the first bitstream, the interval between two adjacent intra-prediction modes is greater than the interval between two adjacent intra-prediction modes in the second bitstream.

[0070] In one possible design, the difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, and the second reconstructed image is a reconstructed image in the second bitstream or the second encoding process.

[0071] In one possible design, the second encoding module is used to: determine multiple second partitioning methods based on the encoding information of the first encoding and the first reconstructed image, and select one of the multiple second partitioning methods as the first partitioning method; and / or, determine multiple second encoding parameters based on the encoding information of the first encoding and the first reconstructed image, and select one of the multiple second encoding parameters as the first encoding parameter.

[0072] Among them, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and multiple third reconstructed images. The multiple third reconstructed images include the second reconstructed image. The multiple third reconstructed images are reconstructed images obtained by performing multiple second encoding processes on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters. Alternatively, the multiple third reconstructed images are reconstructed images of multiple third bitstreams. The multiple third bitstreams are obtained by performing multiple second encoding processes on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters.

[0073] In one possible design, the second encoding module is further configured to: obtain the prediction mode of the first encoding. When the prediction mode of the first encoding is inter-frame prediction, the steps of obtaining the encoding information of the first encoding and performing second encoding in the full intra-frame prediction mode on the image to be encoded or the first reconstructed image based on the encoding information of the first encoding are executed to generate the second bitstream. When the prediction mode of the first encoding is intra-frame prediction, the first bitstream is used as the second bitstream.

[0074] In one possible design, the image to be encoded is the source video image; or, the image to be encoded is a block of images obtained by dividing the source video image.

[0075] In one possible design, the first encoding parameter includes a first quantization parameter or a first code rate. The second encoding parameter includes a second quantization parameter or a second code rate.

[0076] Fourthly, this application provides a video image processing apparatus. This apparatus can be an electronic device or a server, such as a chip or system-on-a-chip in the electronic device or server, or a functional module in the electronic device or server for implementing the second aspect or any possible implementation thereof. For example, the video image processing apparatus includes: an acquisition module for acquiring at least one first image to be encoded and a second image to be encoded, wherein the second image to be encoded is a video image preceding the at least one first image to be encoded; a first encoding module for performing a first encoding on each of the at least one first image to be encoded to generate a first bitstream; and a second encoding module for determining, based on at least one first reconstructed image, at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the second image to be encoded, wherein the at least one first reconstructed image is the first bitstream or a reconstructed image during the first encoding process. The second encoding module is further configured to perform a second encoding on the second image to be encoded based on at least one of the first partitioning method or the first encoding parameter to generate a second bitstream.

[0077] In one possible design, there is at least one first image to be encoded and at least one first reconstructed image. The difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold. The second reconstructed image is obtained by decoding the first bitstream using the third reconstructed image as a reference image. The third reconstructed image is the second bitstream or the reconstructed image in the second encoding process.

[0078] In one possible design, the number of at least one first image to be encoded is one, the number of at least one first reconstructed image is one, and the second encoding module is used to: select one second partitioning method as the first partitioning method from multiple second partitioning methods based on the first reconstructed image; and / or, select one second encoding parameter as the first encoding parameter from multiple second encoding parameters based on the first reconstructed image.

[0079] Among them, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the multiple fourth reconstructed images. The multiple fourth reconstructed images include the second reconstructed image. The multiple fourth reconstructed images are obtained by decoding the first bitstream using multiple fifth reconstructed images as reference images. The multiple fifth reconstructed images are reconstructed images of multiple third bitstreams. The multiple third bitstreams are obtained by performing multiple second encodings on the second image to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters. Alternatively, the multiple fifth reconstructed images are reconstructed images during the process of performing multiple second encodings on the second image to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters.

[0080] In one possible design, the first encoding module is further configured to: perform a first encoding on the second image to be encoded before performing a first encoding on at least one first image to be encoded, to generate a fourth bitstream. The second encoding module is further configured to: obtain the prediction mode of the first encoding. When the prediction mode of the first encoding is inter-frame prediction, the step of determining at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the second image to be encoded, based on at least one first reconstructed image, is executed. When the prediction mode of the first encoding is intra-frame prediction, the fourth bitstream is used as the second bitstream.

[0081] In one possible design, at least one first image to be encoded is at least one first source video image, and the second image to be encoded is a second source video image.

[0082] In one possible design, the first encoding parameter includes a first quantization parameter or a first code rate. The second encoding parameter includes a second quantization parameter or a second code rate.

[0083] Fifthly, embodiments of this application provide a video image processing apparatus, comprising: one or more processors; and a memory for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect or any one of the first aspects, or perform the method as described in the second aspect or any one of the second aspects.

[0084] Sixthly, embodiments of this application provide a computer-readable storage medium including a first bitstream and a second bitstream obtained according to the method according to the first aspect or any one of the first aspects, or including a first bitstream and a second bitstream obtained according to the method according to the second aspect or any one of the second aspects.

[0085] In a seventh aspect, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method as described in the first aspect or any one of the first aspects, or the method as described in the second aspect or any one of the second aspects.

[0086] Eighthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect or any one of the first aspects, or the method as described in the second aspect or any one of the second aspects.

[0087] It should be understood that the third to eighth aspects of this application are consistent with the technical solutions of the first to second aspects of this application, and the beneficial effects obtained by each aspect and the corresponding feasible implementation are similar, so they will not be described again. Attached Figure Description

[0088] Figure 1A This is a block diagram for implementing an example of the video encoding and decoding system 10 of the embodiments of this application;

[0089] Figure 1B This is a block diagram for implementing an example of the video decoding system 40 according to the embodiments of this application;

[0090] Figure 2 This is a block diagram for implementing the encoder 20 instance structure of the embodiments of this application;

[0091] Figure 3 This is a block diagram for implementing the decoder 30 instance structure of the embodiments of this application;

[0092] Figure 4 This is a block diagram for implementing an example of the video decoding device 400 according to embodiments of this application;

[0093] Figure 5 This is a block diagram of another example of an encoding or decoding device for implementing the embodiments of this application;

[0094] Figure 6 This is a schematic diagram illustrating an application scenario of multi-camera shooting of sports events according to an embodiment of this application;

[0095] Figure 7 A schematic diagram of the decoding frame trajectory for switching from the current video content to another video content is provided in an embodiment of this application;

[0096] Figure 8 A schematic diagram of a video image processing system provided in an embodiment of this application;

[0097] Figure 9 This is a schematic diagram of a video image processing method provided in an embodiment of this application;

[0098] Figure 10 This is a schematic flowchart of a video image processing method provided in an embodiment of this application;

[0099] Figure 11 This is a schematic flowchart of a video image processing method provided in an embodiment of this application;

[0100] Figure 12 This is a schematic flowchart of a video image processing method provided in an embodiment of this application;

[0101] Figure 13 This is a schematic flowchart of a video image processing method provided in an embodiment of this application;

[0102] Figure 14 This is a schematic flowchart of a video image processing method provided in an embodiment of this application;

[0103] Figure 15 This is a schematic flowchart of a video image processing method provided in an embodiment of this application;

[0104] Figure 16 A schematic diagram illustrating that the first and second codes provided in the embodiments of this application use the same TU partitioning method;

[0105] Figure 17 A schematic diagram of the quantization parameters of the first encoding and the second encoding provided for embodiments of this application;

[0106] Figure 18 A schematic diagram illustrating the quantization parameters transmitted from the first code to the second code according to an embodiment of this application;

[0107] Figure 19 A schematic diagram illustrating the arrangement of the first and second bitstreams when the stream identifier (stream_id) is 2, as provided in this embodiment of the application.

[0108] Figure 20 A schematic diagram illustrating the arrangement of the first and second bitstreams when the stream identifier (stream_id) is 3, as provided in this embodiment of the application.

[0109] Figure 21 This application provides three methods for combining the first and second bitstreams into a single bitstream.

[0110] Figure 22 This is a schematic diagram of the structure of a video image processing device provided in an embodiment of this application. Detailed Implementation

[0111] The embodiments of this application are described below with reference to the accompanying drawings. In the following description, reference is made to the accompanying drawings, which form part of this disclosure and illustrate specific aspects of the embodiments of this application or to which specific aspects of the embodiments of this application may be used. It should be understood that the embodiments of this application may be used in other aspects and may include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of this application is defined by the appended claims. For example, it should be understood that the disclosure of the described methods is equally applicable to corresponding devices or systems for performing the methods, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units, each performing one or more of multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific apparatus is described based on one or more units such as functional units, the corresponding method may include a step to perform the functionality of one or more units (e.g., a step to perform the functionality of one or more units, or multiple steps, each of which performs the functionality of one or more units among a plurality of units), even if such one or more steps are not explicitly described or illustrated in the accompanying drawings. Furthermore, it should be understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0112] The technical solutions involved in the embodiments of this application may be applied not only to existing video coding standards (such as H.264, HEVC, etc.) but also to future video coding standards (such as H.266). The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. A brief introduction to some concepts that may be involved in the embodiments of this application is given below.

[0113] Video coding generally refers to the processing of a sequence of images that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously. As used herein, video coding refers to either video encoding or video decoding. Video coding is performed on the source side and typically involves processing (e.g., by compression) the raw video images to reduce the amount of data required to represent them, thereby enabling more efficient storage and / or transmission. Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video images. The term "encoding" of video images in the embodiments should be understood as referring to the "encoding" or "decoding" of a video sequence. The combination of encoding and decoding components is also referred to as codec (encoding and decoding, CODEC).

[0114] A video sequence consists of a series of pictures, which are further divided into slices, and slices into blocks. Video coding is performed on a block-by-block basis. In some newer video coding standards, the concept of a block has been further expanded. For example, the H.264 standard uses macroblocks (MBs), which can be further divided into multiple prediction blocks (partitions) for predictive coding. The High Efficiency Video Coding (HEVC) standard uses basic concepts such as coding units (CUs), prediction units (PUs), and transform units (TUs) to functionally divide various block units, and employs a novel tree-based structure for description. For instance, a CU can be divided into smaller CUs using a quadtree, and these smaller CUs can be further divided, forming a quadtree structure. The CU is the basic unit for partitioning and encoding the image. Similar tree structures exist for PUs and TUs. A PU corresponds to a prediction block and is the basic unit for predictive coding. CUs are further divided into multiple PUs according to partitioning patterns. TU can correspond to a transform block, which is the basic unit for transforming the prediction residual. However, whether it is CU, PU or TU, they all essentially belong to the concept of a block (or image block).

[0115] For example, in HEVC, the CTU is split into multiple CUs using a quadtree structure represented as a coding tree. At the CU level, a decision is made on whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode picture regions. Each CU can be further split into one, two, or four PUs based on the PU splitting type. The same prediction process is applied within a PU, and relevant information is transmitted to the decoder based on the PU. After obtaining residual blocks by applying the prediction process based on the PU splitting type, the CU can be segmented into transform units (TUs) according to other quadtree structures similar to the coding tree used for CUs. In the latest developments in video compression technology, quadtree and binary tree (QTBT) frame segmentation is used to divide coding blocks. In the QTBT block structure, CUs can be square or rectangular in shape.

[0116] In this paper, for ease of description and understanding, the image block to be encoded in the current encoded image is referred to as the current block. For example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded. The decoded image block in the reference image used to predict the current block is called the reference block. That is, the reference block is the block that provides a reference signal for the current block, where the reference signal represents the pixel value within the image block. The block in the reference image that provides the prediction signal for the current block can be called the prediction block, where the prediction signal represents the pixel value, sampled value, or sampled signal within the prediction block. For example, after traversing multiple reference blocks, an optimal reference block is found. This optimal reference block will provide prediction for the current block; this block is called the prediction block.

[0117] In lossless video coding, the original video images can be reconstructed, meaning the reconstructed video images have the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed, for example, through quantization, to reduce the amount of data required to represent the video images. However, the decoder cannot fully reconstruct the video images, meaning the quality of the reconstructed video images is lower or worse than the original.

[0118] Several H.261 video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding in the transform domain for applying quantization). Each frame of a video sequence is typically segmented into a non-overlapping set of blocks, usually encoded at the block level. In other words, the encoder typically processes, i.e., encodes the video at the block (video block) level, for example, by generating prediction blocks through spatial (intra-frame) and temporal (inter-frame) predictions, subtracting the prediction blocks from the current block (the currently processed or pending block) to obtain residual blocks, transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder replicates the decoder processing loop, causing the encoder and decoder to generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructions for processing, i.e., encoding subsequent blocks.

[0119] The system architecture used in the embodiments of this application is described below. See also Figure 1A , Figure 1A A schematic block diagram of the video encoding and decoding system 10 used in the embodiments of this application is provided as an example. Figure 1AAs shown, the video encoding and decoding system 10 may include a source device 12 and a destination device 14. The source device 12 generates encoded video data; therefore, the source device 12 may be referred to as a video encoding device. The destination device 14 can decode the encoded video data generated by the source device 12; therefore, the destination device 14 may be referred to as a video decoding device. Various embodiments of the source device 12, the destination device 14, or both may include one or more processors and memory coupled to said one or more processors. The memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory, or any other media that can be used to store desired program code in the form of computer-accessible instructions or data structures, as described herein. The source device 12 and the destination device 14 may include a variety of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, handsets such as so-called "smartphones," televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, wireless communication devices, or the like.

[0120] Although Figure 1A Source device 12 and destination device 14 are illustrated as separate devices, but device embodiments may also include the functionality of both source device 12 and destination device 14, or both; that is, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality. In such embodiments, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0121] A communication connection can be established between source device 12 and destination device 14 via link 13, through which destination device 14 can receive encoded video data from source device 12. Link 13 may include one or more media or devices capable of moving encoded video data from source device 12 to destination device 14. In one example, link 13 may include one or more communication media enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. In this example, source device 12 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video data to destination device 14. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, wide area network, or global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices facilitating communication from source device 12 to destination device 14.

[0122] The source device 12 includes an encoder 20. Optionally, the source device 12 may also include an image source 16, an image preprocessor 18, and a communication interface 22. In specific implementations, the encoder 20, image source 16, image preprocessor 18, and communication interface 22 may be hardware components or software programs within the source device 12. These are described below:

[0123] Image source 16 may include or be any type of image capture device for, for example, capturing real-world images, and / or any type of image or commentary (for screen content encoding, some text on the screen is also considered part of the image or picture to be encoded) generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source 16 may be a camera for capturing images or a memory for storing images. Image source 16 may also include any type of (internal or external) interface for storing previously captured or generated images and / or acquiring or receiving images. When image source 16 is a camera, image source 16 may be, for example, a local or integrated camera integrated into a source device; when image source 16 is a memory, image source 16 may be a local or, for example, integrated memory integrated into a source device. When the image source 16 includes an interface, the interface may be, for example, an external interface for receiving images from an external video source. The external video source may be, for example, an external image capture device, such as a camera, external storage, or an external image generation device. The external image generation device may be, for example, an external computer graphics processor, a computer, or a server. The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, or an optical interface.

[0124] An image can be viewed as a two-dimensional array or matrix of pixels. Pixels in the array are also called sampling points. The number of sampling points in the array or image along the horizontal and vertical directions (or axes) defines the image's size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as or contain three sampling arrays. For example, in RGB format or color space, an image includes corresponding red, green, and blue sampling arrays. However, in video coding, each pixel is typically represented in a luma / chroma format or color space. For example, for a YUV format image, this includes a luma component indicated by Y (sometimes also indicated by L) and two chroma components indicated by U and V. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components U and V represent chroma or color information components. Accordingly, a YUV format image includes a luma sampling array of luma sample values ​​(Y) and two chroma sampling arrays of chroma values ​​(U and V). An RGB format image can be converted or transformed to a YUV format, and vice versa; this process is also called color transformation or conversion. If the image is black and white, it may only include a luminance sampling array. In this embodiment, the image transmitted from image source 16 to image processor can also be referred to as raw image data 17.

[0125] Image preprocessor 18 is configured to receive raw image data 17 and perform preprocessing on the raw image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by image preprocessor 18 may include retouching, color format conversion (e.g., from RGB format to YUV format), color correction, or noise reduction.

[0126] Encoder 20 (or video encoder 20) is used to receive preprocessed image data 19 and process the preprocessed image data 19 using a relevant prediction mode (such as the prediction mode in the various embodiments of this document), thereby providing encoded image data 21 (which will be further based on...). Figure 2 or Figure 4 or Figure 5 (Description of structural details of encoder 20). In some embodiments, encoder 20 can be used to perform the various embodiments described below to implement the chroma block prediction method described in this application on the encoding side.

[0127] Communication interface 22 can be used to receive encoded image data 21 and transmit the encoded image data 21 to destination device 14 or any other device (such as a memory) via link 13 for storage or direct reconstruction. The other device can be any device used for decoding or storage. Communication interface 22 can, for example, be used to encapsulate the encoded image data 21 into a suitable format, such as data packets, for transmission over link 13.

[0128] Destination device 14 includes decoder 30. Optionally, destination device 14 may also include communication interface 28, image post-processor 32, and display device 34. These are described below:

[0129] Communication interface 28 can be used to receive encoded image data 21 from source device 12 or any other source, such as a storage device, for example, an encoded image data storage device. Communication interface 28 can be used to transmit or receive encoded image data 21 via link 13 between source device 12 and destination device 14 or via any type of network, such as a direct wired or wireless connection, any type of network such as wired or wireless networks or any combination thereof, or any type of private and public network, or any combination thereof. Communication interface 28 can be used, for example, to decapsulate data packets transmitted by communication interface 22 to obtain encoded image data 21.

[0130] Both communication interface 28 and communication interface 22 can be configured as unidirectional or bidirectional communication interfaces, and can be used, for example, to send and receive messages to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission.

[0131] Decoder 30 (or decoder 30) is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (hereinafter further based on...) Figure 3 or Figure 4 or Figure 5 (Description of structural details of decoder 30). In some embodiments, decoder 30 can be used to perform the various embodiments described below to implement the chroma block prediction method described in this application on the decoding side.

[0132] Image post-processor 32 is used to perform post-processing on decoded image data 31 (also known as reconstructed image data) to obtain post-processed image data 33. The post-processing performed by image post-processor 32 may include: color format conversion (e.g., from YUV format to RGB format), color correction, retouching or resampling, or any other processing, and may also be used to transmit the post-processed image data 33 to display device 34.

[0133] Display device 34 is used to receive post-processed image data 33 to display the image to, for example, a user or viewer. Display device 34 can be or may include any class of displays for presenting reconstructed images, such as integrated or external displays or monitors. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other class of display.

[0134] Although, Figure 1A Source device 12 and destination device 14 are illustrated as separate devices, but device embodiments may also include the functionality of both source device 12 and destination device 14, or both; that is, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality. In such embodiments, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0135] Based on the description, it is obvious to those skilled in the art that the functionality of different units is different. Figure 1A The presence and (precise) division of the functionality of the source device 12 and / or destination device 14 shown may vary depending on the actual device and application. The source device 12 and destination device 14 may include any of a variety of devices, encompassing any category of handheld or stationary devices, such as laptops or notebooks, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, cameras, in-vehicle devices, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not use any type of operating system.

[0136] Both encoder 20 and decoder 30 can be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is implemented in part in software, the device can store software instructions in a suitable non-transitory computer-readable storage medium, and one or more processors can be used to execute the instructions in hardware to perform the technology of this disclosure. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) can be considered as one or more processors.

[0137] In some cases, Figure 1A The video encoding and decoding system 10 shown is merely an example, and the technology of this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other instances, data may be retrieved from local storage, streamed over a network, etc. A video encoding device may encode data and store it in storage, and / or a video decoding device may retrieve data from storage and decode it. In some instances, encoding and decoding are performed by devices that do not communicate with each other but only encode data to storage and / or retrieve data from storage and decode the data.

[0138] See Figure 1B , Figure 1B It is a inclusion according to an exemplary embodiment Figure 2 encoder 20 and / or Figure 3 The illustrated diagram shows an example of a video decoding system 40 with decoder 30. The video decoding system 40 can implement various combinations of technologies according to embodiments of this application. In the illustrated embodiments, the video decoding system 40 may include an imaging device 41, an encoder 20, a decoder 30 (and / or a video codec implemented by logic circuitry 47 of processing unit 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0139] like Figure 1B As shown, the imaging device 41, antenna 42, processing unit 46, logic circuit 47, encoder 20, decoder 30, processor 43, memory 44, and / or display device 45 are capable of communicating with each other. As discussed, although the video decoding system 40 is illustrated with encoder 20 and decoder 30, in different instances, the video decoding system 40 may contain only encoder 20 or only decoder 30.

[0140] In some instances, antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, display device 45 can be used to present video data. In some instances, logic circuitry 47 can be implemented using processing unit 46. Processing unit 46 can include application-specific integrated circuit (ASIC) logic, graphics processor, general-purpose processor, etc. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general-purpose processor, etc. In some instances, logic circuitry 47 can be implemented in hardware, such as dedicated video encoding hardware, while processor 43 can be implemented in general-purpose software, operating system, etc. Furthermore, memory 44 can be any type of memory, such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory 44 can be implemented using cache memory. In some instances, logic circuitry 47 may access memory 44 (e.g., for implementing an image buffer). In other instances, logic circuitry 47 and / or processing unit 46 may include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0141] In some instances, the encoder 20 implemented via logic circuitry may include (e.g., implemented via processing unit 46 or memory 44) an image buffer and (e.g., implemented via processing unit 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the encoder 20 implemented via logic circuitry 47 to implement a reference. Figure 2 And / or any other encoder system or subsystem described herein, and the various modules discussed herein. Logic circuits may be used to perform the various operations discussed herein.

[0142] In some instances, decoder 30 can be implemented in a similar manner via logic circuitry 47 to implement reference. Figure 3The decoder 30 and / or any other decoder system or subsystem described herein are various modules discussed. In some instances, the logic circuit-implemented decoder 30 may include an image buffer (implemented via processing unit 2820 or memory 44) and a graphics processing unit (e.g., implemented via processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the decoder 30 implemented via logic circuit 47 to implement reference Figure 3 And / or the various modules discussed in any other decoder system or subsystem described herein.

[0143] In some instances, antenna 42 can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to encoded segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining encoded segmentation). Video decoding system 40 may also include decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to display the video frames.

[0144] It should be understood that, for the instances described with reference to encoder 20 in the embodiments of this application, decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0145] It should be noted that the encoder 20 and decoder 30 in the embodiments of this application can be encoders / decoders corresponding to video standard protocols such as H.263, H.264, HEVV, MPEG-2, MPEG-4, VP8, VP9, ​​or next-generation video standard protocols (such as H.266).

[0146] See Figure 2 , Figure 2 A schematic / conceptual block diagram illustrating an example of an encoder 20 used to implement embodiments of this application is shown. Figure 2In this example, encoder 20 includes a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a buffer 216, a loop filter unit 220, a decoded picture buffer (DPB) 230, a prediction processing unit 260, and an entropy coding unit 270. Prediction processing unit 260 may include inter-frame prediction unit 244, intra-frame prediction unit 254, and mode selection unit 262. Inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0147] For example, the residual calculation unit 204, transform processing unit 206, quantization unit 208, prediction processing unit 260, and entropy coding unit 270 form the forward signal path of the encoder 20, while, for example, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, and prediction processing unit 260 form the backward signal path of the encoder, wherein the backward signal path of the encoder corresponds to the signal path of the decoder (see...). Figure 3 Decoder 30 in the middle.

[0148] Encoder 20 receives, for example, image 201 or image block 203 of image 201 via input 202, such as images in a sequence of images forming a video or video sequence. Image block 203 may also be referred to as the current image block or the image block to be encoded, and image 201 may be referred to as the current image or the image to be encoded (especially in video encoding when distinguishing the current image from other images, such as previously encoded and / or decoded images in the same video sequence, i.e., including the current image).

[0149] Embodiments of encoder 20 may include a segmentation unit ( Figure 2 (Not shown in the image) is used to segment image 201 into multiple blocks, such as image blocks 203, typically into multiple non-overlapping blocks. The segmentation unit can be used to apply the same block size and corresponding raster defining the block size to all images in a video sequence, or to change the block size between images, subsets, or groups of images, and segment each image into corresponding blocks.

[0150] In one instance, the prediction processing unit 260 of encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0151] As with image 201, image patch 203 is also, or can be considered as, a two-dimensional array or matrix of sampled points with sampled values, although its size is smaller than that of image 201. In other words, image patch 203 may include, for example, a single sampled array (e.g., a luminance array in the case of black and white image 201) or three sampled arrays (e.g., a luminance array and two chrominance arrays in the case of a color image) or any other number and / or type of array depending on the applied color format. The number of sampled points in the horizontal and vertical directions (or axes) of image patch 203 defines the size of image patch 203.

[0152] like Figure 2 The encoder 20 shown is used to encode the image 201 block by block, for example, to perform encoding and prediction for each image block 203.

[0153] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block 203 and the prediction block 265 (other details of the prediction block 265 are provided below). For example, the residual block 205 is obtained in the sample domain by subtracting the sample value of the prediction block 265 from the sample value of the image block 203 on a sample-by-sample (pixel-by-pixel) basis.

[0154] The transform processing unit 206 applies a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 can also be called transform residual coefficients and represent the residual block 205 in the transform domain.

[0155] Transform processing unit 206 can be used to apply integer approximations of DCT / DST, such as those specified for HEVC / H.265. These integer approximations are typically scaled by a factor compared to orthogonal DCT transforms. To maintain the norm of the residual blocks processed by the forward and inverse transforms, an additional scaling factor is applied as part of the transform process. The scaling factor is typically chosen based on certain constraints, such as a power of 2 for the shift operation, the bit depth of the transform coefficients, accuracy, and implementation cost. For example, a specific scaling factor can be specified on the decoder 30 side via, for example, inverse transform processing unit 212 (and on the encoder 20 side via, for example, inverse transform processing unit 212 for the corresponding inverse transform), and correspondingly, a corresponding scaling factor can be specified on the encoder 20 side via transform processing unit 206 for the forward transform.

[0156] Quantization unit 208 is used to quantize the transform coefficients 207, for example, by applying scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be referred to as quantized residual coefficients 209. The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index of a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization can involve division by the quantization step size and a corresponding quantization or inverse quantization, performed, for example, by inverse quantization 210, or it can involve multiplication by the quantization step size. The quantization step size can be determined using quantization parameters according to some standard embodiments, such as HEVC. Generally, the quantization step size can be calculated based on the quantization parameters using a fixed-point approximation of an equation involving division. Additional scaling factors can be introduced for quantization and inverse quantization to recover the norm of the residual block, which may have been modified due to the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one example implementation, the scaling of the inverse transform and inverse quantization can be combined. Alternatively, a custom quantization table can be used and transmitted from the encoder to the decoder via a signal, for example, in a bitstream. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0157] The inverse quantization unit 210 is used to apply the inverse quantization of the quantization unit 208 to the quantized coefficients to obtain the dequantized coefficients 211. For example, it applies the inverse quantization scheme of the quantization scheme applied by the quantization unit 208 based on or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 can also be called the dequantized residual coefficients 211, corresponding to the transform coefficients 207, although the loss due to quantization is usually different from that of the transform coefficients.

[0158] The inverse transform processing unit 212 is used to apply the inverse transform of the transform applied by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the inverse transform block 213 in the sample domain. The inverse transform block 213 can also be referred to as the inverse transform dequantized block 213 or the inverse transform residual block 213.

[0159] Reconstruction unit 214 (e.g., summer 214) is used to add inverse transform block 213 (i.e. reconstructed residual block 213) to prediction block 265 to obtain reconstructed block 215 in the sample domain, for example, by adding the sample value of reconstructed residual block 213 to the sample value of prediction block 265.

[0160] Optionally, for example, buffer unit 216 (or simply "buffer" 216) of line buffer 216 is used to buffer or store reconstructed block 215 and corresponding sample values ​​for, for example, intra-frame prediction. In other embodiments, the encoder can be used to perform any type of estimation and / or prediction, such as intra-frame prediction, using the unfiltered reconstructed block and / or corresponding sample values ​​stored in buffer unit 216.

[0161] For example, embodiments of encoder 20 can be configured such that buffer unit 216 is used not only to store the reconstructed block 215 for intra-frame prediction 254, but also for loop filter unit 220 (in Figure 2 (not shown in the image), and / or, for example, such that buffer unit 216 and decoded image buffer unit 230 form a buffer. Other embodiments may be used to filter block 221 and / or blocks or samples from decoded image buffer 230 (not shown in the image), and / or, for example, such that buffer unit 216 and decoded image buffer unit 230 form a buffer. Figure 2 (Not shown in the image) is used as input or basis for intra-frame prediction 254.

[0162] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, thereby facilitating pixel transformation or improving video quality. Loop filter unit 220 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. Although loop filter unit 220 is used in... Figure 2 The image shown is an in-loop filter, but in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as the filtered reconstructed block 221. The decoded image buffer 230 can store the reconstructed encoded block after the loop filter unit 220 performs the filtering operation on the reconstructed encoded block.

[0163] An embodiment of encoder 20 (correspondingly, loop filter unit 220) can be used to output loop filter parameters (e.g., sample adaptive offset information), for example, directly output or entropy encoded by entropy coding unit 270 or any other entropy coding unit, such that decoder 30 can receive and apply the same loop filter parameters for decoding.

[0164] The decoded picture buffer (DPB) 230 can be a reference picture memory for storing reference picture data for the encoder 20 to encode video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM)) or other types of memory devices. The DPB 230 and buffer 216 can be provided by the same memory device or separate memory devices. In one example, the decoded picture buffer (DPB) 230 is used to store filtered block 221. The decoded picture buffer 230 can be further used to store other previously filtered blocks of the same current picture or different pictures, such as previously reconstructed pictures, for example, previously reconstructed and filtered blocks 221, and can provide complete previously reconstructed, i.e., decoded pictures (and corresponding reference blocks and samples) and / or partially reconstructed current pictures (and corresponding reference blocks and samples), for example, for inter-frame prediction. In one instance, if reconstructed block 215 is reconstructed without in-loop filtering, the decoded picture buffer (DPB) 230 is used to store reconstructed block 215.

[0165] The prediction processing unit 260, also known as the block prediction processing unit 260, is used to receive or acquire image block 203 (current image block 203 of current image 201) and reconstructed image data, such as reference samples of the same (current) image from buffer 216 and / or reference image data 231 of one or more previously decoded images from decoded image buffer 230, and to process such data to make predictions, i.e., to provide prediction block 265 which can be inter-frame prediction block 245 or intra-frame prediction block 255.

[0166] The mode selection unit 262 can be used to select a prediction mode (e.g., intra-frame or inter-frame prediction mode) and / or the corresponding prediction block 245 or 255 used as prediction block 265 to compute residual block 205 and reconstructed block 215.

[0167] Embodiments of the mode selection unit 262 can be used to select a prediction mode (e.g., from those prediction modes supported by the prediction processing unit 260), which provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both. The mode selection unit 262 can be used to determine the prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum RDO optimization, or selecting the prediction mode whose relevant rate distortion at least meets the prediction mode selection criteria.

[0168] The following will explain in detail the prediction processing performed by instances of encoder 20 (e.g., by prediction processing unit 260) and the mode selection performed (e.g., by mode selection unit 262).

[0169] As described above, encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0170] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in H.265, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in the developing H.266.

[0171] In possible implementations, the set of inter-frame prediction modes depends on the available reference image (i.e., at least a portion of the decoded image stored in the DBP 230 as described above) and other inter-frame prediction parameters, such as whether to use the entire reference image or only a portion of the reference image, such as a search window region surrounding the current block, to search for the best matching reference block, and / or whether to apply pixel interpolation such as half-pixel and / or quarter-pixel interpolation. The set of inter-frame prediction modes may include, for example, an Advanced Motion Vector Prediction (AMVP) mode and a merge mode. In specific implementations, the set of inter-frame prediction modes may include the control point-based AMVP mode improved in embodiments of this application, and an improved control point-based merge mode. In one instance, the intra-frame prediction unit 254 may be used to perform any combination of the inter-frame prediction techniques described below.

[0172] In addition to the prediction modes mentioned above, the embodiments of this application may also apply skip mode and / or direct mode.

[0173] The prediction processing unit 260 can be further used to segment the image block 203 into smaller block partitions or sub-blocks, for example, by iteratively using quad-tree (QT) segmentation, binary-tree (BT) segmentation, or triple-tree (TT) segmentation, or any combination thereof, and to perform prediction for each of the block partitions or sub-blocks, for example, wherein mode selection includes selecting the tree structure of the segmented image block 203 and selecting a prediction mode to be applied to each of the block partitions or sub-blocks.

[0174] Inter-frame prediction unit 244 may include motion estimation (ME) units. Figure 2 (not shown in the image) and motion compensation (MC) unit ( Figure 2 (Not shown in the image). The motion estimation unit is used to receive or acquire image block 203 (current image block 203 of current image 201) and decoded image 231, or at least one or more previously reconstructed blocks, such as reconstructed blocks of one or more other / different previously decoded images 231, to perform motion estimation. For example, a video sequence may include the current image and the previously decoded image 31, or in other words, the current image and the previously decoded image 31 may be part of an image sequence that forms a video sequence, or form that image sequence.

[0175] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and send it to the motion estimation unit ( Figure 2(Not shown in the image) Provides a reference image and / or provides the offset (spatial offset) between the position (X, Y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter. This offset is also called a motion vector (MV).

[0176] The motion compensation unit is used to acquire inter-frame prediction parameters and perform inter-frame prediction based on or using these parameters to obtain inter-frame prediction block 245. (The motion compensation unit...) Figure 2 Motion compensation performed (not shown) can involve extracting or generating prediction blocks based on motion / block vectors determined through motion estimation (possibly performing interpolation with subpixel accuracy). Interpolation filtering can generate additional pixel samples from known pixel samples, potentially increasing the number of candidate prediction blocks available for encoding picture blocks. Once the motion vector of the PU for the current picture block is received, the motion compensation unit 246 can locate the prediction block pointed to by the motion vector in a list of reference pictures. The motion compensation unit 246 can also generate syntax elements associated with blocks and video stripes for the decoder 30 to use when decoding picture blocks of the video stripe.

[0177] Specifically, the inter-frame prediction unit 244 can transmit syntax elements to the entropy coding unit 270. These syntax elements include inter-frame prediction parameters (e.g., indication information for selecting the inter-frame prediction mode for the current block prediction after traversing multiple inter-frame prediction modes). In some application scenarios, if there is only one inter-frame prediction mode, the inter-frame prediction parameters may not be carried in the syntax elements; in this case, the decoding end 30 can directly use the default prediction mode for decoding. It is understood that the inter-frame prediction unit 244 can be used to perform any combination of inter-frame prediction techniques.

[0178] Intra-prediction unit 254 is used to acquire, for example, image block 203 (current image block) of the same image and one or more previously reconstructed blocks, such as reconstructed adjacent blocks, for intra-prediction. For example, encoder 20 can be used to select an intra-prediction mode from a plurality of (predetermined) intra-prediction modes.

[0179] An embodiment of encoder 20 can be used to select an intra-prediction mode based on optimization criteria, such as based on minimum residual (e.g., providing an intra-prediction mode that most closely resembles the current image block 203) or minimum bitrate distortion.

[0180] Intra-prediction unit 254 is further configured to determine intra-prediction block 255 based on intra-prediction parameters as selected for the intra-prediction mode. In any case, after selecting the intra-prediction mode for the block, intra-prediction unit 254 is also configured to provide intra-prediction parameters to entropy coding unit 270, i.e., to provide information indicating the selected intra-prediction mode for the block. In one instance, intra-prediction unit 254 can be used to perform any combination of intra-prediction techniques.

[0181] Specifically, the intra-prediction unit 254 can transmit syntax elements to the entropy coding unit 270. These syntax elements include intra-prediction parameters (e.g., indication information for selecting the intra-prediction mode for the current block prediction after traversing multiple intra-prediction modes). In some application scenarios, if there is only one intra-prediction mode, the intra-prediction parameters may not be carried in the syntax elements; in this case, the decoder 30 can directly use the default prediction mode for decoding.

[0182] Entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., variable length coding (VLC), context adaptive VLC (CAVLC), arithmetic coding, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to one or all (or not to apply) of the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, and / or loop filter parameters to obtain encoded picture data 21 that can be output as, for example, an encoded bitstream 21, through output 272. The encoded bitstream can be transmitted to video decoder 30, or archived for later transmission or retrieval by video decoder 30. Entropy coding unit 270 can also be used to entropy code other syntax elements of the current video strip being encoded.

[0183] Other structural variations of the video encoder 20 can be used to encode video streams. For example, a non-transform-based encoder 20 can directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another embodiment, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0184] Specifically, in the embodiments of this application, the encoder 20 can be used to implement the video image processing method described in the embodiments below.

[0185] It should be understood that other structural variations of the video encoder 20 can be used to encode the video stream. For example, for certain image blocks or frames, the video encoder 20 can directly quantize the residual signal without processing by the transform processing unit 206, and correspondingly, without processing by the inverse transform processing unit 212; or, for certain image blocks or frames, the video encoder 20 does not generate residual data, and correspondingly, without processing by the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210, and the inverse transform processing unit 212; or, the video encoder 20 can directly store the reconstructed image block as a reference block without processing by the filter 220; or, the quantization unit 208 and the inverse quantization unit 210 in the video encoder 20 can be combined. The loop filter 220 is optional, and for lossless compression coding, the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210, and the inverse transform processing unit 212 are optional. It should be understood that, depending on the application scenario, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 may be selectively enabled.

[0186] See Figure 3 , Figure 3 A schematic / conceptual block diagram illustrating an example of a decoder 30 for implementing embodiments of this application is shown. The video decoder 30 is used to receive encoded picture data (e.g., encoded bitstream) 21, for example, encoded by the encoder 20, to obtain a decoded picture 231. During the decoding process, the video decoder 30 receives video data from the video encoder 20, such as encoded video bitstreams representing picture blocks of encoded video stripes and associated syntax elements.

[0187] exist Figure 3 In one example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a buffer 316, a loop filter 320, a decoded image buffer 330, and a prediction processing unit 360. Prediction processing unit 360 may include an inter-frame prediction unit 344, an intra-frame prediction unit 354, and a mode selection unit 362. In some instances, video decoder 30 can perform substantially the same functions as the referenced unit. Figure 2 The video encoder 20 describes the encoding passes as the inverse of the decoding passes.

[0188] Entropy decoding unit 304 is used to perform entropy decoding on encoded image data 21 to obtain, for example, quantization coefficients 309 and / or decoded encoded parameters. Figure 3(Not shown in the image), for example, any one or all of the (decoded) inter-frame prediction, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 is further configured to forward the inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the prediction processing unit 360. The video decoder 30 may receive syntax elements at the video strip level and / or video block level.

[0189] The inverse quantization unit 310 can function the same as the inverse quantization unit 110, the inverse transform processing unit 312 can function the same as the inverse transform processing unit 212, the reconstruction unit 314 can function the same as the reconstruction unit 214, the buffer 316 can function the same as the buffer 216, the loop filter 320 can function the same as the loop filter 220, and the decoded image buffer 330 can function the same as the decoded image buffer 230.

[0190] The prediction processing unit 360 may include an inter-frame prediction unit 344 and an intra-frame prediction unit 354, wherein the inter-frame prediction unit 344 may be functionally similar to the inter-frame prediction unit 244, and the intra-frame prediction unit 354 may be functionally similar to the intra-frame prediction unit 254. The prediction processing unit 360 is typically used to perform block prediction and / or obtain prediction blocks 365 from encoded data 21, and to receive or obtain prediction-related parameters and / or information about the selected prediction mode from, for example, an entropy decoding unit 304 (explicitly or implicitly).

[0191] When a video strip is encoded as an intra-coded (I) strip, the intra-prediction unit 354 of the prediction processing unit 360 generates a prediction block 365 for a picture block of the current video strip based on the intra-prediction mode of the signal representation and data from previously decoded blocks of the current frame or picture. When a video frame is encoded as an inter-coded (i.e., B or P) strip, the inter-prediction unit 344 (e.g., a motion compensation unit) of the prediction processing unit 360 generates a prediction block 365 for a video block of the current video strip based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-frame prediction, a prediction block can be generated from a reference picture within a list of reference pictures. The video decoder 30 can construct a list of reference frames, list 0 and list 1, based on reference pictures stored in the DPB 330 using a default construction technique.

[0192] The prediction processing unit 360 is used to determine prediction information for video blocks of the current video slice by parsing motion vectors and other syntax elements, and to generate prediction blocks for the current video slice for proper decoding using the prediction information. In one embodiment of this application, the prediction processing unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction), inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice) for the video blocks used to encode the video slice, the construction information of one or more of the reference picture lists for the slice, the motion vector for each inter-frame coded video block of the slice, the inter-frame prediction state of each inter-frame coded video block of the slice, and other information to decode the video blocks of the current video slice. In another embodiment of this disclosure, the syntax elements received by the video decoder 30 from the bitstream include syntax elements from one or more of the adaptive parameter set (APS), sequence parameter set (SPS), picture parameter set (PPS), or slice header.

[0193] The inverse quantization unit 310 can be used to inverse quantize (i.e., dequantize) the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 304. The inverse quantization process may involve using quantization parameters calculated by the video encoder 20 for each video block in the video strip to determine the degree of quantization to be applied and similarly determine the degree of inverse quantization to be applied.

[0194] The inverse transform processing unit 312 is used to apply an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to generate a residual block in the pixel domain.

[0195] Reconstruction unit 314 (e.g., summer 314) is used to add inverse transform block 313 (i.e. reconstructed residual block 313) to prediction block 365 to obtain reconstructed block 315 in the sample domain, for example by adding the sample values ​​of reconstructed residual block 313 to the sample values ​​of prediction block 365.

[0196] Loop filter unit 320 (during or after the encoding loop) is used to filter the reconstructed block 315 to obtain the filtered block 321, thereby facilitating pixel transformation or improving video quality. In one instance, loop filter unit 320 can be used to perform any combination of the filtering techniques described below. Loop filter unit 320 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. Although loop filter unit 320 is used in... Figure 3 The loop filter unit 320 is shown as an in-loop filter, but in other configurations, it can be implemented as a post-loop filter.

[0197] The decoded video block 321 in the given frame or image is then stored in the decoded image buffer 330, which stores a reference image for subsequent motion compensation.

[0198] Decoder 30 is used, for example, to output decoded image 31 via output 332 for presentation to or viewing by the user.

[0199] Other variations of the video decoder 30 can be used to decode compressed bitstreams. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another embodiment, the video decoder 30 may have an inverse quantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0200] Specifically, in this embodiment, the decoder 30 is used to implement the video image processing method described in the following embodiments.

[0201] It should be understood that other structural variations of the video decoder 30 can be used to decode the encoded video bitstream. For example, the video decoder 30 may generate an output video stream without processing by filter 320; or, for certain image blocks or frames, the entropy decoding unit 304 of the video decoder 30 may not decode the quantized coefficients, and accordingly, processing by the inverse quantization unit 310 and the inverse transform processing unit 312 is unnecessary. The loop filter 320 is optional; and for lossless compression, the inverse quantization unit 310 and the inverse transform processing unit 312 are optional. It should be understood that, depending on the application scenario, the inter-frame prediction unit and the intra-frame prediction unit may be selectively enabled.

[0202] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a video decoding device 400 (e.g., a video encoding device 400 or a video decoding device 400) provided in an embodiment of this application. The video decoding device 400 is suitable for implementing the embodiments described herein. In one embodiment, the video decoding device 400 may be a video decoder (e.g., ... Figure 1A decoder 30) or video encoder (e.g. Figure 1A The encoder 20). In another embodiment, the video decoding device 400 may be the one described above. Figure 1A decoder 30 or Figure 1A One or more components in encoder 20.

[0203] The video decoding device 400 includes: an input port 410 and a receiving unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include photoelectric conversion components and electro-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450 for the input or output of optical or electrical signals.

[0204] Processor 430 is implemented in both hardware and software. Processor 430 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes a decoding module 470 (e.g., encoding module 470 or decoding module 470). Encoding / decoding module 470 implements the embodiments disclosed herein to implement the chroma block prediction method provided in the embodiments of this application. For example, encoding / decoding module 470 implements, processes, or provides various encoding operations. Therefore, encoding / decoding module 470 provides a substantial improvement to the functionality of video decoding device 400 and affects the transitions of video decoding device 400 to different states. Alternatively, encoding / decoding module 470 can be implemented with instructions stored in memory 460 and executed by processor 430.

[0205] Memory 460 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs while they are selectively executed, and to store instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random access memory (SRAM).

[0206] See Figure 5 , Figure 5 It is available as an exemplary embodiment. Figure 1A A simplified block diagram of device 500 comprising one or both of the source device 12 and the destination device 14. Device 500 can implement the technology of this application. In other words, Figure 5 This is a schematic block diagram illustrating one implementation of an encoding or decoding device (referred to as decoding device 500) according to an embodiment of this application. The decoding device 500 may include a processor 510, a memory 530, and a bus system 550. The processor and memory are connected via the bus system. The memory stores instructions, and the processor executes the instructions stored in the memory. The decoding device's memory stores program code, and the processor can call the program code stored in the memory to execute various video encoding or decoding methods described in this application, particularly various novel random access stream quality control methods. To avoid repetition, detailed descriptions are omitted here.

[0207] In this embodiment, the processor 510 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0208] The memory 530 may include a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may also be used as memory 530. Memory 530 may include code and data 531 accessed by processor 510 using bus 550. Memory 530 may further include an operating system 533 and an application program 535, which includes at least one program that allows processor 510 to execute the video encoding or decoding methods described in this application (especially the video image processing methods described in this application). For example, application program 535 may include applications 1 to N, which further include a video encoding or decoding application (hereinafter referred to as a video decoding application) that executes the video encoding or decoding methods described in this application.

[0209] In addition to the data bus, the bus system 550 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 550 in the diagram.

[0210] Optionally, the decoding device 500 may also include one or more output devices, such as a display 570. In one example, the display 570 may be a haptic display that combines a display with a haptic unit capable of operatively sensing touch input. The display 570 may be connected to the processor 510 via a bus 550.

[0211] The technical solutions of the embodiments of this application are described in detail below:

[0212] First, some technical terms or concepts applicable to the embodiments of this application will be introduced.

[0213] I-frame: In the field of video coding, a frame that can be decoded independently without relying on other frames is generally referred to as an "I-frame". The coding prediction method for all blocks of an I-frame is intra-frame prediction.

[0214] P-frame: In the field of video coding, a frame that is referenced from the preceding frame and marked as a P-frame type in the bitstream is generally referred to simply as a "P-frame".

[0215] B-frame: In the field of video coding, it can refer to both the preceding and following frames, and is generally referred to as "B-frame". The embodiments of this application are mainly aimed at low-latency scenarios, so in order to reduce latency, it is generally encoded as "P-frame".

[0216] Random access stream: For ease of description, in this embodiment, a video stream in which all frames are I-frames, or where all frames are in intra-prediction mode, is simply referred to as a "random access stream". It should be noted that a random access stream can also be called a second stream, etc., and the name is not limited thereto. The encoding process used to generate the random access stream is called second encoding.

[0217] Long Group of Pictures (GOP) Stream: For ease of description, this application embodiment refers to a video stream in which at least one P-frame is present between adjacent I-frames, or a video stream that allows inter-frame prediction modes, as a "long GOP stream." It should be noted that a long GOP stream can also be called a first bitstream or a basic stream, etc., and its name is not limited thereto. The encoding process used to generate the GOP stream is called first encoding.

[0218] Reconstructed Frame: During the encoding process, the encoding result of the previous frame needs to be decoded and saved for reference by subsequent frames. The decoded frame is generally called a "reconstructed frame." This reconstructed frame is consistent with the frame decoded by the decoder, which is called encoding-decoding consistency. For ease of description, the frames decoded by the decoder in this application embodiment are also collectively referred to as "reconstructed frames."

[0219] Highly interactive scenarios: Application scenarios that can receive user input and provide feedback in real time. Examples include gaming and live streaming interactive scenarios.

[0220] Camera position: A camera at a location belongs to a camera position. When the playback changes from the content of one camera position to the content of another camera position, a camera position switch occurs.

[0221] Viewpoint: The direction the camera is facing or the direction the user is looking at at any given moment. The viewpoint changes when the camera rotates or the user turns their head.

[0222] Multi-camera shooting: Using two or more cameras to simultaneously shoot the same scene from multiple angles and directions.

[0223] To reduce the latency of accessing (including decoding or playing) video content and meet the low-latency requirements of some application scenarios, the encoding end can provide two bitstreams for the same video content: one is a long GOP bitstream, and the other is a random access bitstream. All frames in the random access bitstream are encoded in intra-predictive mode. When video content needs to be accessed, the frame at the access time in the random access stream of the video content is first decoded. Then, the decoded result of this frame is used as the reference frame for the long GOP bitstream of the video content, thereby enabling rapid access to the video content.

[0224] Accessing video content includes initially accessing video content or switching from one video content to another. For example, initially accessing video content could be based on a detected user click to start playing a live stream, initiating the decoding of the video content. Switching from one video content to another could be as follows: Figure 6 The example shown is switching from decoding one camera's video to another.

[0225] By providing two bitstreams for the same video content at the encoding end, latency for accessing video content can be reduced. Figure 6 The following example illustrates the application scenario of multi-camera shooting of sports events. Figure 6 As shown, multiple cameras are deployed at different locations at the sports event venue, for example, such as... Figure 6 The seven cameras shown are designated Camera A, Camera B, Camera C, Camera D, Camera E, Camera F, and Camera G. Cameras in different positions can capture the same scene from different angles, resulting in a set of video signals. This set of video signals can include video from camera A, camera B, camera C, camera D, camera E, camera F, and camera G. Each of these camera positions can function as a separate video content. Multiple camera positions (A, B, C, D, E, F, and G) can provide users with a multi-angle, three-dimensional visual experience. The encoding end can provide each camera position (A, B, C, D, E, F, or G) video with the two bitstreams described above. Users can switch between camera positions from one angle to another using appropriate interactive methods. The decoding end switches from decoding the video stream from one camera position to decoding the video stream from another camera position based on user operation. Figure 7 This is a schematic diagram of the decoding frame trajectory when switching from the current video content to another video content, provided as an embodiment of this application. For example, the current video content is from camera A, and the other video content is from camera B. Figure 7 As shown, the decoding end decodes frames numbered 0 (#0), 1 (#1), and 2 (#2) of the long GOP bitstream of the current video content. When decoding frame number 3 (#3) of the current video content, a video content switch occurs, i.e., the decoding switches from decoding the current video content to decoding another video content. The decoding end can acquire and decode the frames at the switching moment in the random access bitstream of the other video content. For example, as... Figure 7The image shows frame #3 of the random access stream of another video content. After decoding frame #3 of the random access stream of the other video content, the decoding result (i.e., the reconstructed frame) is placed in the reference frame list buffer. Then, decoding begins on frame #4 of the long GOP stream of the other video content. The decoding of frame #4 is referenced to the decoding result of frame #3 of the random access stream, thus completing the decoding and accessing the other video content. In this way, the decoding end does not need to start decoding from the previous I-frame (i.e., frame #0) in the long GOP stream of the other video content, nor does it need to wait for the next I-frame (i.e., frame #9) to start decoding.

[0226] In this approach, reconstructed frames from the random access stream of the video content are used as reference frames for the long GOP stream. While this allows for fast access, it introduces encoding / decoding inconsistencies. When the quality of the random access stream and the long GOP stream differs significantly, subjectively visible block artifacts and noticeable artifacts can easily appear. Analysis reveals that the primary cause is that during the encoding of the long GOP stream, the frames used for encoding prediction are reconstructed frames from that stream, while the reconstructed frames from the random access stream are used for accessing the long GOP stream. This results in a lack of quality matching between the reconstructed frames from the random access stream and the reconstructed frames from the long GOP stream during encoding.

[0227] To address the issue of quality mismatch in reconstructed frames from different bitstreams, this application proposes a video image processing method, also known as a random access quality control method, which aims to improve the decoding quality of accessed video content, reduce block artifacts, and eliminate some artifacts while ensuring low-latency access to video content.

[0228] Before describing the technical solutions of the embodiments of this application, the video image processing system of the embodiments of this application will first be described with reference to the accompanying drawings. See also Figure 8 , Figure 8This is a schematic diagram of a video image processing system provided in an embodiment of this application. The video image processing system may include a server 801 and a terminal 802. The server 801 can communicate with the terminal 802, for example, via wireless-fidelity (Wi-Fi), Bluetooth, or 2G / 3G / 4G / 5G cellular communication. It should be understood that other communication methods, including future communication methods, can also be used between the server 801 and the terminal 802, and no specific limitations are made thereto. Figure 8 The system may include multiple terminals 802, but only one terminal 801 is used as an example. This application does not provide specific examples of each terminal 802.

[0229] The aforementioned terminal 802 can be various types of devices equipped with display components. For example, terminal 802 can be a mobile phone, tablet computer, laptop computer, smart TV, or other terminal devices. Figure 8 (Taking a mobile phone as an example, the terminal can also be a device used for virtual scene interaction, including VR glasses, AR devices, MR interactive devices, etc. The terminal can also be wearable electronic devices such as smartwatches and smart bracelets, and can also be a device mounted on vehicles, autonomous vehicles, drones, industrial robots, etc. This application does not specifically limit the specific form of the terminal.)

[0230] In addition, the aforementioned terminals may also be referred to as user equipment (UE), subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, terminal equipment, access terminal, mobile terminal, wireless terminal, smart terminal, remote terminal, handheld device, user agent, mobile client, client, or any other suitable term.

[0231] The aforementioned server 801 can be one or more physical servers. Figure 8 (Taking a physical server as an example) It can also be a computer cluster, a virtual machine or cloud server in a cloud computing scenario, and so on.

[0232] In one example, terminal 802 may have a client installed. This client could be a video playback application, a live streaming application (e.g., e-commerce live streaming, game live streaming, etc.), a video conferencing application, or a game application—any application involving video encoding and decoding (app). Terminal 802 can run the client based on user actions (e.g., clicks, touches, swipes, shaking, voice control, etc.), access video content, and display the video content on a display component.

[0233] Server 801 can function as source device 12 as described in the above embodiments. Using the video image processing method of this application, it provides two bitstreams of equal or equivalent quality for the same video content. One bitstream is a long GOP bitstream, and the other is a random access bitstream. Terminal 802 can function as destination device 14 as described in the above embodiments. By decoding the random access bitstream and the long GOP bitstream, it enables rapid access to video content.

[0234] Specifically, server 801 can acquire video images, which can be video images captured by a camera or decoded video images. The camera can be, for example... Figure 6The image shows any camera position. Server 801 can use the video image processing method of this embodiment to provide two bitstreams of equal or equivalent quality for the same video content. One bitstream is a long GOP bitstream, and the other is a random access bitstream. Server 801 can provide these two bitstreams to the client. In one implementation, for video-on-demand applications, the client can support user-demanded video content functionality, and server 801 can store both bitstreams. Upon receiving a video content request from the client, both bitstreams are sent to the client. The client can first decode the random access bitstream, and then decode the long GOP bitstream based on the decoding result of the random access bitstream to quickly access the video content. Alternatively, upon receiving a video content request from the client, if the video content corresponding to time t0 is an instantaneous decoding refresh (IDR) frame in the long GOP bitstream, the server can send the long GOP bitstream to the client. The client can directly decode the long GOP bitstream to access the video content. If the video content corresponding to time t0 is not an IDR frame in the long GOP stream, the server can send both a random access stream (containing the video content from time t0 onwards) and a long GOP stream to the client. The client can first decode the random access stream, and then decode the long GOP stream based on the decoding result to quickly access the video content. The client can also render and display the decoded video content to present it to the user. Alternatively, for live streaming applications, the client can support live video content functionality. When the server (801) receives a request to access live video content, it can send both streams to the client. The client can first decode the random access stream, and then decode the long GOP stream based on the decoding result to quickly access the video content. The client can also render and display the decoded video content to present it to the user.

[0235] In another example, an intermediate node 803 can be set between server 801 and terminal 802. Server 801 can act as source device 12 as described in the above embodiments, providing two bitstreams of the same or equivalent quality for the same video content through the video image processing method of this application embodiment. One bitstream is a long GOP bitstream, and the other is a random access bitstream. The intermediate node 803 can act as destination device 14 as described in the above embodiments, decoding the random access bitstream and the long GOP bitstream. The intermediate node 803 provides the decoded video content to terminal 802 to achieve fast access to video content. For example, the intermediate node 803 can be a node in a content delivery network (CDN).

[0236] In one possible implementation, for live streaming applications, the client can support live video content functionality. When server 801 receives a request to access live video content, it can send two types of bitstreams to intermediate node 803. Intermediate node 803 can first decode the randomly accessed bitstream, and then decode a long GOP bitstream based on the decoding result. This intermediate node 803 provides the decoded video content to terminal 802 to achieve rapid access to video content. The client can also render and display the decoded video content to present the video content to the user.

[0237] For example, a client sends a request to the server to access live video content. This request requests access to live content starting from time t0. When the server receives the request from the client, it can send a random access bitstream containing video content from time t0 onwards and a long GOP bitstream to an intermediate node or the client. The intermediate node or the client can first decode the random access bitstream and then decode the long GOP bitstream based on the decoding result to quickly access the video content.

[0238] It should be noted that, in this embodiment of the application, any of the above-mentioned applications of terminal 801 can be applications built into terminal 801 itself, or applications provided by third-party service providers that are installed by the user, and there is no specific limitation in this regard.

[0239] It should be noted that the two bitstreams of the same or equivalent quality involved in the embodiments of this application refer to the reconstructed frames of the long GOP bitstream and the reconstructed frames of the corresponding random access bitstream having the same or equivalent quality. This is so that using the reconstructed frames of the random access bitstream as reference frames to decode the long GOP bitstream is beneficial to reducing block artifacts and eliminating some artifact effects.

[0240] The reconstructed frames of long GOP bitstreams of the same or equivalent quality and the reconstructed frames of the corresponding random access bitstreams satisfy one or more of the following:

[0241] The difference between the reconstructed frame of the long GOP stream and the reconstructed frame of the corresponding random access stream is less than the difference threshold; or,

[0242] The similarity between the reconstructed frames of the long GOP bitstream and the reconstructed frames of the corresponding random access bitstream is higher than the similarity threshold; or,

[0243] The difference between the reconstructed frames of the long GOP stream and the corresponding reconstructed frames of the random access stream is smaller than the difference between the reconstructed frames of the long GOP stream and the random access stream obtained by encoding the same video content using other partitioning methods or coding parameters; or,

[0244] The similarity between reconstructed frames of a long GOP stream and the corresponding reconstructed frames of a random access stream is higher than the similarity between reconstructed frames of a long GOP stream and a random access stream obtained by encoding the same video content using other partitioning methods or encoding parameters; or,

[0245] The difference between the pixel values ​​at the same position in the reconstructed frame of the long GOP bitstream and the reconstructed frame of the corresponding random access bitstream is less than the pixel value threshold; for example, the pixel value threshold can be 128, or it can be other values, which will not be illustrated in the embodiments of this application.

[0246] The reconstructed frames of long GOP streams and the corresponding reconstructed frames of random access streams specifically refer to the reconstructed frames of long GOP streams and random access streams at the same position. Figure 7 Taking another example of a long GOP stream and a random access stream of video content, the reconstructed frames of the long GOP stream and the random access stream at the same position can include the reconstructed frame of frame number 3 (#3 frame) of the long GOP stream and the reconstructed frame of frame number 3 (#3 frame) of the random access stream.

[0247] The video image processing method of this application embodiment can adjust the encoding of the second bitstream based on the encoding result of the first bitstream (e.g., the reconstructed image of the first bitstream or the reconstructed image in the first encoding process, and / or the encoding information of the first encoding) to achieve that the reconstructed frames of the first bitstream and the corresponding reconstructed frames of the second bitstream have the same or equivalent quality. The video image processing method of this application embodiment can encode the input image to be encoded to generate a first bitstream and a second bitstream. That is, the same video content can be encoded to generate a first bitstream and a second bitstream of the same or equivalent quality. The reconstructed frames of the first bitstream and the corresponding reconstructed frames of the second bitstream have the same or equivalent quality. Here, the first bitstream can be a long GOP bitstream as described above. Here, the second bitstream can be a random access bitstream as described above. Adjusting the encoding of the second bitstream based on the encoding result of the first bitstream can have different specific implementations. For example, the reconstructed frames of the first bitstream and the corresponding reconstructed frames of the second bitstream can be of the same or equivalent quality through the following embodiments.

[0248] In the first bitstream, the interval between two adjacent intra-prediction modes is greater than the interval between two adjacent intra-prediction modes in the second bitstream.

[0249] Please see Figure 9 , Figure 9 This is a schematic diagram illustrating a video image processing method provided in an embodiment of this application. This embodiment can be executed by an encoding device. The encoding device can be applied to the source device 12 in the above embodiments, for example, such as... Figure 8 Server 801 in the illustrated embodiment. (As shown) Figure 9 As shown, the encoding device can acquire an image to be encoded, perform a first encoding on the image to be encoded, and generate a first bitstream. Then, based on the encoding information from the first encoding and / or the first reconstructed image, a second encoding using intra-frame prediction mode is performed on the image to be encoded or the first reconstructed image to generate a second bitstream. The first reconstructed image is either the first bitstream or a reconstructed image from the first encoding process.

[0250] It should be noted that, as Figure 9 As shown, the first encoding is used to generate the first bitstream, and the second encoding is used to generate the second bitstream. The first and second bitstreams are generated through different encoding processes. Regarding the transmission of the first and second bitstreams... Figure 9 As not shown, the first and second bitstreams can be transmitted independently or interleaved before transmission.

[0251] The following is based on Figures 10 to 15 The embodiments shown above relate to the above. Figure 9 The several possible implementation methods of the illustrated embodiments are explained in detail. Regarding the second encoding process, Figures 10 to 15In the illustrated embodiment, based on the first reconstructed image, a second encoding in full intra-frame prediction mode is performed on the image to be encoded or the first reconstructed image to generate a second bitstream; or, based on the encoding information of the first reconstructed image and the first encoding, a second encoding in full intra-frame prediction mode is performed on the image to be encoded or the first reconstructed image to generate a second bitstream.

[0252] Please see Figure 10 , Figure 10 This is a flowchart illustrating a video image processing method provided in an embodiment of this application. The method portion of this embodiment can be executed by an encoding device. The encoding device can be applied to the source device 12 in the above embodiments, for example, as... Figure 8 The server 801 in the illustrated embodiment. It should be understood that the series of steps or operations involved in the embodiments of this application can be executed in various orders and / or occur simultaneously, and the execution order is not fixed. Figure 10 The step numbers shown serve as a constraint. For example... Figure 10 The method shown may include the following implementation steps:

[0253] Step 1001: Obtain the image to be encoded.

[0254] The image to be encoded in this application embodiment can be a video image captured by a camera or other acquisition device, or a decoded video image. The decoded video image can be an image obtained by decoding a compressed video image.

[0255] In one implementation, the image to be encoded can be a source video image. Therefore, the source video image is subjected to first encoding and second encoding through the following steps to generate a first bitstream and a second bitstream, achieving frame-level synchronous output. In another implementation, the image to be encoded can be an image block obtained by dividing the source video image. Therefore, the divided image block is subjected to first encoding and second encoding through the following steps to generate a first bitstream and a second bitstream, achieving block-level synchronous output.

[0256] Step 1002: Perform the first encoding on the image to be encoded to generate the first bitstream.

[0257] The first encoding may include one or more processing steps such as prediction, transformation, quantization, and entropy coding. For example, the image to be encoded may be predicted, transformed, and quantized to generate first encoded data, and then entropy coding may be performed on the first encoded data to generate a first bitstream including the first encoded data.

[0258] Optionally, the prediction mode of the first encoding can be inter-frame prediction. The image to be encoded is first encoded to generate a first bitstream comprising P frames or P blocks. Alternatively, the prediction mode of the first encoding can also be intra-frame prediction. The image to be encoded is first encoded to generate a first bitstream comprising frames or I blocks comprising the full intra-frame prediction mode. For example, as... Figure 7 The long GOP bitstream shown, i.e., the first bitstream here, can include P-frames and frames in full intra-frame prediction mode.

[0259] Step 1003: Based on the first reconstructed image, determine at least one of the first partitioning method or the first encoding parameter used for the second encoding of the image to be encoded or the first reconstructed image, wherein the first reconstructed image is the first bitstream or the reconstructed image in the first encoding process.

[0260] This application embodiment can decode a first bitstream to obtain a first reconstructed image. Alternatively, this application embodiment can acquire the reconstructed image during the first encoding process. For example, during the first encoding process, the aforementioned first encoded data is subjected to inverse quantization, inverse transform, and other processing to obtain a first reconstructed image. Based on the first reconstructed image, a first partitioning method and / or first encoding parameters are determined for the second encoding of the image to be encoded or the first reconstructed image.

[0261] The first encoding parameter may include, but is not limited to, the first quantization parameter (QP) or the first code rate.

[0262] In one possible implementation, the image to be encoded can be a source video image. Accordingly, the first reconstructed image is the first reconstructed frame, obtained by decoding the first bitstream corresponding to the source video image, or obtained by performing inverse quantization, inverse transform, or other processing on the first encoded data corresponding to the source video image during the first encoding process. In another possible implementation, the image to be encoded can be an image block obtained by dividing the source video image. Accordingly, the first reconstructed image is the first reconstructed image block, obtained by decoding the first bitstream corresponding to the divided image block, or obtained by performing inverse quantization, inverse transform, or other processing on the encoded data corresponding to the divided image block during the first encoding process.

[0263] Optionally, another possible implementation of step 1003 above involves determining at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the image to be encoded or the first reconstructed image, based on the encoding information of the first reconstructed image and the first encoding. The encoding information of the first encoding may include one or more of the first encoding partitioning method, the first encoding quantization parameter, and the first encoding encoding distortion information.

[0264] Step 1004: Perform second encoding of the image to be encoded or the first reconstructed image in full-frame prediction mode according to at least one of the first partitioning method or the first encoding parameters to generate a second bitstream.

[0265] Based on the first partitioning method and / or the first encoding parameters, the image to be encoded or the first reconstructed image is subjected to second encoding in full-frame prediction mode to generate a second bitstream.

[0266] In other words, the prediction mode of the second encoding can be intra-frame prediction. Therefore, the image to be encoded or the first reconstructed image is subjected to second encoding to generate a second bitstream that includes frames or I-blocks with full intra-frame prediction modes. For example, as... Figure 7 The random access bitstream shown, i.e. the second bitstream here, may include frames in full intra-frame prediction mode.

[0267] The second encoding may include one or more processing steps such as prediction, transformation, quantization, and entropy coding. For example, the image to be encoded or the first reconstructed image may be predicted, transformed, and quantized to generate second encoded data, and then entropy coding may be performed on the second encoded data to generate a second bitstream including the second encoded data.

[0268] The difference between the second and first encoding methods lies in their prediction modes; the second encoding uses an intra-frame prediction mode. It's understandable that the encoding parameters and other information can also differ between the second and first encoding methods.

[0269] In some embodiments, the difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold. The second reconstructed image is obtained by decoding the second bitstream or by performing inverse quantization, inverse transform, or other processing on the second encoded data during the second encoding process. The difference threshold or similarity threshold can be set reasonably according to requirements.

[0270] The difference is used to represent the difference between the features of the first reconstructed image and the features of the second reconstructed image. The difference can be measured using metrics such as mean absolute differences (MAD), sum of absolute differences (SAD), sum of squared differences (SSD), mean square differences (MSD), or sum of absolute transformed differences (SATD). The larger the MAD, SAD, SSD, MSD, or SATD, the greater the difference, and the more different the qualities of the first and second reconstructed images. The smaller the MAD, SAD, SSD, MSD, or SATD, the smaller the difference, and the more similar or comparable the qualities of the first and second reconstructed images. Similarity is used to represent the similarity between the features of the first reconstructed image and the features of the second reconstructed image. Similarity can be measured using metrics such as MAD, SAD, SSD, MSD, or SATD. The larger the MAD, SAD, SSD, MSD, or SATD, the lower the similarity, and the more different the qualities of the first and second reconstructed images. The smaller the values ​​of MAD, SAD, SSD, MSD, or SATD, the higher the similarity, and the more similar or comparable the quality of the first and second reconstructed images.

[0271] The aforementioned difference thresholds can be set to 0 or as needed. For example, when MAD is selected as the metric for measuring difference, the luminance signal threshold can be set to 4, and the chrominance signal threshold can be set to 2. Correspondingly, when SAD, SSD, or MSD are selected as the metric for measuring difference, the luminance signal threshold can be set to 4xN, 16xN, and 16, respectively, and the chrominance signal threshold can be set to 2xN, 4xN, and 4, respectively, where N is the total number of pixels within the range of the object being measured (which can be a coded block or an image). When SATD is selected as the metric for measuring difference, the threshold can be set to 0. The similarity threshold can be set similarly.

[0272] Optionally, a specific implementation of step 1003 above may be as follows: based on the first reconstructed image, determine a plurality of second partitioning methods, and select one of the plurality of second partitioning methods as the first partitioning method; and / or, based on the first reconstructed image, determine a plurality of second coding parameters, and select one of the plurality of second coding parameters as the first coding parameter.

[0273] Thus, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and multiple third reconstructed images. These multiple third reconstructed images are reconstructed images obtained by performing multiple second encodings on the image to be encoded or the first reconstructed image, respectively, according to the aforementioned multiple second partitioning methods and / or the aforementioned multiple second encoding parameters. The multiple third reconstructed images can be reconstructed images from multiple second encoding processes, or they can be reconstructed images from multiple bitstreams obtained through multiple second encodings.

[0274] For example, the encoding device can perform multiple second encodings on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters to generate multiple third encoded data. Then, during the multiple second encoding processes, the multiple third encoded data are subjected to inverse quantization, inverse transform, and other processing to obtain multiple third reconstructed images. By comparing the similarity between each of the multiple third reconstructed images and the first reconstructed image, the one with the highest similarity is selected as the second reconstructed image. In other words, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the multiple third reconstructed images. The third encoded data corresponding to the third reconstructed image with the highest similarity is used as the second encoded data to generate a bitstream including the second encoded data. Alternatively, the bitstream corresponding to the third reconstructed image with the highest similarity is used as the aforementioned second bitstream.

[0275] Optionally, before step 1003, the video image processing method of this application embodiment may further include: determining whether the prediction mode of the first encoding is intra-frame prediction. If the prediction mode of the first encoding is inter-frame prediction, then step 1003 is executed. If the prediction mode of the first encoding is intra-frame prediction, the first bitstream is used as the second bitstream, or the first encoded data is used as the second encoded data to generate the second bitstream. In this way, when the image to be encoded is first encoded to obtain a first bitstream including P frames or P blocks, steps 1003 and 1004 can be executed to perform second encoding on the image to be encoded or the first reconstructed image to generate a second bitstream including frames or I blocks in the full intra-frame prediction mode. When the image to be encoded is first encoded to obtain a first bitstream including frames or I blocks in the full intra-frame prediction mode, steps 1003 and 1004 can be skipped, and the frames or I blocks in the full intra-frame prediction mode can be directly used as the second encoded data, thereby improving encoding efficiency.

[0276] For example, as mentioned above Figure 7 Taking another example of a long GOP bitstream and random access bitstream of video content, the encoding device performs a first encoding on the image to be encoded to generate a long GOP bitstream including frame number 3 (frame #3). The encoding device determines whether the prediction mode of the first encoding is intra-frame prediction. Figure 7As shown, the prediction mode for the first encoding at this point is inter-frame prediction. Subsequently, the encoding device can perform a second encoding on the image to be encoded or the reconstructed frame of frame number 3 (#3 frame) by executing steps 1003 and 1004, to generate a random access bitstream for frame number 3 (#3 frame). The prediction mode for frame number 3 (#3 frame) in the random access bitstream is intra-frame prediction.

[0277] It should be noted that in some embodiments, when the image to be encoded is first encoded to obtain a first bitstream including frames or I-blocks of the full intra-frame prediction mode, the frames or I-blocks of the full intra-frame prediction mode may not be used as the second encoded data. That is, the second bitstream may also not include the frames or I-blocks of the full intra-frame prediction mode, which can be reasonably set according to the video transmission requirements.

[0278] It should also be noted that the aforementioned second encoded data can also be referred to as a random access frame.

[0279] Optionally, taking the encoding device applied to a server as an example, the server can store a first bitstream and a second bitstream. Upon receiving a video content request from a client, the server sends the first and second bitstreams to the client.

[0280] In one example, in a video-on-demand application scenario, a client requests playback starting at time t0 of a video. The server will download the first bitstream, which includes the video content at time t0. If the video content corresponding to time t0 is an access frame (e.g., a frame in the full intra-frame prediction mode described above) within the first bitstream, the server will only provide this first bitstream to the client. The client then decodes and plays this first bitstream. For example, the video content corresponding to time t0, within a long GOP bitstream, might look like this... Figure 7 The frame with encoding 0 (#0 frame) shown in the example only provides the long GOP stream to the client. If the video content corresponding to time t0 is not an access frame in the first stream, the server also needs to send the client the random access frame at time t0 or the closest random access frame to time t0 in the second stream. The client can first decode the random access frame, and then use the reconstructed frame of that random access frame to decode the reference frame of the second stream, thus decoding and playing the second stream. For example, the video content corresponding to time t0 in the long GOP stream is as follows: Figure 7 If the frame encoded as 3 (#3 frame) is shown, the server also needs to send the frame encoded as 3 (#3 frame) from the random access stream to the client. The client can first decode the frame encoded as 3 (#3 frame) from the random access stream, use the reconstructed frame of the frame encoded as 3 (#3 frame) from the random access stream as a reference frame, decode the frame encoded as 4 (#4 frame) from the long GOP stream, and then decode and play the subsequent frames of the long GOP stream.

[0281] Another example is a live streaming application scenario. A client requests to access the live stream at time t0, and the server sends a first bitstream and a second bitstream to the client. The client first decodes the random access frame at time t0 or the closest random access frame to time t0 in the second bitstream. Then, using the reconstructed frame of the random access frame as a reference frame, the client decodes the subsequent frames of the first bitstream. The subsequent frames of the first bitstream refer to the frames in the first bitstream that are after the time of the random access frame.

[0282] In this embodiment, the image to be encoded is first encoded to generate a first bitstream. Based on the first reconstructed image, at least one of a first partitioning method or a first encoding parameter is determined for the second encoding of the image to be encoded or the first reconstructed image. Based on the first partitioning method or the first encoding parameter, the image to be encoded or the first reconstructed image is second encoded to generate a second bitstream. The first reconstructed image is the first bitstream or the reconstructed image in the first encoding process. Thus, based on the first bitstream or the reconstructed image in the first encoding process, the encoding of the second bitstream is adjusted to ensure that the reconstructed image of the first bitstream and the corresponding reconstructed image of the second bitstream have the same or equivalent quality. This improves the decoding quality of the accessed video content while meeting the requirements for low-latency access, reduces block artifacts, and eliminates some artifact effects.

[0283] The following explanation uses the image to be encoded as an image block after dividing the source video image to illustrate the video image processing method of this application embodiment.

[0284] Please see Figure 11 , Figure 11 This is a flowchart illustrating a video image processing method provided in an embodiment of this application. The method portion of this embodiment can be executed by an encoding device. The encoding device can be applied to the source device 12 in the above embodiments, for example, as... Figure 8 The server 801 in the illustrated embodiment. This embodiment uses the Nth image block of the Kth frame as an example. The encoding device can encode the Nth image block of the Kth frame to generate a first bitstream and a second bitstream. For example... Figure 11 The method shown may include the following implementation steps:

[0285] Step 1101: Obtain the Nth image block of the Kth frame.

[0286] The encoding device can receive the input Kth frame image, divide the Kth frame image into blocks, and obtain multiple image blocks of the Kth frame image. This application embodiment uses the encoding of the Nth image block of the Kth frame image as an example for illustration; other image blocks can be processed in the same or similar ways, and this application embodiment will not explain them one by one.

[0287] Step 1102: Perform the first encoding on the Nth image block to generate the first bitstream.

[0288] The encoding device can use information such as prediction mode P, partitioning method D, and quantization parameter QP to perform a first encoding on the Nth image block to generate a first bitstream. This first bitstream may include the first encoded data of the Nth image block. During the first encoding process, a reconstructed image block A of the Nth image block can also be generated.

[0289] Step 1103: Obtain the Nth image patch, the prediction mode P, and the reconstructed image patch A.

[0290] Using the Nth image block and the reconstructed image block A as input information, we begin encoding to generate the second bitstream.

[0291] Optionally, the partitioning method D and quantization parameter QP involved in the first encoding can also be used as input information.

[0292] Step 1104: Determine whether the prediction mode P is intra-frame prediction. If yes, use the encoding result of the Nth image block in the first bitstream as the encoding result of the current block in the second bitstream. If no, proceed to step 1005.

[0293] When the prediction mode P of the Nth image block is intra-frame prediction, the encoding result of the current block of the second bitstream directly uses the encoding result of the Nth image block of the first bitstream. When the prediction mode P of the Nth image block is inter-frame prediction, the original image block to be encoded at the position corresponding to the reconstructed image block A in the Kth frame image, i.e., the aforementioned Nth image block, is subjected to the second encoding of the intra-frame prediction mode through the following steps.

[0294] Step 1105: Divide the Nth image block into segments using a specific method, and perform second encoding of all sub-blocks in the full-frame prediction mode to obtain the reconstructed image block B of the second bitstream.

[0295] For example, the Nth image block is divided in a certain way, and a second encoding process is performed on all the sub-blocks after the division, including intra-frame prediction, transformation, quantization, inverse quantization, and inverse transformation, to obtain the reconstructed image block B of the second bitstream.

[0296] The method for partitioning the Nth image block includes, but is not limited to, partitioning the Nth image block into 2Nx2N, NxN, or no partitioning. For example, partitioning the Nth image block into 2Nx2N means dividing the Nth image block into 2Nx2N sub-blocks. N is any positive integer greater than 1.

[0297] When this step is performed for the Nth image block for the first time, the coding information and / or coding parameters used in the second coding process, such as partitioning, intra-frame prediction, transformation, and quantization, including partitioning method, QP, and bit rate, can be randomly generated coding information and / or coding parameters during initialization, or they can be the coding information and / or coding parameters used by the I-frame preceding the Kth image. For example, for QP, the average QP of one or more of the nearest I-frames preceding the Kth image can be used.

[0298] Reconstructed image block B is the image block in the reconstructed image of the Kth frame of the second bitstream that corresponds to reconstructed image block A. That is, it is the image block in the reconstructed image of the Kth frame of the second bitstream that has the same position as reconstructed image block A in the Kth frame of the first bitstream. Reconstructed image block B can consist of reconstructed blocks from one or more coded sub-blocks.

[0299] Step 1106: Calculate the similarity cost function values ​​of reconstructed image patch A and reconstructed image patch B.

[0300] Specifically, the similarity cost function value f(partition method, QP) between reconstructed image block A and reconstructed image block B can be calculated according to the following formula (1).

[0301]

[0302] Where i represents the index of a pixel in the reconstructed image block, I represents the total number of pixels in the reconstructed image block, B(partition method, QP, i) represents the reconstructed pixel value at the i-th pixel position using the quantization parameter QP under one partition method for reconstructed image block B, and A(i) represents the reconstructed pixel value at the i-th pixel position for reconstructed image block A. T can be 1 or 2.

[0303] Optionally, in addition to formula (1), other methods can be used to evaluate the similarity between two reconstructed image patches, including but not limited to MAD, SAD, SSD, MSD, SATD, etc. For example, the similarity between two reconstructed image patches can be evaluated by any one of the following formulas (2) to (5), or by calculating the sum of the absolute values ​​of the difference images of the two reconstructed image patches after Adama transform.

[0304] The similarity cost function of MAD is used to evaluate the similarity between two reconstructed image patches:

[0305]

[0306] The similarity cost function of SAD is used to evaluate the similarity between two reconstructed image patches:

[0307]

[0308] The similarity cost function of SSD is used to evaluate the similarity between two reconstructed image patches:

[0309]

[0310] The similarity cost function of MSD is used to evaluate the similarity between two reconstructed image patches:

[0311]

[0312] The similarity cost function of SATD is used to evaluate the similarity between two reconstructed image patches: the sum of the absolute values ​​of the difference images of the two reconstructed image patches after Adama transform is calculated to evaluate the similarity between the two reconstructed image patches.

[0313] Among them, the similarity cost function values ​​of reconstructed image block A and reconstructed image block B are calculated using any of the above similarity cost functions. The smaller the similarity cost function value, the higher the similarity between reconstructed image block A and reconstructed image block B.

[0314] Step 1107: Determine whether the similarity cost function values ​​of reconstructed image block A and reconstructed image block B are less than the similarity cost function threshold, or whether they are finitely iteratively local optimal. If yes, proceed to step 1109; otherwise, proceed to step 1108.

[0315] Optionally, if the similarity cost function value between reconstructed image block A and reconstructed image block B is less than the similarity cost function value threshold, then step 1109 is executed directly. If the result is a finite iterative local optimum, then step 1109 is executed directly. Specifically, a finite iterative local optimum means that, after using all partitioning methods and / or encoding parameters to perform a second encoding on the Nth image block, and calculating the similarity cost function values ​​between reconstructed image block B and reconstructed image block A for each partitioning method and / or encoding parameter, the one with the highest similarity (e.g., the smallest similarity cost function value) is selected as the finite iterative local optimum. Then, for the encoded data corresponding to the reconstructed image block B with the highest similarity, and the partitioning method and / or encoding parameters, step 1109 is executed.

[0316] The similarity cost function threshold can be flexibly set according to requirements. For example, it can be set to 0, or the similarity cost function threshold for evaluating similarity using MAD, SAD, SSD, and MSD for luminance signals can be set to 4, 4xI, 16xI, and 16, respectively. In other words, when the average luminance difference between reconstructed image block B and reconstructed image block A is less than 4 or 16, step 1109 is executed. As another example, the similarity cost function threshold for evaluating similarity using MAD, SAD, SSD, and MSD for chrominance signals can be set to 2, 2xI, 4xI, and 4, respectively. In other words, when the average luminance difference between reconstructed image block B and reconstructed image block A is less than 2 or 4, step 1109 is executed. Yet another example, the similarity cost function threshold for evaluating similarity using SATD can be set to 0.

[0317] Optionally, in the process of comparing the similarity of two reconstructed image patches, if there are individual pixels with large differences, such as a difference in grayscale values ​​exceeding 128, then this segmentation method and the corresponding quantization result can be discarded.

[0318] After performing step 1109, step 1101 can be repeated to begin the first encoding of the (N+1)th image block of the Kth frame image until the first and second encodings of the entire frame are completed.

[0319] Step 1108: Change the partitioning method and / or encoding parameters, and repeat step 1105.

[0320] This application embodiment may have multiple partitioning methods and / or encoding parameters. One partitioning method and / or encoding parameter may be selected from the multiple partitioning methods and / or encoding parameters, and step 1105 may be executed repeatedly to traverse the multiple partitioning methods and / or encoding parameters, perform the second encoding on the Nth image block, and calculate the similarity cost function values ​​of the reconstructed image block B and the reconstructed image block A for each partitioning method and / or encoding parameter.

[0321] Taking the encoding parameter QP as an example, within a certain range (e.g., 0 to 51), QP is selected with a certain step size (e.g., 1 or 2), and step 1105 is repeated until the finite number of QPs is enumerated.

[0322] Taking the partitioning method as an example, for each partitioning method, execute step 1105 until all the finite partitioning methods have been enumerated.

[0323] Step 1109: Perform entropy encoding on the second encoded data, as well as the partitioning method and / or encoding parameters, to generate a second bitstream and complete the second encoding of the Nth image block.

[0324] The second bitstream may include the second encoded data of the Nth image block.

[0325] If the similarity cost function value between reconstructed image patch A and reconstructed image patch B is less than the similarity cost function value threshold, then the second encoded data here is the encoded data corresponding to reconstructed image patch B. The encoded data corresponding to reconstructed image patch B refers to the encoded data obtained using a partitioning method and / or encoding parameters. Reconstructed image patch B can be the reconstructed image of this encoded data.

[0326] If the finite iterative local optimum is found, then the second encoded data here is the encoded data corresponding to the reconstructed image block B that is the finite iterative local optimum. Specifically, finite iterative local optimum means that all partitioning methods and / or encoding parameters have been used to encode the Nth image block, and the similarity between the reconstructed image block B and the reconstructed image block A for each partitioning method and / or encoding parameter has been calculated. The one with the highest similarity is selected as the finite iterative local optimum, and the encoded data corresponding to the reconstructed image block B with the highest similarity is used as the second encoded data here.

[0327] Optionally, other information can be combined to determine whether to execute step 1105. For example, the frame type used in the first encoding, the reference relationship, or whether temporal motion vector prediction is turned off can be combined to determine whether to execute step 1105. For example, when there is only a single frame reference in the first encoding and temporal motion vector prediction is turned off, step 1105 is executed, that is, the random access frame of the second bitstream is encoded.

[0328] Optionally, other information can be combined to determine whether to execute step 1105. For example, the parameters of tools such as intra-frame prediction strong filtering and SAO filtering used in the first encoding can be combined to determine whether to execute step 1105. For example, if there is a first encoding with intra-frame prediction strong filtering or SAO filtering enabled, step 1105 is not executed, and a randomly accessed frame is encoded.

[0329] In this embodiment, the encoding of the second bitstream is adjusted based on the encoding result of the first bitstream, including the Nth image block, to ensure that the reconstructed image blocks of the first bitstream and the corresponding reconstructed image blocks of the second bitstream have the same or equivalent quality. This improves the decoding quality of the accessed video content while maintaining low latency access, reduces block artifacts, and eliminates some artifacts. Through image block-level encoding of the first and second bitstreams, synchronous output of the two bitstreams at the image block level can be achieved, enabling faster generation of random access frames for the second bitstream used to access the video content and reducing access latency.

[0330] The following explanation uses the image to be encoded as the source video image to illustrate the video image processing method of this application embodiment.

[0331] Please see Figure 12 , Figure 12This is a flowchart illustrating a video image processing method provided in an embodiment of this application. The method portion of this embodiment can be executed by an encoding device. The encoding device can be applied to the source device 12 in the above embodiments, for example, as... Figure 8 The server 801 in the illustrated embodiment. This embodiment takes the Kth frame image as an example. The encoding device can encode the Kth frame image to generate a first bitstream and a second bitstream. Figure 12 The method shown may include the following implementation steps:

[0332] Step 1201: Perform first encoding on the Kth frame image to generate the first bitstream.

[0333] The encoding device can receive the input Kth frame image. This application embodiment uses the encoding of the Kth frame image as an example for illustration. Other frames can be processed in the same or similar way, and this application embodiment will not explain them one by one.

[0334] The encoding device can use information such as prediction mode P, partitioning method D, and quantization parameter QP to perform first encoding on the Kth frame image to generate a first bitstream. This first bitstream may include the first encoded data of the Kth frame image. During the first encoding process, a reconstructed frame A of the Kth frame image can also be generated.

[0335] Step 1202: Divide the Kth frame image into segments using a certain method, and perform second encoding of all image blocks in the full-frame prediction mode to obtain the reconstructed frame B of the second bitstream.

[0336] Using the Kth frame image and the reconstructed frame A as input information, we begin encoding to generate the second bitstream.

[0337] Optionally, the partitioning method D and quantization parameter QP involved in the first encoding can also be used as input information.

[0338] For example, the Kth frame image is divided in a certain way, and a second encoding process is performed on all the sub-blocks after the division, including intra-frame prediction, transformation, quantization, inverse quantization, and inverse transformation, to obtain the reconstructed frame B of the second bitstream.

[0339] When this step 1202 is performed for the first time on the Kth frame image, the coding information and / or coding parameters used in the second coding process, such as partitioning, intra-frame prediction, transformation, and quantization, including, for example, partitioning method, QP, and bit rate, can be randomly generated coding information and / or coding parameters during initialization, or they can be the coding information and / or coding parameters used in the I-frames preceding the Kth frame image. For example, for QP, the average QP of one or more of the nearest I-frames preceding the Kth frame image can be used.

[0340] Reconstructed frame B can consist of reconstructed blocks of one or more coded image blocks.

[0341] Optionally, when the prediction mode P of the Kth frame image is intra-frame prediction, the second encoded data of the current frame of the second bitstream directly uses the first encoded data of the Kth frame image of the first bitstream. When the prediction mode P of the Kth frame image is not intra-frame prediction (for example, the Kth frame image is a P frame, or there is a P block), the Kth frame image is encoded using the second encoding mode of intra-frame prediction in step 1202.

[0342] Step 1203: Calculate the similarity cost function values ​​of reconstructed frame A and reconstructed frame B.

[0343] Specifically, the similarity cost function value f(partition method, QP) between reconstructed frame A and reconstructed frame B can be calculated according to the following formula (6).

[0344]

[0345] Where i represents the index of a pixel in the reconstructed frame, I represents the total number of pixels in the reconstructed frame, B(partition method, QP, i) represents the reconstructed pixel value at the i-th pixel position using the quantization parameter QP under one partition method of reconstructed frame B, and A(i) represents the reconstructed pixel value at the i-th pixel position of reconstructed frame A. T can be 1 or 2.

[0346] Optionally, in addition to formula (6), other methods can be used to evaluate the similarity between two reconstructed frames, including but not limited to MAD, SAD, SSD, MSD, SATD, etc.

[0347] Step 1204: Change the partitioning method and / or encoding parameters, and repeat step 1202.

[0348] This application embodiment may have multiple partitioning methods and / or encoding parameters. One partitioning method and / or encoding parameter may be selected from the multiple partitioning methods and / or encoding parameters, and step 1202 may be executed repeatedly to traverse the multiple partitioning methods and / or encoding parameters, encode the Kth frame image, and calculate the similarity cost function values ​​of the reconstructed frame B and the reconstructed frame A for each partitioning method and / or encoding parameter.

[0349] Taking the encoding parameter QP as an example, within a certain range (e.g., 0 to 51), QP is selected with a certain step size (e.g., 1 or 2), and step 1202 is repeated until the finite number of QPs is enumerated.

[0350] Taking the partitioning method as an example, for each partitioning method, execute step 1202 until all the finite partitioning methods have been enumerated.

[0351] Step 1205: Perform entropy encoding on the second encoded data, as well as the partitioning method and / or encoding parameters, to generate the second bitstream and complete the encoding of the Kth frame image.

[0352] The second bitstream may include the second encoded data of the Kth frame image.

[0353] The second encoded data here is the encoded data corresponding to the reconstructed frame B, which is the local optimum of the finite iteration. Specifically, the local optimum of the finite iteration means that the K-th frame image has been encoded using all partitioning methods and / or encoding parameters, and the similarity between the reconstructed frame B and the reconstructed frame A for each partitioning method and / or encoding parameter has been calculated. The one with the highest similarity is selected as the local optimum of the finite iteration, and the encoded data corresponding to the reconstructed frame B with the highest similarity is used as the second encoded data here.

[0354] Optionally, if the similarity cost function value between reconstructed frame A and reconstructed frame B is less than a similarity cost function value threshold, then the second encoded data here is the encoded data corresponding to reconstructed frame B. The encoded data corresponding to reconstructed frame B refers to the encoded data obtained using a partitioning method and / or encoding parameters. Reconstructed frame B can be the reconstructed image of this encoded data.

[0355] Optionally, in the process of comparing the similarity of two reconstructed frames, if there are individual pixels with large differences, such as a difference in grayscale values ​​exceeding 128, then this division method and the corresponding quantization result can be discarded.

[0356] After performing step 1205, step 1201 can be repeated to begin encoding the K+1th frame image.

[0357] In this embodiment, the encoding of the second bitstream is adjusted based on the encoding result of the first bitstream, including the Kth frame image, to ensure that the reconstructed frames of the first bitstream and the corresponding reconstructed frames of the second bitstream have the same or equivalent quality. This improves the decoding quality of the accessed video content while maintaining low latency, reduces block artifacts, and eliminates some artifacts. By encoding the first and second bitstreams at the frame level, the quality of the first and second bitstreams is kept consistent at the frame level.

[0358] The video image processing method of the above embodiments controls the encoding of the current image of the second bitstream based on the reconstructed image of the current image (e.g., the current frame or the current block) of the first bitstream, so as to achieve that the reconstructed image of the first bitstream and the corresponding reconstructed image of the second bitstream have the same or equivalent quality. This application also provides a video image processing method according to the following embodiment, which controls the encoding of a second image to be encoded in the second bitstream based on at least one first image to be encoded in the first bitstream, wherein the second image to be encoded is a video image preceding at least one first image to be encoded, so as to achieve that the reconstructed image of the first bitstream and the corresponding reconstructed image of the second bitstream have the same or equivalent quality.

[0359] Please see Figure 13 , Figure 13 This is a flowchart illustrating a video image processing method provided in an embodiment of this application. The method portion of this embodiment can be executed by an encoding device. The encoding device can be applied to the source device 12 in the above embodiments, for example, as... Figure 8 The server 801 in the illustrated embodiment. It should be understood that the series of steps or operations involved in the embodiments of this application can be executed in various orders and / or occur simultaneously, and the execution order is not fixed. Figure 13 The step numbers shown serve as a constraint. For example... Figure 13 The method shown may include the following implementation steps:

[0360] Step 1301: Obtain at least one first image to be encoded and one second image to be encoded.

[0361] The second image to be encoded is a video image preceding at least one image to be encoded.

[0362] The second image to be encoded and one or more first images to be encoded in this application embodiment can be video images captured by a camera or other acquisition device, or decoded video images. The decoded video image can be an image obtained by decoding a compressed video image.

[0363] Step 1302: Perform first encoding on at least one first image to be encoded to generate a first bitstream.

[0364] The first encoding may include one or more processing steps such as prediction, transformation, quantization, and entropy coding. For example, at least one first image to be encoded may be predicted, transformed, and quantized to generate one or more first encoded data, and then entropy coding may be performed on the one or more first encoded data to generate a first bitstream including one or more first encoded data.

[0365] Optionally, the prediction mode of the first encoding can be inter-frame prediction. Alternatively, the prediction mode of the first encoding can also be intra-frame prediction.

[0366] Step 1303: Based on at least one first reconstructed image, determine at least one of the first partitioning method or first encoding parameters used for the second encoding of the second image to be encoded, wherein at least one first reconstructed image is a first bitstream or a reconstructed image in the first encoding process.

[0367] This application embodiment can decode a first bitstream to obtain one or more first reconstructed images. Alternatively, this application embodiment can perform inverse quantization, inverse transform, or other processing on one or more first encoded data during the first encoding process to obtain one or more first reconstructed images. Then, this application embodiment can determine a first partitioning method and / or first encoding parameters for performing a second encoding on a second image to be encoded based on one or more first reconstructed images.

[0368] In one implementation, at least one first image to be encoded can be at least one first source video image. Correspondingly, at least one first reconstructed image here is at least one first reconstructed frame. The at least one first reconstructed frame is obtained by decoding the first bitstream corresponding to each of the at least one first source video image, or it is a reconstructed frame of the first encoded data corresponding to each of the at least one first source video image in the first encoding process.

[0369] Step 1304: Perform second encoding of the second image to be encoded in full-frame prediction mode according to at least one of the first partitioning method or the first encoding parameters to generate a second bitstream.

[0370] According to the first partitioning method and / or the first encoding parameters, the second image to be encoded is subjected to second encoding in full-frame intra-prediction mode to generate a second bitstream. The second bitstream may include second encoded data.

[0371] The prediction mode for the second encoding can be intra-frame prediction. Therefore, the second image to be encoded is subjected to second encoding to generate a second bitstream that includes frames with full intra-frame prediction modes.

[0372] In some embodiments, the difference between the at least one first reconstructed image and the at least one second reconstructed image is less than a difference threshold, or the similarity between the at least one first reconstructed image and the at least one second reconstructed image is higher than a similarity threshold, and the at least one second reconstructed image is obtained by decoding the first bitstream using a third reconstructed image as a reference image. The difference threshold or similarity threshold can be set reasonably according to requirements. The third reconstructed image is a reconstructed image in the second bitstream or during the second encoding process.

[0373] In this process, the number of at least one first reconstructed image is the same as the number of at least one second reconstructed image. For example, an image to be encoded is first encoded to generate a first bitstream; based on a first reconstructed image, a second image to be encoded is second encoded to generate a second bitstream. The difference between the first and second reconstructed images is less than a difference threshold, or the similarity between the first and second reconstructed images is greater than a similarity threshold. The second reconstructed image is obtained by decoding the first bitstream using a third reconstructed image as a reference image.

[0374] For a detailed explanation of the differences and similarities, please refer to [link / reference]. Figure 10 The explanation of step 1004 in the illustrated embodiment will not be repeated here.

[0375] For example, multiple images to be encoded are first encoded to generate a first bitstream. Based on multiple first reconstructed images, a second image to be encoded is second encoded to generate a second bitstream. The difference between the multiple first reconstructed images and the multiple second reconstructed images is less than a difference threshold, or the similarity between the multiple first reconstructed images and the multiple second reconstructed images is higher than a similarity threshold. The multiple second reconstructed images are obtained by decoding the first bitstream using a third reconstructed image as a reference image. The difference between the multiple first reconstructed images and the multiple second reconstructed images can be a weighted sum of the differences between each of the multiple first reconstructed images and its corresponding second reconstructed image. The similarity between the multiple first reconstructed images and the multiple second reconstructed images can be a weighted sum of the similarities between each of the multiple first reconstructed images and its corresponding second reconstructed image.

[0376] Optionally, the above-mentioned at least one first image to be encoded is a first image to be encoded, and correspondingly, the at least one first reconstructed image is a first reconstructed image. A specific implementation of the above-mentioned step 1303 may be as follows: based on the first reconstructed image, select one of the multiple second partitioning methods as the first partitioning method; and / or select one of the multiple second encoding parameters as the first encoding parameter.

[0377] For example, the encoding device can perform multiple second encodings on the second image to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters to generate multiple third bitstreams. Each of the multiple third bitstreams may include a third encoded data. Then, the encoding device can decode the first bitstream using multiple fifth reconstructed images as reference images to obtain multiple fourth reconstructed images. By comparing the similarity between each of the multiple fourth reconstructed images and the first reconstructed image, the device selects the one with the highest similarity as the second reconstructed image. In other words, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the multiple fourth reconstructed images. The third bitstream corresponding to the fourth reconstructed image with the highest similarity is used as the second bitstream, or the third encoded data corresponding to the fourth reconstructed image with the highest similarity is used as the second encoded data to generate a second bitstream including the second encoded data.

[0378] Among them, the multiple fifth reconstructed images are reconstructed images of multiple third bitstreams, or reconstructed images from the multiple second encoding processes mentioned above.

[0379] Optionally, the at least one first image to be encoded can be multiple first images to be encoded, and correspondingly, the at least one first reconstructed image can be multiple first reconstructed images. A specific implementation of step 1303 can be as follows: based on the multiple first reconstructed images, select one second partitioning method as the first partitioning method from multiple second partitioning methods; and / or, select one second encoding parameter as the first encoding parameter from multiple second encoding parameters.

[0380] For example, taking m first images to be encoded as an example, the encoding device can perform x second encodings on the second images to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters to generate x third bitstreams. Each of the x third bitstreams can include one third encoded data. Then, the encoding device can use x fifth reconstructed images as reference images to decode the first bitstream to obtain x×m fourth reconstructed images. This can be understood as x sets of fourth reconstructed images, each set including m fourth reconstructed images. One of the x fifth reconstructed images is used as a reference image to decode the first bitstream, resulting in m fourth reconstructed images, i.e., a set of fifth reconstructed images. By comparing the similarity between each of the x sets of fourth reconstructed images and the m first reconstructed images, the set with the highest similarity is selected as the second reconstructed image corresponding to each of the m first images to be encoded. The third bitstream corresponding to the m fourth reconstructed images with the highest similarity is used as the second bitstream, or the third encoded data corresponding to the m fourth reconstructed images with the highest similarity is used as the second encoded data to generate a bitstream including the second encoded data. The third bitstream corresponding to the m most similar fourth reconstructed images refers to the m most similar fourth reconstructed images, which are obtained by decoding the first bitstream using the fifth reconstructed image of the third bitstream as a reference image.

[0381] The similarity between the second reconstructed image corresponding to each of the m first images to be encoded and the m first reconstructed images is a weighted sum of the similarities between the second reconstructed image corresponding to each of the m first images to be encoded and the corresponding first reconstructed image.

[0382] For example, the multiple first images to be encoded include m first images to be encoded, and the multiple first reconstructed images include m first reconstructed images, where the m first reconstructed images are A1, A2, ..., A m Let m be any positive integer greater than 1. The multiple fourth reconstructed images corresponding to the i-th first image among the m first images to be encoded are C. 1i , ..., C xi x is any positive integer greater than 1, and i ranges from 2 to m. x can represent the xth second encoding.

[0383] Multiple fifth reconstructed images are B1, ..., B x .

[0384] C 11 C is obtained by decoding the first encoded data corresponding to the first image to be encoded, using B1 as a reference image. 12 To make C 11 The first encoded data corresponding to the second first image to be encoded is obtained by decoding the reference image, ..., C 1m To make C 1(m-1)The first encoded data is obtained by decoding the first encoded data corresponding to the m-th first image to be encoded, which is used as a reference image.

[0385] C x1 To make B x C is obtained by decoding the first encoded data corresponding to the first image to be encoded, which is used as a reference image. x2 To make C x1 The first encoded data corresponding to the second first image to be encoded is obtained by decoding the reference image, ..., C xm To make C x(m-1) The first encoded data is obtained by decoding the first encoded data corresponding to the m-th first image to be encoded, which is used as a reference image.

[0386] Optionally, before step 1303, the video image processing method of this embodiment may further include: performing first encoding on a second image to be encoded to generate a fourth bitstream before performing first encoding on at least one first image to be encoded, and determining whether the prediction mode of the first encoding is intra-frame prediction. If the prediction mode of the first encoding is inter-frame prediction, then step 1303 is executed. If the prediction mode of the first encoding is intra-frame prediction, the fourth bitstream is used as the second bitstream. Thus, when performing first encoding on the second image to be encoded to obtain a first bitstream including P frames or P blocks, steps 1303 and 1304 can be executed to perform second encoding on the second image to be encoded to generate a second bitstream including frames in the full intra-frame prediction mode. When performing first encoding on the second image to be encoded to obtain a first bitstream including frames in the full intra-frame prediction mode, steps 1303 and 1304 can be skipped, and the frames in the full intra-frame prediction mode can be directly used as the second encoded data, thereby improving encoding efficiency.

[0387] For example, as mentioned above Figure 7 Taking another example of a long GOP bitstream and random access bitstream of video content, the encoding device performs a first encoding on the second image to be encoded to generate a long GOP bitstream including frame number 3 (frame #3). The encoding device determines whether the prediction mode of the first encoding is intra-frame prediction. Figure 7 As shown, the prediction mode for the first encoding at this point is inter-frame prediction. Subsequently, the encoding device can perform first encoding on the first image to be encoded by executing steps 1301 to 1304 to generate a long GOP bitstream including frame number 4 (#4 frame). Based on the reconstructed frame of frame number 4 (#4 frame) in the long GOP bitstream, the second image to be encoded is second encoded to generate a random access bitstream of frame number 3 (#3 frame). The prediction mode for frame number 3 (#3 frame) in the random access bitstream is intra-frame prediction.

[0388] It should be noted that in some embodiments, when performing first encoding on at least one first image to be encoded to obtain a first bitstream including frames in the full intra-frame prediction mode, the frames in the full intra-frame prediction mode may not be used as second encoded data. That is, the second bitstream may also not include the frames in the full intra-frame prediction mode, which can be reasonably set according to video transmission requirements.

[0389] It should also be noted that the aforementioned second encoded data can also be referred to as a random access frame.

[0390] Optionally, taking the encoding device applied to a server as an example, the server can store a first bitstream and a second bitstream. Upon receiving a video content request from a client, the server sends the first and second bitstreams to the client.

[0391] In this embodiment, at least one first image to be encoded is first encoded to generate a first bitstream. Based on at least one first reconstructed image, at least one of a first partitioning method or a first encoding parameter is determined for second encoding of a second image to be encoded. Based on at least one of the first partitioning method or the first encoding parameter, the second image to be encoded is second encoded to generate a second bitstream. The at least one first reconstructed image is either the first bitstream or a reconstructed image from the first encoding process. Thus, the encoding of the second bitstream is adjusted based on the encoding result of the first bitstream to ensure that the reconstructed frames of the first bitstream and the corresponding reconstructed frames of the second bitstream have the same or equivalent quality. This improves the decoding quality of the accessed video content while maintaining low latency access, reduces block artifacts, and eliminates some artifact effects.

[0392] The following explanation uses at least one first image to be encoded as a source video image to illustrate the video image processing method of this application embodiment.

[0393] Please see Figure 14 , Figure 14 This is a flowchart illustrating a video image processing method provided in an embodiment of this application. The method portion of this embodiment can be executed by an encoding device. The encoding device can be applied to the source device 12 in the above embodiments, for example, as... Figure 8 The server 801 in the illustrated embodiment. This embodiment uses at least one first image to be encoded as the (K+1)th frame image and the second image to be encoded as the Kth frame image as an example. The encoding device can adjust the second encoding of the Kth frame image based on the encoding result of the (K+1)th frame image to generate a first bitstream and a second bitstream of consistent quality. For example... Figure 14 The method shown may include the following implementation steps:

[0394] Step 1401: Perform first encoding on the K+1th frame image to generate the first bitstream.

[0395] The encoding device can use information such as prediction mode P, partitioning method D, and quantization parameter QP to perform first encoding on the (K+1)th frame image to generate a first bitstream. This first bitstream may include the first encoded data of the (K+1)th frame image. During the first encoding process, a reconstructed frame A of the (K+1)th frame image can also be generated.

[0396] Step 1402: Divide the Kth frame image into segments using a certain method, and perform second encoding of all image blocks in the full-frame prediction mode to obtain the reconstructed frame B of the second bitstream.

[0397] Using the Kth frame image and the reconstructed frame A as input information, we begin encoding to generate the second bitstream.

[0398] Optionally, the partitioning method D and quantization parameter QP involved in the first encoding can also be used as input information.

[0399] For example, the Kth frame image is divided in a certain way, and a second encoding process is performed on all the divided image blocks, including intra-frame prediction, transformation, quantization, inverse quantization, and inverse transformation, to obtain the reconstructed frame B of the second bitstream.

[0400] When this step 1402 is performed for the first time on the Kth frame image, the coding information and / or coding parameters used in the second coding process, such as partitioning, intra-frame prediction, transformation, and quantization, including, for example, partitioning method, QP, and bit rate, can be randomly generated coding information and / or coding parameters during initialization, or they can be the coding information and / or coding parameters used in the I-frames preceding the Kth frame image. For example, for QP, the average QP of one or more of the nearest I-frames preceding the Kth frame image can be used.

[0401] Reconstructed frame B can consist of reconstructed blocks of one or more coded image blocks.

[0402] Optionally, when the prediction mode P of the Kth frame image of the first bitstream is intra-frame prediction, the second encoded data of the Kth frame image of the second bitstream directly uses the first encoded data of the Kth frame image of the first bitstream. When the prediction mode P of the Kth frame image of the first bitstream is not intra-frame prediction (for example, the Kth frame image of the first bitstream is a P frame, or there is a P block), then in step 1302, the Kth frame image is encoded using the second encoding mode of intra-frame prediction.

[0403] Step 1403: Use the reconstructed frame B as the reference frame for decoding the K+1th frame of the first bitstream, and decode the first bitstream to obtain another reconstructed frame C of the K+1th frame.

[0404] Step 1404: Calculate the similarity cost function values ​​of reconstructed frame A and reconstructed frame C.

[0405] Specifically, the similarity cost function value f(partition method, QP) between reconstructed frame A and reconstructed frame C can be calculated according to the following formula (7).

[0406]

[0407] Where i represents the index of a pixel in the reconstructed frame, I represents the total number of pixels in the reconstructed frame, C(partition method, QP, i) represents the reconstructed pixel value at the i-th pixel position using the quantization parameter QP under one partition method of reconstructed frame C, and A(i) represents the reconstructed pixel value at the i-th pixel position of reconstructed frame A. T can be 1 or 2.

[0408] Optionally, in addition to formula (7), other methods can be used to evaluate the similarity between two reconstructed frames, including but not limited to MAD, SAD, SSD, MSD, SATD, etc.

[0409] Step 1405: Change the partitioning method and / or encoding parameters, and repeat step 1402.

[0410] This application embodiment may have multiple partitioning methods and / or encoding parameters. One partitioning method and / or encoding parameter may be selected from the multiple partitioning methods and / or encoding parameters, and step 1402 may be executed repeatedly to traverse the multiple partitioning methods and / or encoding parameters, encode the Kth frame image, and calculate the similarity cost function values ​​of the reconstructed frame C and the reconstructed frame A for each partitioning method and / or encoding parameter.

[0411] Taking the encoding parameter QP as an example, within a certain range (e.g., 0 to 51), QP is selected with a certain step size (e.g., 1 or 2), and step 1302 is repeated until the finite number of QPs is enumerated.

[0412] Taking the partitioning method as an example, for each partitioning method, execute step 1302 until all the finite partitioning methods have been enumerated.

[0413] Step 1406: Perform entropy encoding on the second encoded data, as well as the partitioning method and / or encoding parameters, to generate the second bitstream and complete the encoding of the Kth frame image.

[0414] The second bitstream may include the second encoded data of the Kth frame image.

[0415] The second encoded data here is the encoded data corresponding to the reconstructed frame C, which is the local optimum of the finite iteration. Specifically, the local optimum of the finite iteration means that the K-th frame image has been encoded using all partitioning methods and / or encoding parameters, and the similarity between the reconstructed frame C and the reconstructed frame A for each partitioning method and / or encoding parameter has been calculated. The one with the highest similarity is selected as the local optimum of the finite iteration, and the encoded data corresponding to the reconstructed frame C with the highest similarity is used as the second encoded data here.

[0416] Optionally, if the similarity cost function value between reconstructed frame A and reconstructed frame C is less than a similarity cost function value threshold, then the second encoded data here is the encoded data corresponding to reconstructed frame C. The encoded data corresponding to reconstructed frame C refers to the encoded data obtained by performing a second encoding on the Kth frame image using a partitioning method and / or encoding parameters. Decoding this encoded data yields reconstructed frame B. Using reconstructed frame B as a reference frame, decoding the first encoded data yields reconstructed frame C.

[0417] Optionally, in the process of comparing the similarity of two reconstructed frames, if there are individual pixels with large differences, such as a difference in grayscale values ​​exceeding 128, then this division method and the corresponding quantization result can be discarded.

[0418] After performing step 1406, step 1401 can be repeated to begin encoding the K+2 frame image.

[0419] In this embodiment, based on the encoding result of the first bitstream including the (K+1)th frame image, the second encoding of the Kth frame image is adjusted to ensure that the reconstructed frames of the first bitstream and the corresponding reconstructed frames of the second bitstream have the same or equivalent quality. This improves the decoding quality of the accessed video content while maintaining low latency, reduces block artifacts, and eliminates some artifacts. By simulating the decoding method at the decoding end and adjusting the encoding method of the second bitstream, the encoding-decoding inconsistency effect is reduced, which is beneficial for eliminating block artifacts.

[0420] The following explanation uses at least one first image to be encoded as multiple source video images to illustrate the video image processing method of this application embodiment.

[0421] Please see Figure 15 , Figure 15 This is a flowchart illustrating a video image processing method provided in an embodiment of this application. The method portion of this embodiment can be executed by an encoding device. The encoding device can be applied to the source device 12 in the above embodiments, for example, as... Figure 8The server 801 in the illustrated embodiment. This embodiment uses at least one first image to be encoded, including frame K+1, frame K+2, ..., frame K+m, and the second image to be encoded being frame K. The encoding device can adjust the second encoding of frame K based on the encoding results of frames K+1, K+2, ..., and frame K+m to generate a first bitstream and a second bitstream of consistent quality. For example... Figure 15 The method shown may include the following implementation steps:

[0422] Step 1501: Perform first encoding on the K+1 frame image, the K+2 frame image, ..., and the K+m frame image respectively to generate the first bitstream.

[0423] The encoding device can use information such as prediction mode P, partitioning method D, and quantization parameter QP to perform first encoding on the (K+1)th frame image, the (K+2)th frame image, ..., and the (K+m)th frame image, respectively, to generate a first bitstream including the first encoded data of the (K+1)th frame image, the (K+2)th frame image, ..., and the (K+m)th frame image. During the first encoding process, reconstructed frames A1, A2, ..., and A3 of the (K+1)th frame image can also be generated. m m is a positive integer greater than or equal to 2.

[0424] Step 1502: Divide the Kth frame image into segments using a specific method, and perform second encoding of all image blocks in the full-frame prediction mode to obtain the reconstructed frame B of the second bitstream.

[0425] The (K+1)th frame, the (K+2)th frame, ..., the (K+m)th frame, and the reconstructed frame A1 of the (K+1)th frame, the reconstructed frame A2 of the (K+2)th frame, ..., the reconstructed frame A of the (K+m)th frame. m As input information, encoding begins to generate the second bitstream.

[0426] Optionally, the partitioning method D and quantization parameter QP involved in the first encoding can also be used as input information.

[0427] For example, the Kth frame image is divided in a certain way, and a second encoding process is performed on all the sub-blocks after the division, including intra-frame prediction, transformation, quantization, inverse quantization, and inverse transformation, to obtain the reconstructed frame B of the second bitstream.

[0428] When this step 1502 is performed for the first time on the Kth frame image, the coding information and / or coding parameters used in the second coding process, such as partitioning, intra-frame prediction, transformation, and quantization, including, for example, partitioning method, QP, and bit rate, can be randomly generated coding information and / or coding parameters during initialization, or they can be the coding information and / or coding parameters used in the I-frames preceding the Kth frame image. For example, for QP, the average QP of one or more of the nearest I-frames preceding the Kth frame image can be used.

[0429] Reconstructed frame B can consist of reconstructed blocks of one or more coded image blocks.

[0430] Optionally, when the prediction mode P of the Kth frame image of the first bitstream is intra-frame prediction, the second encoded data of the Kth frame image of the second bitstream directly uses the first encoded data of the Kth frame image of the first bitstream. When the prediction mode P of the Kth frame image of the first bitstream is not intra-frame prediction (for example, the Kth frame image of the first bitstream is a P frame, or there is a P block), then in step 1502, the Kth frame image is encoded using the second encoding mode of intra-frame prediction.

[0431] Step 1503: Using the reconstructed frame B as the reference frame for decoding the (K+1)th frame of the first bitstream, decode the (K+1)th frame of the first bitstream to obtain another reconstructed frame C1 of the (K+1)th frame. Using the reconstructed frame C1 as the reference frame for decoding the (K+2)th frame of the first bitstream, decode the (K+2)th frame of the first bitstream to obtain another reconstructed frame C2 of the (K+2)th frame, and so on, reconstructing frame C... m As a reference frame for decoding the (K+m)th frame of the first bitstream, the (K+m)th frame of the first bitstream is decoded to obtain another reconstructed frame C of the (K+m)th frame. m .

[0432] Step 1504: Calculate reconstructed frame A1, reconstructed frame C1, reconstructed frame A2, reconstructed frame C2, ..., reconstructed frame A m and reconstructed frame C m The similarity cost function values ​​are weighted and summed.

[0433] Specifically, it can be calculated according to the following formula (8).

[0434]

[0435] Where i represents the index of a pixel in the reconstructed frame, N represents the total number of pixels in the reconstructed frame, and C m (Partition method, QP, i) represents the pixel value at the i-th pixel position of the reconstructed frame B, which is decoded using the partition method and quantization parameter QP as the reference frame in the first bitstream. m(i) represents the reconstructed pixel value at the i-th pixel position of the reconstructed frame generated by the first encoding. T can be 1 or 2, w m This represents the weighting coefficient for the similarity of the m-th reconstructed frame.

[0436] Optionally, different weighting coefficients can be selected based on the distance to the Mth frame image, and For example, for m=2, we can choose w1=0.6 and w2=0.4.

[0437] Optionally, for reconstructed frames A1 and C1, reconstructed frames A2 and C2... reconstructed frame A m and reconstructed frame C m The weighted sum of the similarity cost function values ​​can be evaluated not only by formula (8), but also by other methods, including but not limited to MAD, SAD, SSD, MSD, SATD, etc.

[0438] Step 1505: Change the partitioning method and / or encoding parameters, and repeat step 1502.

[0439] This application embodiment can have multiple partitioning methods and / or encoding parameters. One partitioning method and / or encoding parameter can be selected from the multiple partitioning methods and / or encoding parameters, and step 1402 can be repeatedly executed to traverse the multiple partitioning methods and / or encoding parameters, perform second encoding on the Kth frame image, and calculate the reconstructed frame C1, reconstructed frame C2, ..., and reconstructed frame C for each partitioning method and / or encoding parameter. m , and reconstructed frame A1, reconstructed frame A2, ..., and reconstructed frame A m The similarity.

[0440] Taking the encoding parameter QP as an example, within a certain range (e.g., 0 to 51), QP is selected with a certain step size (e.g., 1 or 2), and step 1502 is repeated until the finite number of QPs is enumerated.

[0441] Taking the partitioning method as an example, for each partitioning method, execute step 1402 until all the finite partitioning methods have been enumerated.

[0442] Step 1506: Perform entropy encoding on the second encoded data, as well as the partitioning method and / or encoding parameters, to generate the second bitstream and complete the encoding of the Kth frame image.

[0443] The second bitstream may include the second encoded data of the Kth frame image.

[0444] The second encoded data here refers to the reconstructed frames C1, C2, ..., C1, which are locally optimal in finite iterations. mThe corresponding encoded data. Finite iterative local optimum specifically refers to the situation where all partitioning methods and / or encoding parameters have been used to encode the Kth frame image, and the reconstructed frames C1, C2, ..., and C3 for each partitioning method and / or encoding parameter have been calculated. m With reconstructed frame A1, reconstructed frame A2, ..., and reconstructed frame A m The similarity is used to select the highest similarity as the local optimum in finite iterations. The reconstructed frames C1, C2, ..., C3 with the highest similarity are then used as the local optimum in finite iterations. m The corresponding encoded data serves as the second encoded data here. Reconstructed frame C1, reconstructed frame C2, ..., and reconstructed frame C... m The corresponding encoded data refers to the encoded data obtained by performing a second encoding on the Kth frame image using a certain partitioning method and / or encoding parameters. Decoding this encoded data yields the reconstructed frame B. Using the reconstructed frame B as a reference frame, decoding the (K+1)th frame of the first bitstream yields the reconstructed frame C1. Using the reconstructed frame C1 as a reference frame, decoding the (K+2)th frame of the first bitstream yields the reconstructed frame C2, and so on, yielding the reconstructed frame C... m .

[0445] Optionally, in the process of comparing the similarity of two reconstructed frames, if there are individual pixels with large differences, such as a difference in grayscale values ​​exceeding 128, then this division method and the corresponding quantization result can be discarded.

[0446] In this embodiment, based on the encoding results of a first bitstream including the (K+1)th frame, the (K+2)th frame, ..., the (K+m)th frame, the second encoding of the Kth frame is adjusted to ensure that the reconstructed frame of the Kth frame from the first bitstream has the same or comparable quality to the reconstructed frame of the Kth frame from the second bitstream. This improves the decoding quality of the accessed video content while maintaining low latency, reduces block artifacts, and eliminates some artifacts. By simulating the decoding method at the decoding end and adjusting the encoding method of the second bitstream, the encoding-decoding inconsistency effect is reduced, which is beneficial for eliminating block artifacts.

[0447] This application also provides the following embodiments, which are related to the above. Figure 9 Several other possible implementations of the illustrated embodiment will be specifically explained. For the second encoding process, in the following embodiment, based on the encoding information of the first encoding, a second encoding in full-frame intra-prediction mode is performed on the image to be encoded or the first reconstructed image to generate a second bitstream.

[0448] The encoding information of the first code may include one or more of the following: the division method of the first code, the quantization parameters of the first code, and the encoding distortion information of the first code.

[0449] One possible implementation, according to embodiments of this application, is to perform a second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode based on the encoding information of the first encoding. This may include one or more of the following: performing a second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode using the same partitioning method as the first encoding; or performing a second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode using the same quantization parameters as the first encoding; or determining the quantization parameters of the second encoding based on the encoding distortion information of the first encoding, and performing a second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode based on the quantization parameters of the second encoding.

[0450] (1) Division method

[0451] The first code can be divided into TU, PU, ​​or CU partitioning methods.

[0452] For example, since the distortion of the encoding mainly comes from quantization, the second encoding chooses to use the same TU partitioning method as the first encoding, for example, Figure 16 This is a schematic diagram illustrating that the first and second codes provided in this application use the same TU partitioning method. Both the first and second codes can use the following... Figure 16 The TU partitioning method shown is as follows. This effectively ensures the consistency of distortion boundaries, meaning that distortion exists at the same boundaries, which helps maintain the quality of the first and second bitstreams to be comparable or identical.

[0453] Optionally, in addition to the TU partitioning method of the first code, the PU or CU partitioning method of the first code can also be transmitted. The second code is controlled based on the PU or CU partitioning method of the first code. The PU partitioning method of the first code is used to quickly determine the prediction direction of the PU at the corresponding position in the second code. For example, if the PU of a certain coded block in the first code is intra, then the PU of the corresponding coded block in the second code can remain consistent with that of the first code; or, the PU of the second code does not cross the PU boundary of the corresponding position in the first code. The CU partitioning method of the first code can serve as a reference for the pre-partitioning of CUs in the second code. For example, the CUs of the second code are only partitioned within the CU range of the first code, ensuring that the CUs of the second code do not cross the CU boundaries of the first code.

[0454] (2) Quantization parameters

[0455] The input images for the first and second encoding are similar or identical. For example, the input for the first encoding is the image to be encoded, and the input for the second encoding is the image to be encoded and a first reconstructed image. The first reconstructed image is a reconstructed image of the image to be encoded using the first encoding. Therefore, the spatiotemporal complexity of the video inputs for the first and second encodings is similar or identical. To reduce computational complexity, the second encoding can utilize the quantization parameter distribution information of the first encoding. This quantization parameter distribution information incorporates the differences in human eye sensitivity to different spatiotemporal complexities to design the quantization parameters for each encoding block. To improve the quality of the second encoding, while ensuring the differences in quantization parameter distribution, a quantization parameter offset can be superimposed on the quantization parameters of each encoding block. `qp_offset` represents this quantization parameter offset. Figure 17 A schematic diagram of the quantization parameters of the first encoding and the second encoding provided for embodiments of this application, to illustrate... Figure 17 Taking the example shown, based on the quantization parameters of the first code and qp_offset (-3 as shown in the figure), the quantization parameters of the second code can be obtained. Specifically, the QP of each coded block of the first code can be subtracted by 3 to obtain the QP of each coded block of the second code. For example, as shown... Figure 17 The QP of the first row and first column of the first encoding shown is 32. Based on this, the QP of the first row and first column of the second encoding can be obtained as 29.

[0456] Optionally, the quantization parameter transfer unit size can be equal, for example... Figure 17 Each small square shown (i.e., quantization parameter transfer unit) is 16x16 or 64x64 pixels in size, meaning that every 16x16 or 64x64 pixel unit has the same QP value.

[0457] Optionally, the sizes of the quantization parameter transfer units can be unequal. Figure 18 A schematic diagram illustrating the quantization parameters transmitted from the first code to the second code in an embodiment of this application is shown below. Figure 18 As shown, the sizes of different small squares (i.e., quantization parameter transfer units) can be different.

[0458] Optionally, the quantization parameters used in each encoding quantization process are calculated from the quantization parameters of the parameter set (such as the syntax element init_qp_minus26 in PPS), the slice-level quantization parameter offset (such as slice_qp_delta in the slice head), and the quantization parameter offset of the current coding block (such as mb_delta_quant). Therefore, the quantization parameters passed from the first encoding to the second encoding include, but are not limited to, one or any combination of the quantization parameters in the parameter set, the slice-level quantization parameter offset, and the quantization parameter offset of the coding block, or the final quantization parameters used in the final first encoding quantization process.

[0459] (3) Encoding distortion information

[0460] During the second encoding process, the encoding distortion threshold of the second encoding can be determined based on the encoding distortion information of the first encoding. For example, for a certain encoding block of the first bitstream, the encoding distortion information uses the MAD index, which has a value of 4. Then, when making encoding decisions in the second encoding, if the MAD index between the predicted frame and the frame to be encoded, or between the reconstructed frame and the frame to be encoded, is less than 4, the decision-making process is terminated early, and the current encoding strategy is taken as the optimal encoding strategy.

[0461] Optionally, the encoding distortion information can use one or more of the commonly used metrics such as MAD, SAD, SSD, MSD, and SATD.

[0462] In another possible implementation, according to embodiments of this application, performing a second encoding of the image to be encoded or the first reconstructed image using a full intra-frame prediction mode based on the encoding information of the first encoding may include: determining quantization parameters for the second encoding based on the encoding information of the first encoding and feature information of the image to be encoded; and performing a second encoding of the image to be encoded or the first reconstructed image using the quantization parameters of the second encoding.

[0463] The feature information of the image to be encoded may include one or more of the following: the content complexity of the image to be encoded, the color classification information of the image to be encoded, the contrast information of the image to be encoded, and the content segmentation information of the image to be encoded.

[0464] (1) Content complexity

[0465] The image content inputs of the first encoding and the second encoding are similar or the same. For example, the input of the first encoding is the image to be encoded, and the input of the second encoding is the image to be encoded and the first reconstructed image. The first reconstructed image is the reconstructed image of the image to be encoded by the first encoding. Therefore, the spatiotemporal complexity of the video inputs of the first encoding and the second encoding are similar or the same. The content complexity analyzed in the first encoding process can be passed to the second encoding. The second encoding does not need to repeat the calculation. It can directly calculate the encoding parameters such as the quantization coefficient based on the content complexity to guide the second encoding to generate the second bitstream.

[0466] (2) Regional Information

[0467] Because the complexity of the content varies and the human eye's sensitivity to distortion differs, the second encoder can set different quantization parameter offsets (qp_offset) for different regions based on the region information of the first encoder. For example, a qp_offset of -5 can be used for complex regions, while a qp_offset of -3 can be used for simple regions. Of course, it is understandable that other values ​​can also be used, but these embodiments in this application will not be illustrated one by one.

[0468] Optionally, the second encoding can be adjusted to set different encoding parameters for different regions based on the encoding distortion information, color classification, contrast information, and content segmentation information of the first encoding.

[0469] The video image processing method of this application can complete the encoding of the first bitstream and the second bitstream through any of the above embodiments. In order to facilitate transmission and video decoding, the embodiments of this application can also carry the identification information of the first bitstream and the second bitstream through the following embodiments, so that the decoding device can distinguish the first bitstream and the second bitstream according to the identification information, decode the corresponding bitstream, and quickly access the video content.

[0470] If no random access is required, the encoding device can encapsulate and transmit the first bitstream. The decoding device can receive, decode, or display the first bitstream. If random access is required at a certain moment, the encoding device can select the second bitstream corresponding to the random access frame at that moment or the next moment for encapsulation and transmission. The encoding device can encapsulate and transmit the first bitstream at subsequent moments. The decoding device can first receive, decode, or display the second bitstream, then receive the first bitstream, and decode or display the first bitstream based on the reconstructed image from the second bitstream.

[0471] (1) Add bitstream identification information to parameter sets such as video parameter set (VPS), sequence parameter set (SPS), or picture parameter set (PPS).

[0472] The bitstream receiving device (e.g., a decoding device) can determine whether the received bitstream supports single-frame random access based on the identification information, and distinguish whether the received bitstream belongs to a long GOP stream or a basic stream (i.e., the first bitstream mentioned above), or contains two bitstreams (the first bitstream and the second bitstream).

[0473] Taking PPS as an example, as shown in Table 1:

[0474] Table 1 shows the addition of stream identifier information to the PPS parameter set.

[0475]

[0476] In Table 1, u(1) represents a 1-bit unsigned integer in the encoding standard, ue(v) represents Columbus code encoding, and information is added to the PPS parameter set to identify whether the current bitstream supports single-frame random access and the type of the current bitstream. The meanings of the syntax elements are as follows:

[0477] Single-frame random access enable flag (single_insert_enabled_flag): A value of 1 indicates that the bitstream supports single-frame random access, and a value of 0 indicates that the bitstream does not support single-frame random access.

[0478] Stream ID (stream_id): This value exists when single_insert_enabled_flag is 1.

[0479] When this value is 0, it indicates that the current bitstream is a long GOP bitstream / basic stream.

[0480] When this value is 1, it indicates that the current bitstream is a random access stream.

[0481] When the value is 2, it means that the current bitstream contains both long GOP bitstream data and random access bitstream data, and the long GOP bitstream data corresponding to the same video frame content (same PTS) is before the random access bitstream data. Figure 19 This is a schematic diagram illustrating the arrangement of the first and second bitstreams when the stream identifier (stream_id) is 2, as provided in this embodiment of the application. For example, as shown... Figure 19 As shown, the first stream may include long GOP frame 1 and long GOP frame 2, and the second stream may include random access frame 1 and random access frame 2. Long GOP frame 1 and random access frame 1 are the same video frame content, and long GOP frame 2 and random access frame 2 are the same video frame content. The stream identifier (stream_id) in PPS is 2, as shown below. Figure 19 As shown, long GOP frame 1 precedes random access frame 1, and long GOP frame 2 precedes random access frame 2.

[0482] When the value is 3, it means that the current bitstream contains both long GOP bitstream data and random access bitstream data, and the long GOP bitstream data corresponding to the same video frame content (same PTS) is after the random access bitstream data. Figure 20 This is a schematic diagram illustrating the arrangement of the first and second bitstreams when the stream identifier (stream_id) is 3, as provided in this embodiment of the application. For example, as shown... Figure 20 As shown, the first stream may include long GOP frame 1 and long GOP frame 2, and the second stream may include random access frame 1 and random access frame 2. Long GOP frame 1 and random access frame 1 are the same video frame content, and long GOP frame 2 and random access frame 2 are the same video frame content. The stream identifier (stream_id) in PPS is 3, as shown below. Figure 20 As shown, long GOP frame 1 follows random access frame 1, and long GOP frame 2 follows random access frame 2.

[0483] Optionally, information such as the single_insert_enabled_flag and stream_id can also be carried in the VPS or SPS.

[0484] Optionally, the single_insert_enabled_flag and the stream identifier can be combined into a single syntax element. When it is 0, it indicates that the general bitstream does not support single-frame random access; when it is 1, it indicates a long GOP stream; and when it is 2, it indicates a random access stream.

[0485] (2) Add bitstream identifier information to slice_segment_header

[0486] The bitstream receiving device (e.g., a decoding device) can determine whether the received bitstream supports single-frame random access based on the header information of each slice, and distinguish whether the received bitstream belongs to a long GOP stream or a basic stream. One way to carry the slice_segment_header is shown in Table 2:

[0487] Table 2 shows the addition of bitstream identification information to the slice_segment_header.

[0488]

[0489] In Table 2, the meanings of the syntax elements are as follows:

[0490] slice_support_single_insert_enable: A value of 1 indicates that the bitstream supports single-frame random access, while a value of 0 indicates that the bitstream does not support single-frame random access.

[0491] stream_id: This value exists when slice_support_single_insert_enable is 1.

[0492] When this value is 0, it indicates that the current bitstream is a long GOP bitstream;

[0493] When this value is 1, it indicates that the current bitstream is a random access stream;

[0494] Optionally, slice_support_single_insert_enable and slice_id can be combined into a single syntax element. When it is 0, it indicates a general bitstream that does not support single-frame random access; when it is 1, it indicates a long GOP stream; and when it is 2, it indicates a random access stream.

[0495] Optionally, during storage or transmission, the long GOP stream and the random access stream can be binary concatenated and combined into a single bitstream, and the above identification information can be used to distinguish between the long GOP stream and the random access stream. Figure 21 This application provides three methods for combining the first and second bitstreams into a single bitstream. Figure 21 (a) in the figure represents the two data arrangement forms in a single bitstream when the parameter sets of the long GOP stream and the random access stream are the same. Figure 21 In (b), VPS1, SPS1, and PPS1 indicate that the parameter set belongs to the long GOP stream, while VPS2, SPS2, and PPS2 indicate that the parameter set belongs to the random access stream. Each type of stream data can be directly decoded by placing the parameter set in front of it. Figure 21 In (c), the parameter sets of the two bitstreams are arranged together, which facilitates the advance transmission of parameter sets in some scenarios (such as DASH stream transmission). In this case, the slice_pic_parameter_set_id needs to be set with different values ​​in the slice_segment_header data of the long GOP stream and the random access stream. According to the standard protocol, the corresponding PPS parameter set can be found based on the slice_pic_parameter_set_id. The PPS parameter set points to the corresponding SPS through pps_seq_parameter_set_id, and the corresponding parameter set can be found in this way.

[0496] Optionally, the combined stream of the two streams may contain only one or more parameter sets from the VPS, SPS, or PPS in the long GOP stream or random access stream.

[0497] Depending on the specific circumstances (e.g., channel changes, user requests), the receiving or transmitting end selects one of two methods for encapsulating, transmitting, receiving, decoding, or displaying the bitstream data corresponding to long GOP frames and random access frames with the same POC value. If no random access is required, the bitstream data of the long GOP frame is selected for encapsulation, transmission, reception, decoding, or display; if random access is required, the bitstream data corresponding to the random access frame is selected for encapsulation, transmission, reception, decoding, or display.

[0498] Optionally, if the two bitstreams are combined into one bitstream, the long GOP stream data (including the parameter set) and the random access stream (including the parameter set) may not carry the distinguishing identifier of the long GOP stream or random access stream data type in the above VPS, SPS, PPS or slice_segment_header. If no scenario requiring random access occurs, the bitstream data of the long GOP frame can be encapsulated, sent, received, decoded or displayed. If a scenario requiring random access occurs, it is first determined whether the long GOP frame is an intra-frame prediction block. If so, the bitstream data of the long GOP frame is encapsulated, sent, received, decoded or displayed. Otherwise, the bitstream data corresponding to the random access frame is encapsulated, sent, received, decoded or displayed.

[0499] (3) Carry bitstream identification information in supplementary enhancement information (SEI).

[0500] Table 3 General SEI Information Syntax

[0501]

[0502] Sub-stream concatenation of SEI messages syntax

[0503]

[0504] Table 3 introduces a new type, 182, for the SEI type to represent single-frame access information of the current bitstream, adding the information `single_picture_info_insert(payloadSize)`. The meanings of the included syntax elements are as follows:

[0505] single_insert_enabled_flag: A value of 1 indicates that the stream supports random access in a single frame, while a value of 0 indicates that the stream does not support random access in a single frame.

[0506] stream_id: This value exists when single_insert_enabled_flag is 1.

[0507] When this value is 0, it indicates that the current bitstream is a long GOP bitstream / basic stream;

[0508] When this value is 1, it indicates that the current bitstream is a random access stream;

[0509] (4) Carry stream identification information in the stream encapsulation

[0510] Each sub-stream is encapsulated, and each sub-stream can be independently encapsulated in a track, such as a sub-picture track. Syntax descriptions indicating whether the sub-streams can be concatenated can be added to the sub-picture track, as shown in the following example:

[0511] Add the following syntax to the spco box:

[0512]

[0513] The semantics are as follows:

[0514] track_class: When it is 0, it indicates a general bitstream that does not support single-frame random access; when it is 1, it indicates a long GOP stream; when it is 2, it indicates a random access stream.

[0515] (5) Add description bitstream identifier information to the file format.

[0516] This embodiment adds descriptive bitstream type information to the file format specified by the ISO base media file format (ISOBMFF). In the file format, for long GOP streams and random access streams, a Sample Entry Type: 'srand' is added to the video track. When the sample entry name is 'normal', it indicates that the current video track contains a general bitstream that does not support single-frame random access; when the sample entry name is 'base', it indicates a long GOP stream; and when the sample entry name is 'insert', it indicates a random access stream.

[0517] (6) Carry stream identifier information in file description information

[0518] Stream identification information can be carried in file description information, such as in the media presentation description (MPD) file of the DASH protocol. This embodiment provides an example of describing stream type information in an MPD:

[0519]

[0520] In this example, a new EssentialProperty property, srand@value, is specified. The description of the srand@value property is shown in Table 4.

[0521] Table 4 describes the srand@value attribute in "urn:mpeg:dash:srand:2014".

[0522]

[0523] The semantics of the syntax elements are as follows:

[0524] file_class: When it is 0, it indicates a general bitstream that does not support single-frame random access; when it is 1, it indicates a long GOP stream; when it is 2, it indicates a random access stream.

[0525] (7) Carry stream identifier information in custom messages

[0526] The bitstream can be sent using a custom TLV(type, length, value) message pattern, in which case the bitstream identifier information can be carried in the type.

[0527] For example, a TLV message can include a type field, a length field, and a payload field. Type (8 bits): data type, length (32 bits): payload length, payload (variable length): bitstream data.

[0528] Table 5 shows that different types have different payloads.

[0529] Type Semantics Payload 0x00 General bitstreams that do not support random access to single frames General bitstream data 0x01 Long GOP Flow Long GOP bitstream data 0x02 Random access stream Random access stream data other reserve Bitstream or other data

[0530] Therefore, the encoding end of this application embodiment can carry the identification information of the first bitstream and the second bitstream in the bitstream, or in the encapsulation layer, or in the transport protocol layer, so that the decoding end can distinguish the first bitstream and the second bitstream based on the identification information and correctly decode the video content.

[0531] The video image processing method of the present application embodiment has been described in detail above with reference to the accompanying drawings. The following section will describe the method in conjunction with... Figure 22 The video image processing apparatus according to embodiments of this application will be described. It should be understood that the video image processing apparatus is capable of executing the video image processing method according to embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted in the following description of the video image processing apparatus according to embodiments of this application.

[0532] See Figure 22 , Figure 22 This is a schematic diagram of a video image processing apparatus provided in an embodiment of this application. Figure 22 As shown, the video image processing device 2200 may include: an acquisition module 2201, a first encoding module 2202, and a second encoding module 2203.

[0533] The acquisition module 2201 is used to acquire the image to be encoded. The first encoding module 2202 is used to perform a first encoding on the image to be encoded to generate a first bitstream. The second encoding module 2203 is used to perform a second encoding in full-frame prediction mode on the image to be encoded or the first reconstructed image according to the encoding information of the first encoding to generate a second bitstream, wherein the first reconstructed image is the first bitstream or the reconstructed image in the first encoding process.

[0534] In some embodiments, the encoding information of the first code includes one or more of the following: the division method of the first code, the quantization parameters of the first code, and the encoding distortion information of the first code.

[0535] In some embodiments, the second encoding module 2203 is configured to perform at least one of the following: perform a second encoding of the image to be encoded or the first reconstructed image in full intra-frame prediction mode using the same partitioning method as the first encoding; or perform a second encoding of the image to be encoded or the first reconstructed image in full intra-frame prediction mode using the same quantization parameters as the first encoding; or determine the quantization parameters of the second encoding based on the encoding distortion information of the first encoding, and perform a second encoding of the image to be encoded or the first reconstructed image in full intra-frame prediction mode based on the quantization parameters of the second encoding.

[0536] In some embodiments, the second encoding module 2203 is configured to: determine the quantization parameters of the second encoding based on the encoding information of the first encoding and the feature information of the image to be encoded; and perform second encoding of the image to be encoded or the first reconstructed image in full-frame prediction mode based on the quantization parameters of the second encoding.

[0537] In some embodiments, the feature information of the image to be encoded includes one or more of the following: content complexity of the image to be encoded, color classification information of the image to be encoded, contrast information of the image to be encoded, and content segmentation information of the image to be encoded.

[0538] In some embodiments, the second encoding module 2203 is configured to determine, based on the encoding information of the first encoding and the first reconstructed image, at least one of a first partitioning method or a first encoding parameter to be used for the second encoding of the image to be encoded or the first reconstructed image. The second encoding module 2203 is further configured to perform a second encoding of the image to be encoded or the first reconstructed image using an intra-frame prediction mode based on at least one of the first partitioning method or the first encoding parameter.

[0539] In the first bitstream, the interval between two adjacent intra-prediction modes is greater than the interval between two adjacent intra-prediction modes in the second bitstream.

[0540] In some embodiments, the difference between the first reconstructed image and the second reconstructed image is less than a difference threshold or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, and the second reconstructed image is a reconstructed image in the second bitstream or the second encoding process.

[0541] In some embodiments, the second encoding module 2203 is configured to: determine a plurality of second partitioning methods based on the encoding information of the first encoding and the first reconstructed image, and select one of the plurality of second partitioning methods as the first partitioning method; and / or, determine a plurality of second encoding parameters based on the encoding information of the first encoding and the first reconstructed image, and select one of the plurality of second encoding parameters as the first encoding parameter.

[0542] Among them, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and multiple third reconstructed images. The multiple third reconstructed images include the second reconstructed image. The multiple third reconstructed images are reconstructed images obtained by performing multiple second encoding processes on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters. Alternatively, the multiple third reconstructed images are reconstructed images of multiple third bitstreams. The multiple third bitstreams are obtained by performing multiple second encoding processes on the image to be encoded or the first reconstructed image according to multiple second partitioning methods and / or multiple second encoding parameters.

[0543] In some embodiments, the second encoding module 2203 is further configured to: obtain the prediction mode of the first encoding. When the prediction mode of the first encoding is inter-frame prediction, the step of obtaining the encoding information of the first encoding and performing a second encoding of the image to be encoded or the first reconstructed image in the full intra-frame prediction mode according to the encoding information of the first encoding to generate a second bitstream is performed. When the prediction mode of the first encoding is intra-frame prediction, the first bitstream is used as the second bitstream.

[0544] In some embodiments, the image to be encoded is a source video image; or, the image to be encoded is an image block obtained by dividing the source video image.

[0545] It should be noted that the video image processing device 2200 can perform... Figures 9 to 12 Any, or Figures 16 to 21 The method of the encoding apparatus in any of the illustrated embodiments. For specific implementation principles and technical effects, please refer to the detailed explanations of the above method embodiments; they will not be repeated here.

[0546] This application embodiment also provides another video image processing apparatus, employing... Figure 22 The processing apparatus shown has the same structure. The acquisition module is used to acquire at least one first image to be encoded and a second image to be encoded, wherein the second image to be encoded is a video image preceding the at least one first image to be encoded. The first encoding module is used to perform a first encoding on each of the at least one first image to be encoded to generate a first bitstream. The second encoding module is used to determine, based on at least one first reconstructed image, at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the second image to be encoded, wherein the at least one first reconstructed image is either the first bitstream or a reconstructed image during the first encoding process. The second encoding module is further used to perform a second encoding on the second image to be encoded based on at least one of the first partitioning method or the first encoding parameter to generate a second bitstream.

[0547] In the first bitstream, the interval between two adjacent intra-prediction modes is greater than the interval between two adjacent intra-prediction modes in the second bitstream.

[0548] In some embodiments, the number of at least one first image to be encoded is one, the number of at least one first reconstructed image is one, the difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, the second reconstructed image is obtained by decoding the first bitstream using the third reconstructed image as a reference image, and the third reconstructed image is the second bitstream or the reconstructed image in the second encoding process.

[0549] In some embodiments, the number of at least one first image to be encoded is one, the number of at least one first reconstructed image is one, and the second encoding module is configured to: select one second partitioning method as the first partitioning method from a plurality of second partitioning methods based on the first reconstructed image; and / or, select one second encoding parameter as the first encoding parameter from a plurality of second encoding parameters based on the first reconstructed image.

[0550] Among them, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the multiple fourth reconstructed images. The multiple fourth reconstructed images include the second reconstructed image. The multiple fourth reconstructed images are obtained by decoding the first bitstream using multiple fifth reconstructed images as reference images. The multiple fifth reconstructed images are reconstructed images of multiple third bitstreams. The multiple third bitstreams are obtained by performing multiple second encodings on the second image to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters. Alternatively, the multiple fifth reconstructed images are reconstructed images during the process of performing multiple second encodings on the second image to be encoded according to multiple second partitioning methods and / or multiple second encoding parameters.

[0551] In some embodiments, the first encoding module is further configured to: perform a first encoding on the second image to be encoded before performing a first encoding on at least one first image to be encoded, to generate a fourth bitstream. The second encoding module is further configured to: obtain the prediction mode of the first encoding. When the prediction mode of the first encoding is inter-frame prediction, the step of determining at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the second image to be encoded based on at least one first reconstructed image is performed. When the prediction mode of the first encoding is intra-frame prediction, the fourth bitstream is used as the second bitstream.

[0552] In some embodiments, at least one first image to be encoded is at least one first source video image, and the second image to be encoded is a second source video image.

[0553] It should be noted that the video image processing device can perform... Figures 13 to 15 The method of the encoding apparatus in any of the illustrated embodiments. For specific implementation principles and technical effects, please refer to the detailed explanations of the above method embodiments; they will not be repeated here.

[0554] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0555] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically addressed to non-temporary tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0556] Instructions can be executed by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the various illustrative logic blocks, modules, and steps described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.

[0557] The technology of this application can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in this application are intended to emphasize functional aspects of the apparatus for performing the disclosed technology, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided via interoperable hardware units (containing one or more processors as described above).

[0558] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0559] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing video images, characterized in that, include: Obtain the image to be encoded; The image to be encoded is first encoded to generate a first bitstream; Based on the encoding information of the first encoding, the image to be encoded or the first reconstructed image is subjected to a second encoding in full-frame prediction mode to generate a second bitstream. The second bitstream is a random access bitstream, and the first reconstructed image is the first bitstream or the reconstructed image in the first encoding process.

2. The method according to claim 1, characterized in that, The encoding information of the first code includes one or more of the following: the division method of the first code, the quantization parameters of the first code, and the encoding distortion information of the first code.

3. The method according to claim 2, characterized in that, The second encoding of the image to be encoded or the first reconstructed image using the encoding information of the first encoding includes at least one of the following: The image to be encoded or the first reconstructed image is then encoded using the same partitioning method as the first encoding; or... A second encoding of the image to be encoded or the first reconstructed image using the same quantization parameters as the first encoding is performed using intra-frame prediction mode; or... Based on the quantization parameters and quantization parameter offset of the first encoding, the quantization parameters of the second encoding are determined, and based on the quantization parameters of the second encoding, the second encoding of the image to be encoded or the first reconstructed image is performed in the intra-frame prediction mode. or, Based on the encoding distortion information of the first encoding, the quantization parameters of the second encoding are determined, and based on the quantization parameters of the second encoding, the second encoding of the image to be encoded or the first reconstructed image is performed in the intra-frame prediction mode.

4. The method according to claim 1 or 2, characterized in that, The step of performing a second encoding of the image to be encoded or the first reconstructed image in a full-frame intra-prediction mode based on the encoding information of the first encoding includes: The quantization parameters of the second encoding are determined based on the encoding information of the first encoding and the feature information of the image to be encoded. Based on the quantization parameters of the second encoding, the image to be encoded or the first reconstructed image is subjected to a second encoding in the intra-frame prediction mode.

5. The method according to claim 4, wherein the feature information of the image to be encoded includes one or more of the following: the content complexity of the image to be encoded, the color classification information of the image to be encoded, the contrast information of the image to be encoded, and the content segmentation information of the image to be encoded.

6. The method according to claim 1 or 2, characterized in that, The step of performing a second encoding of the image to be encoded or the first reconstructed image in a full-frame intra-prediction mode based on the encoding information of the first encoding includes: Based on the encoding information of the first encoding and the first reconstructed image, determine at least one of the first partitioning method or the first encoding parameter used for the second encoding of the image to be encoded or the first reconstructed image; According to at least one of the first partitioning method or the first encoding parameter, the image to be encoded or the first reconstructed image is subjected to a second encoding in the intra-frame prediction mode.

7. The method according to claim 6, characterized in that, The difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, and the second reconstructed image is the second bitstream or a reconstructed image in the second encoding process.

8. The method according to claim 6, characterized in that, The step of determining at least one of the first partitioning method or first encoding parameters for performing second encoding on the image to be encoded or the first reconstructed image based on the encoding information of the first encoding and the first reconstructed image includes: Based on the encoding information of the first encoding and the first reconstructed image, multiple second partitioning methods are determined, and one of the multiple second partitioning methods is selected as the first partitioning method; and / or, Based on the encoding information of the first encoding and the first reconstructed image, a plurality of second encoding parameters are determined, and one of the plurality of second encoding parameters is selected as the first encoding parameter; Wherein, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the plurality of third reconstructed images, the plurality of third reconstructed images including the second reconstructed image, the plurality of third reconstructed images being reconstructed images obtained by performing the second encoding process on the image to be encoded or the first reconstructed image multiple times according to the plurality of second partitioning methods and / or the plurality of second encoding parameters, or the plurality of third reconstructed images being reconstructed images of multiple third bitstreams, the plurality of third bitstreams being obtained by performing the second encoding process on the image to be encoded or the first reconstructed image multiple times according to the plurality of second partitioning methods and / or the plurality of second encoding parameters.

9. The method according to any one of claims 6-8, characterized in that, The first encoding parameter includes quantization parameters or bit rate.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Obtain the prediction pattern of the first encoding; When the prediction mode of the first encoding is inter-frame prediction, the steps of obtaining the encoding information of the first encoding and performing second encoding in the full intra-frame prediction mode on the image to be encoded or the first reconstructed image according to the encoding information of the first encoding are executed to generate the second bitstream. When the prediction mode of the first encoding is intra-frame prediction, the first bitstream is used as the second bitstream.

11. The method according to any one of claims 1-10, characterized in that, The image to be encoded is a source video image, or the image to be encoded is an image block after the source video image has been divided.

12. A method for processing video images, characterized in that, include: Acquire at least one first image to be encoded and a second image to be encoded, wherein the second image to be encoded is a video image preceding the at least one first image to be encoded; The at least one first image to be encoded is respectively subjected to a first encoding to generate a first bitstream; Based on at least one first reconstructed image, determine at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the second image to be encoded, wherein the at least one first reconstructed image is the first bitstream or a reconstructed image in the first encoding process; The second image to be encoded is subjected to the second encoding in full-frame prediction mode according to at least one of the first division method or the first encoding parameters to generate a second bitstream, wherein the second bitstream is a random access bitstream.

13. The method according to claim 12, characterized in that, The number of the at least one first image to be encoded is one, the number of the at least one first reconstructed image is one, the difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, the second reconstructed image is obtained by decoding the first bitstream using the third reconstructed image as a reference image, and the third reconstructed image is the second bitstream or the reconstructed image in the second encoding process.

14. The method according to claim 12, characterized in that, The number of the at least one first image to be encoded is one, the number of the at least one first reconstructed image is one, and the step of determining at least one of the first partitioning method or first encoding parameters for performing the second encoding on the second image to be encoded based on the at least one first reconstructed image includes: Based on the first reconstructed image, one of the multiple second segmentation methods is selected as the first segmentation method; and / or, Based on the first reconstructed image, one second coding parameter is selected from a plurality of second coding parameters as the first coding parameter; Wherein, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the plurality of fourth reconstructed images, the plurality of fourth reconstructed images including the second reconstructed image, the plurality of fourth reconstructed images being obtained by decoding the first bitstream using the plurality of fifth reconstructed images as reference images respectively, the plurality of fifth reconstructed images being reconstructed images of the plurality of third bitstreams, the plurality of third bitstreams being obtained by performing the second encoding multiple times on the second image to be encoded according to the plurality of second partitioning methods and / or the plurality of second encoding parameters respectively, or, the plurality of fifth reconstructed images being reconstructed images during the process of performing the second encoding multiple times on the second image to be encoded according to the plurality of second partitioning methods and / or the plurality of second encoding parameters respectively.

15. The method according to any one of claims 12-14, characterized in that, Before performing a first encoding on each of the at least one first image to be encoded, the method further includes: The second image to be encoded is subjected to the first encoding to generate a fourth bitstream; Obtain the prediction pattern of the first encoding; When the prediction mode of the first encoding is inter-frame prediction, the step of determining at least one of the first partitioning method or the first encoding parameter for the second encoding of the second image to be encoded based on at least one first reconstructed image is executed. When the prediction mode of the first encoding is intra-frame prediction, the fourth bitstream is used as the second bitstream.

16. The method according to any one of claims 12-15, characterized in that, The at least one first image to be encoded is at least one first source video image, and the second image to be encoded is a second source video image.

17. The method according to any one of claims 12-16, characterized in that, The first encoding parameter includes quantization parameters or bit rate.

18. A video image processing apparatus, characterized in that, include: The acquisition module is used to acquire the image to be encoded. The first encoding module is used to perform a first encoding on the image to be encoded in order to generate a first bitstream; The second encoding module is used to perform second encoding of the image to be encoded or the first reconstructed image in full-frame prediction mode according to the encoding information of the first encoding, so as to generate a second bitstream, wherein the second bitstream is a random access bitstream, and the first reconstructed image is the first bitstream or the reconstructed image in the first encoding process.

19. The apparatus according to claim 18, characterized in that, The encoding information of the first code includes one or more of the following: the division method of the first code, the quantization parameters of the first code, and the encoding distortion information of the first code.

20. The apparatus according to claim 19, characterized in that, The second encoding module is used to perform at least one of the following: The image to be encoded or the first reconstructed image is then encoded using the same partitioning method as the first encoding; or... A second encoding of the image to be encoded or the first reconstructed image using the same quantization parameters as the first encoding is performed using intra-frame prediction mode; or... Based on the quantization parameters and quantization parameter offset of the first encoding, the quantization parameters of the second encoding are determined, and based on the quantization parameters of the second encoding, the second encoding of the image to be encoded or the first reconstructed image is performed in the intra-frame prediction mode. or, Based on the encoding distortion information of the first encoding, the quantization parameters of the second encoding are determined, and based on the quantization parameters of the second encoding, the second encoding of the image to be encoded or the first reconstructed image is performed in the intra-frame prediction mode.

21. The apparatus according to claim 18 or 19, characterized in that, The second encoding module is used for: The quantization parameters of the second encoding are determined based on the encoding information of the first encoding and the feature information of the image to be encoded. Based on the quantization parameters of the second encoding, the image to be encoded or the first reconstructed image is subjected to a second encoding in the intra-frame prediction mode.

22. The apparatus according to claim 21, wherein the feature information of the image to be encoded includes one or more of the following: the content complexity of the image to be encoded, the color classification information of the image to be encoded, the contrast information of the image to be encoded, and the content segmentation information of the image to be encoded.

23. The apparatus according to claim 18 or 19, wherein the second encoding module is configured to determine, based on the encoding information of the first encoding and the first reconstructed image, at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the image to be encoded or the first reconstructed image; The second encoding module is further configured to perform a second encoding of the image to be encoded or the first reconstructed image in an intra-frame prediction mode according to at least one of the first partitioning method or the first encoding parameters.

24. The apparatus according to claim 23, characterized in that, The difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, and the second reconstructed image is the second bitstream or a reconstructed image in the second encoding process.

25. The apparatus according to claim 23, characterized in that, The second encoding module is used for: Based on the encoding information of the first encoding and the first reconstructed image, a plurality of second partitioning methods are determined, and one of the plurality of second partitioning methods is selected as the first partitioning method; And / or, Based on the encoding information of the first encoding and the first reconstructed image, a plurality of second encoding parameters are determined, and one of the plurality of second encoding parameters is selected as the first encoding parameter; Wherein, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the plurality of third reconstructed images, the plurality of third reconstructed images including the second reconstructed image, the plurality of third reconstructed images being reconstructed images obtained by performing the second encoding process on the image to be encoded or the first reconstructed image multiple times according to the plurality of second partitioning methods and / or the plurality of second encoding parameters, or the plurality of third reconstructed images being reconstructed images of multiple third bitstreams, the plurality of third bitstreams being obtained by performing the second encoding process on the image to be encoded or the first reconstructed image multiple times according to the plurality of second partitioning methods and / or the plurality of second encoding parameters.

26. The apparatus according to any one of claims 23-25, characterized in that, The first encoding parameter includes quantization parameters or bit rate.

27. The apparatus according to any one of claims 18-26, characterized in that, The second encoding module is also used for: Obtain the prediction pattern of the first encoding; When the prediction mode of the first encoding is inter-frame prediction, the steps of obtaining the encoding information of the first encoding and performing second encoding in the full intra-frame prediction mode on the image to be encoded or the first reconstructed image according to the encoding information of the first encoding are executed to generate the second bitstream. When the prediction mode of the first encoding is intra-frame prediction, the first bitstream is used as the second bitstream.

28. The apparatus according to any one of claims 18-27, characterized in that, The image to be encoded is a source video image, or the image to be encoded is an image block after the source video image has been divided.

29. A video image processing apparatus, characterized in that, include: The acquisition module is used to acquire at least one first image to be encoded and a second image to be encoded, wherein the second image to be encoded is a video image preceding the at least one first image to be encoded; The first encoding module is used to perform a first encoding on the at least one first image to be encoded to generate a first bitstream; The second encoding module is used to determine, based on at least one first reconstructed image, at least one of a first partitioning method or a first encoding parameter for performing a second encoding on the second image to be encoded, wherein the at least one first reconstructed image is the first bitstream or a reconstructed image in the first encoding process; The second encoding module is further configured to perform the second encoding of the second image to be encoded in the full-frame prediction mode according to at least one of the first division method or the first encoding parameters, so as to generate a second bitstream, wherein the second bitstream is a random access bitstream.

30. The apparatus according to claim 29, characterized in that, The number of the at least one first image to be encoded is one, the number of the at least one first reconstructed image is one, the difference between the first reconstructed image and the second reconstructed image is less than a difference threshold, or the similarity between the first reconstructed image and the second reconstructed image is higher than a similarity threshold, the second reconstructed image is obtained by decoding the first bitstream using the third reconstructed image as a reference image, and the third reconstructed image is the second bitstream or the reconstructed image in the second encoding process.

31. The apparatus according to claim 29, characterized in that, The number of the at least one first image to be encoded is one, the number of the at least one first reconstructed image is one, and the second encoding module is used for: Based on the first reconstructed image, one of the multiple second segmentation methods is selected as the first segmentation method; and / or, Based on the first reconstructed image, one second coding parameter is selected from a plurality of second coding parameters as the first coding parameter; Wherein, the similarity between the first reconstructed image and the second reconstructed image is the highest among the similarities between the first reconstructed image and the plurality of fourth reconstructed images, the plurality of fourth reconstructed images including the second reconstructed image, the plurality of fourth reconstructed images being obtained by decoding the first bitstream using the plurality of fifth reconstructed images as reference images respectively, the plurality of fifth reconstructed images being reconstructed images of the plurality of third bitstreams, the plurality of third bitstreams being obtained by performing the second encoding multiple times on the second image to be encoded according to the plurality of second partitioning methods and / or the plurality of second encoding parameters respectively, or, the plurality of fifth reconstructed images being reconstructed images during the process of performing the second encoding multiple times on the second image to be encoded according to the plurality of second partitioning methods and / or the plurality of second encoding parameters respectively.

32. The apparatus according to any one of claims 29-31, characterized in that, The first encoding module is further configured to: perform the first encoding on the second image to be encoded before performing the first encoding on the at least one first image to be encoded, so as to generate a fourth bitstream; The second encoding module is further configured to: obtain the prediction pattern of the first encoding; When the prediction mode of the first encoding is inter-frame prediction, the step of determining at least one of the first partitioning method or the first encoding parameter for the second encoding of the second image to be encoded based on at least one first reconstructed image is executed. When the prediction mode of the first encoding is intra-frame prediction, the fourth bitstream is used as the second bitstream.

33. The apparatus according to any one of claims 29-32, characterized in that, The at least one first image to be encoded is at least one first source video image, and the second image to be encoded is a second source video image.

34. The apparatus according to any one of claims 29-33, characterized in that, The first encoding parameter includes quantization parameters or bit rate.

35. A video image processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-17.

36. A computer-readable storage medium, characterized in that, It includes a computer program and a bitstream, wherein the computer program, when executed by a processor, implements the video image processing method as described in any one of claims 1-17 to generate the bitstream.

37. A computer program product, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the video image processing method as described in any one of claims 1-17.

38. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on a computer, cause the computer to perform a video image processing method as described in any one of claims 1-17.

Images