Encoding, packaging and display methods, devices and electronic equipment

Abstract

This application provides an encoding, encapsulation, and display method, apparatus, and electronic device. The encoding method includes: acquiring an image to be encoded; then, determining the encoding scale N (N is an integer greater than 1) for the image, and determining the encoding parameters corresponding to each of the N encoding scales, resulting in N sets of encoding parameters. Subsequently, based on the N sets of encoding parameters, the image is encoded N times using a pre-set single-scale encoder to obtain N sets of bitstream data. In this way, encoding the image yields bitstream data with a smaller data volume (i.e., low-encoding-scale bitstream data), thus increasing the probability of the image bitstream data reaching the decoding end during network fluctuations, thereby ensuring smooth video playback. For high-encoding-scale bitstream data, if lost, it will not affect normal decoding and playback at the playback end, and will not cause screen tearing; if it reaches the decoding end, it can also improve the playback quality.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of communication, and in particular, to an encoding, packaging and display method and device and electronic equipment. BACKGROUND

[0002] Currently, services such as video conferencing, video telephony, online education, etc. have requirements not only on the quality and smoothness of the video, but also on the end-to-end delay to ensure that the receiving end can play the video smoothly.

[0003] In actual scenarios, network fluctuations are the norm; and network fluctuations will inevitably affect the smoothness of video playback and reduce user experience. Therefore, how to solve the poor video playback smoothness caused by network fluctuations is one of the problems to be solved. SUMMARY

[0004] To solve the above technical problems, the present application provides an encoding, packaging and display method, device and electronic equipment to solve the problem of video playback freezing caused by network fluctuations and improve the smoothness of video playback.

[0005] In a first aspect, an encoding method is provided, which comprises: obtaining an image to be encoded. Then, determining an encoding scale N (N is an integer greater than 1) for the image, and determining N sets of encoding parameters corresponding to the N encoding scales, wherein each encoding scale corresponds to a set of encoding parameters. Subsequently, according to the N sets of encoding parameters, using a pre-set single-scale encoder to encode the image N times to obtain N sets of code stream data. In this way, the image is encoded to obtain code stream data with a smaller data volume (i.e. code stream data with a low encoding scale), and thus when the network fluctuates, the code stream data of the image can have a higher probability of reaching the decoding end, thereby ensuring the smoothness of video playback. For code stream data with a high encoding scale, if it is lost, it will not affect the normal decoding and playback of the playback end and will not produce a screen full of flowers; if it can reach the decoding end, it can also improve the quality of the playback.

[0006] Exemplarily, the single-scale encoder can refer to an encoder that can only encode an image with one encoding scale and only output code stream data of one encoding scale, such as an H.264 encoder, an H.265 encoder, or an H.266 encoder, an AVS3 encoder, an AV1 encoder, etc., and similar single-scale encoders that can be realized in the future according to the development of encoding technology, and the present application embodiments do not limit this. Exemplarily, the single-scale encoder can be a software encoder or a hardware encoder, and the present application embodiments do not limit this.

[0007] The image can be a frame or a field. When a video signal is sampled, if it is sampled by progressive scanning, a frame image is obtained. When a video signal is sampled, if it is sampled by interlaced scanning (odd and even lines), a frame image is divided into two fields. The embodiments of the present application are exemplarily described by taking the collected image as a frame.

[0008] The encoded image can be a complete frame or a partial region in the frame, such as an image block obtained by dividing the frame in the encoding process, a local region with rich texture information, a local region of interest of a user, or other key regions.

[0009] According to the first aspect, the method further comprises: determining that the image needs to be encoded in multiple scales according to whether a first preset condition is met, the first preset condition being related to a coding importance of the image pair to other images in the video data, and / or a second preset condition being met, the second preset condition being related to an impact of the image pair corresponding to the code stream data on the network, and / or a third preset condition being met, the third preset condition being related to a predicted image quality of the image. When it is determined that the image needs to be encoded in multiple scales, the encoding scale N of the image is determined. In this way, by combining the coding importance of the code stream data corresponding to each frame image, the impact of the code stream data on the network, and the image quality after the image is encoded, the image in the video stream that needs to be encoded in multiple scales can be dynamically and accurately determined.

[0010] Exemplarily, when it is determined that the image does not need to be encoded in multiple scales, the image can be encoded in a single scale. In this way, the calculation amount of the encoder can be reduced, and the encoding efficiency can be improved.

[0011] According to the first aspect, or any one of the implementation forms of the first aspect, the first preset condition comprises at least one of the following: the image is an I frame or an IDR frame; the image is a scene switching frame; the image is a reference frame; and the image is a key frame. In this way, the image important for coding can be encoded in multiple scales, and when the code stream data of the minimum encoding scale of the image important for coding reaches the decoding end, the other frames in the video data can be successfully decoded, thereby ensuring the smoothness of the playback. The image unimportant for coding is not encoded in multiple scales, and if the corresponding code stream data is lost, the influence on the image reconstruction of other frames is small, and the influence on the user watching is also small.

[0012] Exemplarily, the I frame and the IDR frame both refer to a frame encoded by using an intra-frame encoding mode.

[0013] Exemplarily, the reference frame refers to a frame referenced by the encoder for encoding a certain frame, or a frame referenced by the decoder for image reconstruction of a certain frame.

[0014] For example, the key frame refers to a frame that is referenced by more than a preset number threshold. The preset number threshold can be set according to requirements, and embodiments of the present application do not limit this.

[0015] For example, the scene switching frame is the first frame image of a scene after a scene is switched.

[0016] For example, the non-reference frame refers to a frame other than the reference frame.

[0017] According to the first aspect, or any one of the implementations of the first aspect, the second preset condition includes at least one of the following: the data amount of the image corresponding to the code stream data is greater than a preset data amount threshold; and the data amount of the image corresponding to the code stream data is greater than the network transmission capability. In this way, the image with a greater impact on the network is subjected to multi-scale coding, and code stream data with a smaller data amount can be obtained, and thus, when the network fluctuates, the success rate of the image code stream data reaching the decoding end can be improved, and the smoothness of the video playback at the decoding end can be ensured.

[0018] For example, the data amount of the image corresponding to the code stream data is the data amount of the code stream data after the image is coded by the single-scale coder.

[0019] For example, the data amount of the image corresponding to the code stream data is the predicted data amount of the code stream data obtained by analyzing the image.

[0020] According to the first aspect, or any one of the implementations of the first aspect, the predicted image quality of the image includes a first predicted image quality and a second predicted image quality, the first predicted image quality is the predicted image quality of the predicted image coded at a source resolution of the image, and the second predicted image quality is the predicted image quality of the predicted image coded at a resolution lower than the source resolution; and the third preset condition includes that the first predicted image quality is lower than or equal to the second predicted image quality. In this way, the blocking effect at a low code rate can be reduced, and the image quality can be improved. Moreover, for coding of a low-resolution image, the computational amount of the coder can be reduced, and the coding efficiency can be improved. In addition, coding of the low-resolution image can reduce the code rate of the code stream data, and further improve the success rate of the code stream data transmission to the decoding end.

[0021] According to the first aspect, or any one of the implementations of the first aspect, the coding parameter includes at least one of the following: a resolution, a quantization parameter, reference frame information, and a target code rate.

[0022] For example, the resolution refers to the resolution of the image input to the coder.

[0023] For example, the quantization parameter refers to an encoding quantization parameter of the image coded by the encoder. The quantization parameter determines the quality distortion of the coded image. The greater the quantization parameter, the greater the distortion of the coded image relative to the source image. The smaller the quantization parameter, the more serious the distortion of the coded image relative to the source image.

[0024] For example, the reference frame information is used to indicate the reference frame. The reference frame information can be the sequence number (also called POC (Picture Order Count)) of the reference frame entering the encoder, or the distance between the reference frame and the current coded image (the distance is the difference between the sequence number of the reference frame entering the encoder and the sequence number of the current coded image entering the encoder).

[0025] For example, the sequence number of the reference frame entering the encoder can be determined according to the order of the reference frame entering the encoder. For example, two images are collected, the coding scale N = 2 for the first image, and the coding scale N = 2 for the second image. After the first image and the first set of coding parameters corresponding to the first image are input into the encoder, the first image is input to the encoder, and the POC of this image is 1. After the first image and the second set of coding parameters corresponding to the first image are input into the encoder, the second image is input to the encoder, and the POC of this image is 2. After the second image and the first set of coding parameters corresponding to the second image are input into the encoder, the third image is input to the encoder, and the POC of this image is 3, and so on.

[0026] For example, the target code rate refers to the expected code rate of the coded image.

[0027] According to the first aspect, or any one of the implementation manners of the above first aspect, the coding scale N for the image is determined according to the image type of the image and a pre-set correspondence between the image type and the coding scale. In this way, the coding scale can be determined according to the importance of the image to the coding and decoding. For an image with high importance, more coding scales can be set, and the success rate of transmitting more high coding scale code stream data to the decoding end can be improved, thereby improving the quality of other frames that rely on the image with high importance for decoding.

[0028] For example, the image type can refer to the frame type.

[0029] According to the first aspect, or any one of the implementation manners of the above first aspect, the coding scale N for the image is determined according to the coding scale pre-set for each image in the video data to which the image belongs. In this way, the efficiency of determining the coding scale N can be improved, thereby shortening the time length of the code stream data of the image reaching the decoding end.

[0030] According to the first aspect, or any one of the implementations of the first aspect, determining the N sets of encoding parameters corresponding to the N encoding scales comprises: determining the N sets of encoding parameters according to the encoding scale N and a parameter value range of the encoder. In this way, the encoding parameters can be adapted to the parameters of different single-scale encoders.

[0031] According to the first aspect, or any one of the implementations of the first aspect, determining the N sets of encoding parameters corresponding to the N encoding scales comprises:

[0032] According to the first aspect, or any one of the implementations of the first aspect, determining the N sets of encoding parameters corresponding to the N encoding scales comprises: determining the N sets of encoding parameters according to the encoding scale N and a parameter value range of the encoder. In this way, the encoding parameters can be adapted to the parameters of different single-scale encoders.

[0033] According to the first aspect, or any one of the implementations of the first aspect, the encoding parameters comprise reference frame information, the reference frame information is determined according to the first feedback information when the first feedback information is received, the first feedback information indicates that the encoding frame corresponding to the transmitted code stream data is successfully received, the encoding frame is confirmed as a reference frame, and the reference frame information is used to indicate the reference frame. In this way, the encoding frame successfully transmitted to the decoding end can be used as a reference frame, and the success rate of image decoding can be ensured. In addition, when the code stream data of a certain encoding frame is lost, only the decoding and display of the encoding frame are affected, and the decoding and display of other encoding frames are not affected, thereby limiting the influence of data loss on decoding and display.

[0034] Exemplarily, the encoding frame can refer to each frame of image entering the encoder.

[0035] According to the first aspect, or any one of the implementations of the first aspect, the encoding parameters comprise reference frame information, and the reference frame information can be pre-set.

[0036] According to the first aspect, or any one of the implementations of the first aspect, the encoding parameters comprise resolution, when the encoder supports multi-resolution encoding, the i-th encoding in the N times of encoding of the image is performed according to the N sets of encoding parameters and the pre-set single-scale encoder, comprising: when the resolution in the i-th set of encoding parameters is equal to the source resolution of the image, the i-th set of encoding parameters and the image are input into the pre-set single-scale encoder for encoding; when the resolution in the i-th set of encoding parameters is less than the source resolution of the image, the image is down-sampled to the resolution in the i-th set of encoding parameters, and the i-th set of encoding parameters and the down-sampled image are input into the pre-set single-scale encoder for encoding; wherein the encoding parameters corresponding to the i-th encoding are the i-th set of encoding parameters, and i is any integer between 1 and N. In this way, the single-scale encoder supporting multi-resolution encoding can be used to perform multi-scale encoding.

[0037] According to the first aspect, or any one of the implementations of the first aspect, the encoding parameters comprise resolutions, and when the encoder supports multi-resolution encoding, the N times of encoding of the image by the preset single-scale encoder according to the N sets of encoding parameters comprises: when the source resolution of the image is equal to the target encoding resolution, encoding the image by the preset single-scale encoder for the i-th time of encoding; and when the source resolution of the image is greater than the target resolution, down-sampling the image according to the target encoding resolution, and encoding the down-sampled image.

[0038] According to the first aspect, or any one of the implementations of the first aspect, the encoding parameters comprise resolutions, and when the encoder supports multi-resolution encoding, the N times of encoding of the image by the preset single-scale encoder according to the N sets of encoding parameters comprises: when the source resolution of the image is equal to the target encoding resolution, encoding the image by the preset single-scale encoder for the i-th time of encoding; and when the source resolution of the image is greater than the target resolution, down-sampling the image according to the target encoding resolution, and encoding the down-sampled image.

[0039] According to the first aspect, or any one of the implementations of the first aspect, the encoding parameters comprise resolutions, and when the encoder supports multi-resolution encoding, the N times of encoding of the image by the preset single-scale encoder according to the N sets of encoding parameters comprises: when the source resolution of the image is equal to the target encoding resolution, encoding the image by the preset single-scale encoder for the i-th time of encoding; and when the source resolution of the image is greater than the target resolution, down-sampling the image according to the target encoding resolution, and encoding the down-sampled image.

[0040] According to a second aspect, the description information comprises scale identification information, and the method further comprises: determining, according to the scale identification information, that the coding scale corresponding to the code stream data is the lowest coding scale, determining that the transmission strategy of the code stream data is a first transmission strategy; determining, according to the scale identification information, that the coding scale corresponding to the code stream data is other coding scales, determining that the transmission strategy of the code stream data is a second transmission strategy; and wherein the first transmission strategy is superior to the second transmission strategy. In this way, the code stream data of the lowest coding scale can be preferentially transmitted, and the smoothness of video playback can be ensured.

[0041] For example, the first transmission strategy can comprise at least one of: using a transmission channel with a quality of service higher than a first preset quality of service for transmission, retransmitting data and the number of retransmissions being greater than a first preset number of retransmissions, increasing the redundancy of the data to be greater than a first redundancy threshold, and performing FEC (Forward Error Correction) on the data.

[0042] For example, the second transmission strategy can comprise at least one of: using a transmission channel with a quality of service lower than or equal to a first preset quality of service for transmission, not retransmitting data, retransmitting data and the number of retransmissions being less than or equal to a first preset number of retransmissions, not increasing the redundancy of the data, increasing the redundancy of the data to be less than or equal to a first redundancy threshold, and discarding data before transmission.

[0043] According to the second aspect, or any one of the implementations of the second aspect, the description information comprises a frame type, and the method further comprises: determining, when the frame type corresponding to the code stream data is a preset frame type, that the transmission strategy of the code stream data is a third transmission strategy; and determining, when the frame type corresponding to the code stream data is a non-preset frame type, that the transmission strategy of the code stream data is a fourth transmission strategy; and wherein the third transmission strategy is superior to the fourth transmission strategy. The preset frame type comprises at least one of an I frame, an IDR frame, a key frame, a reference frame, and a scene switching frame. In this way, the code stream data of a frame that is important for coding and decoding can be preferentially transmitted, the success rate of decoding of other frames can be improved, and the smoothness of video viewing by a user of a receiving device can be ensured.

[0044] For example, the third transmission strategy can comprise at least one of: using a transmission channel with a quality of service higher than a second preset quality of service for transmission, retransmitting data and the number of retransmissions being greater than a second preset number of retransmissions, increasing the redundancy of the data to be greater than a second redundancy threshold, and performing FEC on the data.

[0045] For example, the fourth transmission strategy can include at least one of the following: transmitting using a transmission channel with a quality of service lower than or equal to the second preset quality of service, not retransmitting the data, retransmitting the data a number of times less than or equal to the second preset number of retransmissions, not increasing the redundancy of the data, increasing the redundancy of the data to less than or equal to the second redundancy threshold, and discarding the data before transmission.

[0046] According to a second aspect, or any possible implementation mode of the second aspect, the description information includes at least one of the following: scale identification information, frame type, and frame identification information.

[0047] For example, the frame identification information can include any identification that can point to a certain frame image, such as a frame number, a display time (a presentation time corresponding to the collected image), and the like.

[0048] In a third aspect, an embodiment of the present application provides a display method, including: obtaining a plurality of reconstructed images obtained by decoding code stream data by a decoder, and obtaining a plurality of sets of description information of the plurality of reconstructed images, wherein each reconstructed image corresponds to a set of description information, and at least one set of description information in the plurality of sets of description information includes first frame identification information; if only one set of description information in the plurality of sets of description information includes the first frame identification information, selecting a reconstructed image corresponding to the first frame identification information to display; and if at least two sets of description information in the plurality of sets of description information include the first frame identification information, selecting one reconstructed image from the at least two reconstructed images corresponding to the first frame identification information to display. In this way, a reconstructed image of a certain coding scale can be selected for each frame to display.

[0049] According to the third aspect, or any possible implementation mode of the third aspect, the description information further includes scale identification information, and selecting one reconstructed image from the plurality of reconstructed images corresponding to the first frame identification information includes: selecting a reconstructed image of the highest coding scale according to the scale identification information of the reconstructed image corresponding to the first frame identification information. The higher the coding scale, the better the image, so that the best image is displayed to the user for each frame.

[0050] According to the third aspect, or any possible implementation mode of the third aspect, the description information further includes scale identification information, and selecting one reconstructed image from the plurality of reconstructed images corresponding to the first frame identification information includes: selecting a reconstructed image whose scale identification information is closest to the scale identification information of a previously displayed reconstructed image according to the scale identification information of the reconstructed image corresponding to the first frame identification information. In this way, the difference between adjacent frames can be reduced, and the visual fluency of the user watching each frame of video can be ensured.

[0051] According to a third aspect, or any possible implementation mode of the third aspect, the description information is obtained by unpacking the received data, wherein the data unpacked from the received data further comprises the code stream data.

[0052] According to a fourth aspect, the embodiments of the present application provide a multi-scale coding device, the multi-scale coding device comprises a pre-processing module and a single-scale coder, the pre-processing module is configured to acquire an image to be coded; determine a coding scale N for the image, wherein N is an integer greater than 1; determine N groups of coding parameters corresponding to the N coding scales, wherein each coding scale corresponds to a group of coding parameters; input the N groups of coding parameters and the image to the single-scale coder; the single-scale coder is configured to code the image N times by using a pre-set single-scale coder according to the N groups of coding parameters, so as to obtain N groups of code stream data.

[0053] According to the fourth aspect, the pre-processing module is configured to determine that the image needs to be coded in a multi-scale manner according to that the coding and decoding importance of the image to other images in the video data meets a first preset condition, and / or the impact degree of the code stream data corresponding to the image on a network meets a second preset condition, and / or the predicted image quality of the image meets a third preset condition.

[0054] According to the fourth aspect, or any possible implementation mode of the fourth aspect, the first preset condition comprises at least one of the following:

[0055] The image is an I frame or an IDR frame;

[0056] The image is a scene switching frame;

[0057] The image is a reference frame;

[0058] The image is a key frame.

[0059] According to the fourth aspect, or any possible implementation mode of the fourth aspect, the second preset condition comprises at least one of the following:

[0060] The data amount of the code stream data corresponding to the image is greater than a preset data amount threshold;

[0061] The data amount of the code stream data corresponding to the image is greater than the network transmission capacity.

[0062] According to the fourth aspect, or any possible implementation mode of the fourth aspect, the predicted image quality of the image comprises a first predicted image quality and a second predicted image quality, the first predicted image quality is the image quality of the predicted image coded at a source resolution of the image, and the second predicted image quality is the image quality of the predicted image coded at a resolution lower than the source resolution; and the third preset condition comprises that the first predicted image quality is lower than or equal to the second predicted image quality.

[0063] According to a fourth aspect, or any possible implementation mode of the fourth aspect, the pre-processing module is configured to determine the coding scale N according to the image type of the image and a preset correspondence between the image type and the coding scale.

[0064] According to a fourth aspect, or any possible implementation mode of the fourth aspect, the pre-processing module is configured to determine the N groups of coding parameters according to the coding scale N and a parameter value range of the encoder.

[0065] According to a fourth aspect, or any possible implementation mode of the fourth aspect, the coding parameters include reference frame information, the reference frame information is determined according to the first feedback information when the first feedback information is received, the first feedback information indicates that the coding frame corresponding to the transmitted bitstream data is successfully received, the coding frame is confirmed as the reference frame, and the reference frame information is used to indicate the reference frame.

[0066] The fourth aspect and any possible implementation mode of the fourth aspect correspond to the first aspect and any possible implementation mode of the first aspect respectively. The technical effects of the fourth aspect and any possible implementation mode of the fourth aspect can refer to the technical effects of the first aspect and any possible implementation mode of the first aspect, which will not be described herein.

[0067] In a fifth aspect, the embodiments of the present application disclose a data encapsulation device, which is used to execute the synchronization method in the second aspect or any possible implementation mode of the second aspect.

[0068] The fifth aspect and any possible implementation mode of the fifth aspect correspond to the second aspect and any possible implementation mode of the second aspect respectively. The technical effects of the fifth aspect and any possible implementation mode of the fifth aspect can refer to the technical effects of the second aspect and any possible implementation mode of the second aspect, which will not be described herein.

[0069] In a sixth aspect, the embodiments of the present application disclose a data encapsulation device, the multi-scale decoding device includes a decoder and a post-processing module, the decoder is configured to decode the bitstream data to obtain a plurality of reconstructed images, and input the plurality of reconstructed images to the post-processing module; the post-processing module is configured to obtain the plurality of reconstructed images decoded by the decoder from the bitstream data, and obtain a plurality of groups of description information of the plurality of reconstructed images, each reconstructed image corresponds to a group of description information, and at least one group of description information in the plurality of groups of description information includes frame identification information; if only one group of description information in the plurality of groups of description information includes first frame identification information, a reconstructed image corresponding to the first frame identification information is selected and input to a display module for display; if at least two groups of description information in the plurality of groups of description information include the first frame identification information, one reconstructed image is selected from the at least two reconstructed images corresponding to the first frame identification information, and the selected reconstructed image is input to the display module for display.

[0070] The sixth aspect and any possible implementation manner of the sixth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively. The technical effects of the sixth aspect and any possible implementation manner of the sixth aspect correspond to the technical effects of the third aspect and any possible implementation manner of the third aspect, which will not be described here.

[0071] In a seventh aspect, an embodiment of the present application provides an electronic device, including the multi-scale coding apparatus in the fourth aspect and the data packaging apparatus in the fifth aspect.

[0072] In an eighth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory being coupled with the processor; the memory storing program instructions, when the program instructions are executed by the processor, causing the electronic device to perform the coding method in the first aspect or any possible implementation manner of the first aspect.

[0073] The eighth aspect and any possible implementation manner of the eighth aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. The technical effects of the eighth aspect and any possible implementation manner of the eighth aspect correspond to the technical effects of the first aspect and any possible implementation manner of the first aspect, which will not be described here.

[0074] In a ninth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory being coupled with the processor; the memory storing program instructions, when the program instructions are executed by the processor, causing the electronic device to perform the data packaging method in the second aspect or any possible implementation manner of the second aspect.

[0075] The ninth aspect and any possible implementation manner of the ninth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively. The technical effects of the ninth aspect and any possible implementation manner of the ninth aspect correspond to the technical effects of the second aspect and any possible implementation manner of the second aspect, which will not be described here.

[0076] In a tenth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, the memory being coupled with the processor; the memory storing program instructions, when the program instructions are executed by the processor, causing the electronic device to perform the display method in the third aspect or any possible implementation manner of the third aspect.

[0077] The tenth aspect and any possible implementation manner of the tenth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively. The technical effects of the tenth aspect and any possible implementation manner of the tenth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively, which will not be described here.

[0078] In a eleventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the encoding method in the first aspect or any possible implementation manner of the first aspect.

[0079] The eleventh aspect and any possible implementation manner of the eleventh aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. The technical effects of the eleventh aspect and any possible implementation manner of the eleventh aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively, which will not be described here.

[0080] In a twelfth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the data packaging method in the second aspect or any possible implementation manner of the second aspect.

[0081] The twelfth aspect and any possible implementation manner of the twelfth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively. The technical effects of the twelfth aspect and any possible implementation manner of the twelfth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively, which will not be described here.

[0082] In a thirteenth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the display method in the third aspect or any possible implementation manner of the third aspect.

[0083] The thirteenth aspect and any possible implementation of the thirteenth aspect correspond to the third aspect and any possible implementation of the third aspect respectively. The technical effects of the thirteenth aspect and any possible implementation of the thirteenth aspect can refer to the technical effects of the third aspect and any possible implementation of the third aspect, which will not be repeated here.

[0084] In a fourteenth aspect, the embodiments of the present application provide a computer storage medium, which stores a computer program. When the computer program is run on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0085] The fourteenth aspect and any possible implementation of the fourteenth aspect correspond to the first aspect and any possible implementation of the first aspect respectively. The technical effects of the fourteenth aspect and any possible implementation of the fourteenth aspect can refer to the technical effects of the first aspect and any possible implementation of the first aspect, which will not be repeated here.

[0086] In a fifteenth aspect, the embodiments of the present application provide a computer storage medium, which stores a computer program. When the computer program is run on a computer or a processor, the computer or the processor executes the data encapsulation method in the second aspect or any possible implementation of the second aspect.

[0087] The fifteenth aspect and any possible implementation of the fifteenth aspect correspond to the second aspect and any possible implementation of the second aspect respectively. The technical effects of the fifteenth aspect and any possible implementation of the fifteenth aspect can refer to the technical effects of the second aspect and any possible implementation of the second aspect, which will not be repeated here.

[0088] In a sixteenth aspect, the embodiments of the present application provide a computer storage medium, which stores a computer program. When the computer program is run on a computer or a processor, the computer or the processor executes the display method in the third aspect or any possible implementation of the third aspect.

[0089] The sixteenth aspect and any possible implementation of the sixteenth aspect correspond to the third aspect and any possible implementation of the third aspect respectively. The technical effects of the sixteenth aspect and any possible implementation of the sixteenth aspect can refer to the technical effects of the third aspect and any possible implementation of the third aspect, which will not be repeated here.

[0090] In a seventeenth aspect, an embodiment of the present application provides a computer program product, which contains a software program. When the software program is executed by a computer or a processor, the steps in the first aspect or any possible implementation manner of the first aspect are executed.

[0091] The seventeenth aspect and any possible implementation manner of the seventeenth aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. For details, refer to the technical effects of the first aspect and any possible implementation manner of the first aspect, which will not be repeated here.

[0092] In an eighteenth aspect, an embodiment of the present application provides a computer program product, which contains a software program. When the software program is executed by a computer or a processor, the steps in the second aspect or any possible implementation manner of the second aspect are executed.

[0093] The eighteenth aspect and any possible implementation manner of the eighteenth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively. For details, refer to the technical effects of the second aspect and any possible implementation manner of the second aspect, which will not be repeated here.

[0094] In a nineteenth aspect, an embodiment of the present application provides a computer program product, which contains a software program. When the software program is executed by a computer or a processor, the steps in the third aspect or any possible implementation manner of the third aspect are executed.

[0095] The nineteenth aspect and any possible implementation manner of the nineteenth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively. For details, refer to the technical effects of the third aspect and any possible implementation manner of the third aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0096] Figure 1 An application scenario schematic diagram is exemplarily shown;

[0097] Figure 2 A scenario schematic diagram is exemplarily shown;

[0098] Figure 3a A processing process schematic diagram is exemplarily shown;

[0099] Figure 3b A reference frame selection schematic diagram is exemplarily shown;

[0100] Figure 4a a data processing schematic diagram shown for example;

[0101] Figure 4b a data processing schematic diagram shown for example;

[0102] Figure 4c a data processing schematic diagram shown for example;

[0103] Figure 5a a processing procedure schematic diagram shown for example;

[0104] Figure 5b a coding schematic diagram shown for example;

[0105] Figure 6a a processing procedure schematic diagram shown for example;

[0106] Figure 6b a processing procedure schematic diagram shown for example;

[0107] Figure 7a a device structure schematic diagram shown for example;

[0108] Figure 7b a processing procedure schematic diagram shown for example;

[0109] Figure 7c a processing procedure schematic diagram shown for example;

[0110] Figure 8 a device structure schematic diagram shown for example. DETAILED DESCRIPTION

[0111] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0112] The term "and / or" in the present application is only used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone.

[0113] The terms "first" and "second" and the like in the specification and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe a specific order of the target objects.

[0114] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0115] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0116] Figure 1 This is a schematic diagram illustrating an application scenario.

[0117] Reference Figure 1 Exemplary Figure 1 The application scenarios can be various video service scenarios, such as video conferencing scenarios, video call scenarios, online education scenarios, remote tutoring scenarios, low-latency live streaming scenarios, cloud gaming scenarios, wireless screen projection scenarios, and wireless extended screen scenarios, etc. This application embodiment does not limit these.

[0118] Reference Figure 1 For example, this application scenario may include a sending device and a receiving device.

[0119] For example, the sending devices include, but are not limited to: servers, PCs (Personal Computers), laptops, tablets, mobile phones, and watches.

[0120] For example, receiving devices include, but are not limited to: PCs, laptops, tablets, mobile phones, and watches.

[0121] For example, in a video conferencing scenario, the sending device can be a PC or a laptop, and the receiving device can be a PC or a laptop.

[0122] For example, in online education scenarios, the sending device can be a PC or laptop, and the receiving device can be a tablet.

[0123] For example, in a cloud gaming scenario, the sending device can be a server, and the receiving device can be a tablet, PC, laptop, or mobile phone.

[0124] For example, the transmitting device may include an image acquisition module, a multi-scale encoding module, and a transmitting module. It should be understood that... Figure 1 This is merely one example of a transmitting device; other embodiments of this application describe transmitting devices with... Figure 1More modules are shown, embodiments of the present application do not limit this.

[0125] The image acquisition module is configured to acquire an image.

[0126] The multi-scale coding module is configured to make a multi-scale coding decision and encode.

[0127] The sending module is configured to send data.

[0128] The receiving device can include a display module, a multi-scale decoding module and a receiving module. Figure 1 The receiving device is only an example, and the receiving device in other embodiments of the present application has more modules than Figure 1 More modules are shown, embodiments of the present application do not limit this.

[0129] The receiving module is configured to receive data.

[0130] The multi-scale decoding module is configured to decode and make a display decision.

[0131] The display module is configured to display.

[0132] Continuing to refer to Figure 1 The sending device acquires an image and sends the acquired image to the receiving device for display, and the process is as follows:

[0133] S101, the image acquisition module outputs the acquired image to the multi-scale coding module.

[0134] The image acquisition module can acquire an image through a camera.

[0135] The image acquisition module can acquire an image output by a graphics card (also known as a display card).

[0136] The image acquisition module can acquire an image through screen scraping.

[0137] The image acquisition module can output the acquired image to the multi-scale coding module after acquiring the image.

[0138] S102, the multi-scale coding module outputs the code stream data to the sending module.

[0139] Exemplarily, after receiving the image, the multi-scale coding module can make a multi-scale coding decision, that is, make a decision on whether the image needs to be multi-scale coded (which can also be referred to as scalable coded or hierarchical coded). When it is determined that the image needs to be multi-scale coded, the image can be multi-scale coded, and when it is determined that the image does not need to be multi-scale coded, the image can be single-scale coded. After coding the image, the multi-scale coding module outputs the coded stream data to the sending module.

[0140] S103, the sending module sends the stream data to the receiving module.

[0141] Exemplarily, after receiving the stream data, the sending module can send the stream data to the receiving device.

[0142] S104, the receiving module outputs the received stream data to the multi-scale decoding module.

[0143] Exemplarily, the receiving module of the receiving device can receive the stream data, and then output the stream data to the multi-scale decoding module.

[0144] S105, the multi-scale decoding module outputs the reconstructed image to the display module.

[0145] Exemplarily, after receiving the stream data, the multi-scale decoding module can decode based on the stream data to reconstruct the image and obtain a reconstructed image. Exemplarily, for an image that has been multi-scale coded, a plurality of reconstructed images of different coding scales are obtained by decoding, at this time, the multi-scale decoding module can make a display decision to select a reconstructed image of a coding scale and output it to the display module. Exemplarily, for an image that has been single-scale coded, a reconstructed image of a coding scale is obtained by decoding, at this time, the multi-scale decoding module can directly output the image to the display module.

[0146] Exemplarily, after receiving the reconstructed image input by the multi-scale decoding module, the display module can display the reconstructed image.

[0147] Figure 2 For the example of the scene diagram.

[0148] Reference Figure 2 Exemplarily, the multi-scale coding module can include a pre-processing module and an encoder.

[0149] Exemplarily, the pre-processing module can be used for multi-scale coding decision, and the encoder can be used for coding.

[0150] It should be noted that the multi-scale encoding module includes a single-scale encoder. The single-scale encoder can refer to an encoder that can only encode an image in one encoding scale and output code stream data in only one encoding scale, such as an H.264 encoder, an H.265 encoder, or an H.266 encoder, an AV1 encoder, an AVS3 encoder, and the like. Embodiments of the present application do not limit this. For example, the single-scale encoder can be a software encoder or a hardware encoder, and embodiments of the present application do not limit this.

[0151] For example, a pre-processing module can be added to a sending device including a single-scale encoder (i.e., the corresponding program code of the pre-processing module is written in the sending device), that is, a multi-scale encoding module can be obtained, so that multi-scale encoding is realized using an existing single-scale encoder.

[0152] Since the multi-scale encoder (such as an SVC (Scalable Video Coding, scalable video coding) encoder, an SHVC (Scalable High efficiency Video Coding, scalable high efficiency video coding) encoder) in the prior art has a relatively high cost, and most of the encoders in the current electronic devices are single-scale encoders, therefore, using the method described above, multi-scale encoding and decoding can be realized in electronic devices that do not support multi-scale encoding without increasing costs.

[0153] Continuing to refer to Figure 2 For example, the sending module can include a data encapsulation module and a data sending module. The data encapsulation module is used to encapsulate data, and the data sending module is used to send the encapsulated data.

[0154] Continuing to refer to Figure 2 For example, the multi-scale decoding module can include a decoder and a post-processing module. The decoder is used for decoding, and the post-processing module is used for display decision.

[0155] For example, the decoder can be a single-scale decoder, which can refer to a decoder that can only decode code stream data in one encoding scale and produce a unique reconstructed image after decoding, such as an H.264 decoder, an H.265 decoder, or an H.266 decoder, an AV1 decoder, an AVS3 decoder, and the like. Embodiments of the present application do not limit this. For example, the single-scale decoder can be a software decoder or a hardware decoder, and embodiments of the present application do not limit this.

[0156] Exemplarily, a post-processing module can be added in the receiving device containing the single-scale decoder (that is, the corresponding program code of the post-processing module is written in the sending device), that is, a multi-scale decoding module can be obtained, so that multi-scale decoding is realized by using the existing single-scale decoder.

[0157] Since the cost of the decoder supporting multi-scale decoding (such as the SVC decoder and the SHVC decoder) is high in the prior art, and the decoder in most electronic devices at present is a single-scale decoder, by using the method described above, multi-scale decoding can be realized in the electronic device without supporting multi-scale coding without increasing the cost.

[0158] With reference to Figure 2 , exemplarily, the receiving module includes a data receiving module and a data parsing module. Exemplarily, the data receiving module is configured to receive data, and the data parsing module is configured to parse the data.

[0159] With reference to Figure 2 , exemplarily, after the image acquisition module acquires an image, the acquired image can be output to the pre-processing module. Exemplarily, in a video service scenario, the image acquisition module can acquire images according to a set video sampling rate to obtain video data corresponding to the video service scenario. Exemplarily, the image acquisition module can transmit images to the pre-processing module at a set interval (the set interval can be set according to requirements), or can transmit one image to the pre-processing module after acquiring one image, and the embodiments of the present application do not limit this. The embodiments of the present application exemplarily take one image in the video data as an example for illustration.

[0160] Exemplarily, the pre-processing module can determine whether the image needs to be encoded in multi-scale, determine the encoding scale N (N is a positive integer greater than 1) of the image when it is determined that the image needs to be encoded in multi-scale, and determine N sets of encoding parameters corresponding to the N encoding scales, wherein each encoding scale corresponds to one encoding parameter (that is, the encoding parameter corresponding to each encoding scale in the N encoding scales is determined). Then, the pre-processing module can instruct the encoder to encode the image N times according to the N sets of encoding parameters, and N sets of code stream data can be obtained, each set of code stream data corresponding to one encoding scale. The way in which the pre-processing module determines whether the image needs to be encoded in multi-scale, the way in which the pre-processing module determines the encoding scale N, and the way in which the pre-processing module determines the encoding parameter are described below.

[0161] Exemplarily, the encoding parameter can include at least one of the following: resolution, quantization parameter, reference frame information, and target code rate. The quantization parameter and the target code rate can be obtained by mutual conversion, so when the encoding parameter contains multiple parameters, the encoding parameter can only contain one of the quantization parameter and the target code rate.

[0162] Exemplarily, the resolution can include a source resolution and / or a target coding resolution.

[0163] Exemplarily, the source resolution refers to a resolution of an image input to the encoder.

[0164] Exemplarily, the target coding resolution refers to a resolution of an image that can be reconstructed after the image is coded, or a resolution actually coded by the encoder.

[0165] It should be understood that when the source resolution and the target coding resolution are the same, only one of them can be input.

[0166] Exemplarily, the quantization parameter refers to a coding quantization parameter used by the encoder to code the image. The quantization parameter determines a quality distortion degree of the image after coding. The greater the quantization parameter, the greater the distortion degree of the image after coding relative to the source image; the smaller the quantization parameter, the smaller the distortion degree of the image after coding relative to the source image.

[0167] Exemplarily, the reference frame can be a frame referenced by the encoder to code a certain frame, or a frame referenced by the decoder to reconstruct an image of a certain frame. The reference frame information is used to indicate the reference frame, and can include frame identification, scale identification information, etc. The frame identification can include any identification that can point to a certain frame image, such as a frame number, a display time (a presentation time corresponding to the collected image). The scale identification information is used to uniquely identify a coding scale.

[0168] Exemplarily, the target code rate refers to a code rate expected after the image is coded.

[0169] It should be understood that the coding parameter can further include other parameters such as a target quality, which is not limited in the present application. Exemplarily, the target quality refers to a quality expected after the image is coded, for example, the target quality can include but is not limited to a target PSNR (Peak Signal to Noise Ratio), a target MSE (Mean Squared Error), a SSIM (Structural SIMilarity), or a VMAF (Video Multimethod Assessment Fusion), etc.

[0170] Exemplarily, after determining the coding parameter, the pre-processing module can input the coding parameter and the image into the encoder. Exemplarily, the pre-processing module can input the image and a group of coding parameters into the encoder for coding each time, until N groups of coding parameters are input into the encoder.

[0171] Exemplarily, when the pre-processing module determines that the image does not need to be encoded in multiple scales, the pre-processing module can determine the encoding parameters of single-scale encoding, and then instruct the encoder to encode the image according to the encoding parameters of single-scale encoding, so as to obtain a group of code stream data.

[0172] Exemplarily, after receiving the encoding parameters and the image, the encoder can encode the image according to the encoding parameters, so as to obtain code stream data, and then transmit the code stream data to the data packaging module.

[0173] Exemplarily, the pre-processing module can also determine the description information of the image. The description information can include various information, including but not limited to: scale identification information, frame identification, and frame type. It should be understood that the description information can also include other information, which is not limited in the present application.

[0174] Exemplarily, the frame type can include but is not limited to: I / IDR frame (Intra frame, intra-coded frame), key frame, scene switching frame, reference frame, and non-reference frame.

[0175] Exemplarily, the I / IDR frame refers to a frame encoded in an intra coding manner.

[0176] Exemplarily, the key frame refers to a frame that is referenced by more than a preset number threshold. The preset number threshold can be set according to requirements, which is not limited in the embodiments of the present application.

[0177] Exemplarily, the scene switching frame is the first frame image of the switched scene after the scene switching.

[0178] Exemplarily, the non-reference frame refers to a frame other than the reference frame.

[0179] Exemplarily, for an image that is encoded in multiple scales, the pre-processing module can determine the scale identification information of the encoding scale corresponding to each group of encoding parameters, the frame identification and the frame type of the image as the description information of the image in the encoding scale. That is, for an image that is encoded in multiple scales, the pre-processing module can generate N groups of description information, each group of description information corresponding to an encoding scale, and the N groups of description information having the same frame identification information.

[0180] Exemplarily, for an image that is encoded in single scale, the pre-processing module can determine the frame identification and the frame type of the image as the description information of the image. It should be understood that the pre-processing module can also determine the scale identification information of the single-scale corresponding encoding scale, the frame identification and the frame type of the image as the description information of the image, which is not limited in the embodiments of the present application.

[0181] Exemplarily, for the image coded in multiple scales for one frame, the data encapsulation module can encapsulate the i-th group of description information and the i-th group of stream data to obtain the encapsulated stream data and description information of the image in the i-th coding scale. Wherein, i is a positive integer less than or equal to N.

[0182] Exemplarily, for the image coded in single scale for one frame, the pre-processing module can encapsulate the description information and the stream data of the image to obtain the encapsulated stream data and description information corresponding to the image.

[0183] Exemplarily, in one possible manner, the data encapsulation module can directly transmit the encapsulated stream data and description information to the data sending module. In another possible manner, the data encapsulation module can also store the encapsulated stream data and description information.

[0184] Exemplarily, the data encapsulation module can also determine the transmission strategy of the encapsulated stream data and description information according to the description information, and then transmit the corresponding transmission strategy to the data sending module while transmitting the encapsulated stream data and description information to the data sending module.

[0185] Exemplarily, when the data encapsulation module determines that the coding scale corresponding to the stream data is the lowest coding scale according to the scale identification information, the data encapsulation module determines that the transmission strategy of the stream data is the first transmission strategy; when the data encapsulation module determines that the coding scale corresponding to the stream data is other coding scale according to the scale identification information, the data encapsulation module determines that the transmission strategy of the stream data is the second transmission strategy; wherein, the first transmission strategy is superior to the second transmission strategy. In this way, the stream data of the lowest coding scale can be transmitted preferentially to improve the success rate of transmitting the stream data of the lowest coding scale to the receiving device, so as to ensure the fluency of the user watching the video in the receiving device.

[0186] Exemplarily, the first transmission strategy can include at least one of the following: using a transmission channel with a quality of service higher than a first preset quality of service for transmission, retransmitting data and the number of retransmissions being greater than a first preset number of retransmissions, increasing the redundancy of the data to be greater than a first redundancy threshold, and performing FEC (Forward Error Correction) on the data.

[0187] Exemplarily, the second transmission strategy can include at least one of the following: using a transmission channel with a quality of service lower than or equal to a first preset quality of service for transmission, not retransmitting data, retransmitting data and the number of retransmissions being less than or equal to a first preset number of retransmissions, not increasing the redundancy of the data, increasing the redundancy of the data to be less than or equal to a first redundancy threshold, and discarding the data before transmission.

[0188] The first preset service quality, the first preset retransmission number, and the first redundancy threshold can be set according to requirements, and embodiments of the present application do not limit this.

[0189] For example, when the data encapsulation module determines that the frame type corresponding to the code stream data is a preset frame type, the data encapsulation module determines that the transmission strategy of the code stream data is a third transmission strategy; when the data encapsulation module determines that the frame type corresponding to the code stream data is a non-preset frame type, the data encapsulation module determines that the transmission strategy of the code stream data is a fourth transmission strategy; wherein the third transmission strategy is superior to the fourth transmission strategy. The preset frame type includes at least one of the following: an I frame, an IDR frame, a key frame, a reference frame, and a scene switching frame. In this way, the code stream data of the frame that is important for coding and decoding can be preferentially transmitted, so as to improve the success rate of successful decoding of other frames, thereby ensuring the fluency of video viewing of a user of a receiving device.

[0190] For example, the third transmission strategy can include at least one of the following: using a transmission channel with a service quality higher than a second preset service quality for transmission, retransmitting data and the retransmission number being greater than a second preset retransmission number, increasing the redundancy of the data to be greater than a second redundancy threshold, and performing FEC (Forward Error Correction) on the data.

[0191] For example, the fourth transmission strategy can include at least one of the following: using a transmission channel with a service quality lower than or equal to a second preset service quality for transmission, not retransmitting data, retransmitting data and the retransmission number being less than or equal to a second preset retransmission number, not increasing the redundancy of the data, increasing the redundancy of the data to be less than or equal to a second redundancy threshold, and discarding the data before sending.

[0192] The second preset service quality, the second preset retransmission number, and the second redundancy threshold can be set according to requirements, and embodiments of the present application do not limit this.

[0193] For example, the second preset service quality and the first preset service quality can be the same or different. The second preset retransmission number and the first preset retransmission number can be the same or different. The second redundancy threshold and the second redundancy threshold can be the same or different. Embodiments of the present application do not limit this.

[0194] For example, after the data receiving module receives the encapsulated code stream data and the description information, the data receiving module can send the encapsulated code stream data and the description information to a receiving device according to the corresponding transmission strategy.

[0195] For example, after the data receiving module of the receiving device receives the encapsulated code stream data and the description information, the data receiving module sends the encapsulated code stream data and the description information to a data analysis module.

[0196] Exemplarily, the data parsing module can unpack the encapsulated code stream data and the description information to obtain the description information and the code stream data. Then, on one hand, the data parsing module can send the code stream data to the decoder, and on the other hand, the data parsing module can send the description information to the post-processing module.

[0197] Exemplarily, the decoder can reconstruct an image based on the code stream data to obtain a reconstructed image, and then can send the reconstructed image to the post-processing module. Wherein, for the image that has been encoded in multiple scales, the corresponding reconstructed image includes N, each reconstructed image corresponds to one encoding scale, and since only one reconstructed image is displayed in the same frame when displaying, the post-processing module is needed to make a display decision. Exemplarily, for the image that has been encoded in single scale, the corresponding reconstructed image is one.

[0198] Exemplarily, after the post-processing module receives the multiple reconstructed images, the post-processing module can select the reconstructed image for display according to the multiple sets of description information corresponding to the multiple reconstructed images. Wherein, each reconstructed image corresponds to a set of description information, and each set of description information includes frame identification information. Exemplarily, in the received multiple reconstructed images, the frame identification information of each reconstructed image can be different, or the frame identification information of part of the reconstructed images can be the same, and then for each frame identification information, the reconstructed image for display can be selected from the reconstructed image corresponding to the frame identification information. When the post-processing module receives the display signal sent by the display module or reaches the display period, the post-processing module can select the frame identification information corresponding to the latest decoded one of the multiple reconstructed images from the multiple sets of description information as the first frame identification information, and then determine the number of description information containing the first frame identification information in the multiple sets of description information. If there is only one set of description information including the first frame identification information in the multiple sets of description information, the reconstructed image corresponding to the first frame identification information is selected for display; if there are at least two sets of description information including the first frame identification information in the multiple sets of description information, one reconstructed image is selected from the at least two reconstructed images corresponding to the first frame identification information, and the selected reconstructed image is displayed.

[0199] In one possible way, the post-processing module can determine the scale identification information of each reconstructed image according to the description information corresponding to each reconstructed image. Then, according to the scale identification information of the multiple reconstructed images corresponding to the first frame identification information, the reconstructed image with the highest encoding scale can be selected from the multiple reconstructed images corresponding to the first frame identification information. Wherein, the higher the encoding scale is, the better the image is, and thus when each frame is displayed to the user, the best image is displayed.

[0200] In a possible manner, the post-processing module can select, from the plurality of reconstructed images corresponding to the first frame identifier information, a reconstructed image with scale identifier information closest to scale identifier information of a previously displayed reconstructed image according to the scale identifier information of the plurality of reconstructed images corresponding to the first frame identifier information. In this way, the difference between adjacent frames can be reduced, and the visual fluency of the user in viewing each frame of the video can be ensured.

[0201] For example, after the display module receives the reconstructed images sent by the post-processing module, the display module can display the received reconstructed images according to the frame numbers of the reconstructed images.

[0202] The manner in which the pre-processing module determines whether the image needs to be encoded in multiple scales, the manner in which the pre-processing module determines the encoding scale N, and the manner in which the pre-processing module determines the encoding parameters are described below.

[0203] The mth(m is a positive integer) frame of image is taken as an example for description.

[0204] Figure 3a A schematic diagram of the processing procedure is shown for example. Refer to Figure 3a For example, the data processing procedure of the pre-processing module is as follows:

[0205] S301: Determine whether the mthframe of image needs to be encoded in multiple scales.

[0206] For example, after the pre-processing module receives the mthframe of image input by the image acquisition module, the pre-processing module can determine whether the image needs to be encoded in multiple scales according to the encoding importance of the mthframe of image to other frames in the video data and / or the impact of the code stream data corresponding to the mthframe of image on the network. When the encoding importance of the mthframe of image to other frames in the video data satisfies a first preset condition and / or the impact of the code stream data corresponding to the mthframe of image on the network satisfies a second preset condition, it is determined that the image needs to be encoded in multiple scales.

[0207] For example, the first preset condition includes at least one of the following:

[0208] The mthframe of image is an I frame or an IDR frame;

[0209] The mthframe of image is a scene switching frame;

[0210] The mthframe of image is a reference frame;

[0211] The mthframe of image is a key frame.

[0212] That is, when the m-th frame image is an I frame or an IDR frame, and / or is a scene switching frame, and / or is a reference frame, and / or is a key frame, it indicates that the m-th frame image is more important for the coding of other frames in the video data, and at this time, it can be determined that the m-th frame image needs to be multi-scale coded to ensure that when the network transmission capability is poor, the probability of successfully transmitting the stream data of the lowest coding scale with the smallest data amount to the receiving device can be increased, and the fluency of the user watching the video on the receiving device can be ensured.

[0213] For example, according to the importance of the m-th frame image for the coding of other frames in the video data, the way of determining whether the m-th frame image needs to be multi-scale coded can include at least one of the following:

[0214] (1) Determine whether the m-th frame image is an I frame or an IDR frame.

[0215] For example, determining whether the m-th frame image is an I frame or an IDR frame is to determine whether the m-th frame image will be coded into an I frame or an IDR frame.

[0216] For example, the number S (S is a positive integer) of images contained in a GOP (Group of picture) can be set in advance according to the video service scenario, and the first frame image in each GOP can be set as an I frame or an IDR frame. After the pre-processing module receives the m-th frame image, it can determine whether the m-th frame image will be coded into an I / IDR frame according to m and S. For example, when m = 1, the pre-processing module can determine that the m-th frame image is an I / IDR frame. When m is greater than 1 and m is less than or equal to S, the pre-processing module can determine that the m-th frame image is not an I / IDR frame. When m is greater than S, the pre-processing module can calculate the remainder of m and S, and if the remainder is equal to 1, the pre-processing module can determine that the m-th frame image is an I / IDR frame, and if the remainder is not equal to 1, the pre-processing module can determine that the m-th frame image is not an I / IDR frame.

[0217] For example, in the process of determining whether the mth frame of image is an I / IDR frame according to m and S, if the pre-processing module receives the display failure information (e.g. a screen of flowers) fed back by the receiving device, the pre-processing module can determine the mth frame of image as an I / IDR frame, and can take the image as the first frame of the GOP and record the value of m as R. After receiving the image after the Rth frame, the pre-processing module can determine whether the mth frame of image is an I / IDR frame according to R, m and S. For example, when (m-R) is greater than 0 and (m-R) is less than or equal to S, the pre-processing module can determine that the mth frame of image is not an I / IDR frame. When (m-R) is greater than S, the pre-processing module can calculate the remainder of (m-R) divided by S, and if the remainder is equal to 1, the pre-processing module can determine that the mth frame of image is an I / IDR frame, and if the remainder is not equal to 1, the pre-processing module can determine that the mth frame of image is not an I / IDR frame.

[0218] (2) Determine whether the mth frame of image is a scene switching frame.

[0219] For example, if m = 1, the pre-processing module can determine that the mth frame of image is a scene switching frame. If m is greater than 1, the pre-processing module can compare the mth frame of image with the (m-1)th frame of image. If the similarity between the mth frame of image and the (m-1)th frame of image is less than or equal to a similarity threshold, it can be determined that the mth frame of image is a scene switching frame. If the similarity between the mth frame of image and the (m-1)th frame of image is greater than a preset similarity threshold, it is determined that the mth frame of image is not a scene switching frame.

[0220] (3) Determine whether the mth frame of image is a reference frame.

[0221] For example, the pre-processing module can assign a corresponding reference frame to each frame of image before sending each frame of image to the encoder.

[0222] For example, determining whether the mth frame of image is a reference frame is to determine whether the mth frame of image is assigned as a reference frame of other frames. For example, the mth frame of image can be assigned as a reference frame of other frames before the mth frame of image, or can be assigned as a reference frame of other frames after the mth frame of image, which will not be described here. When the mth frame of image is assigned as a reference frame of other frames, it can be determined that the mth frame is a reference frame.

[0223] (4) Determine whether the mth frame of image is a key frame.

[0224] For example, the number of other frames that are encoded by using the code stream data of the mth frame of image can be determined, and whether the number is less than a preset number threshold. The preset number threshold can be set according to requirements, for example, a GOP includes 8 frames of image, and the preset number threshold can be 4, that is, 4 frames of image in a GOP depend on the image for encoding and decoding.

[0225] For example, when the number of other frames coded in dependence on the mth frame image is greater than or equal to a preset number threshold, it can be determined that the mth frame image is a key frame. When the number of other frames coded in dependence on the mth frame image is less than the preset number threshold, it can be determined that the mth frame image is a non-key frame.

[0226] For example, the mth frame image can be one or more of a scene switching frame, an I frame, an IDR frame, a reference frame, and a key frame.

[0227] For example, the second preset condition includes at least one of the following:

[0228] The data amount of the code stream data of the mth frame image is greater than a preset data amount threshold.

[0229] The data amount of the code stream data of the mth frame image is greater than a network transmission capability.

[0230] That is, when the data amount of the code stream data of the mth frame image is greater than a preset data amount threshold, or the data amount of the code stream data of the mth frame image is greater than a network transmission capability, it indicates that the impact degree of the code stream data corresponding to the mth frame image on the network is relatively large. At this time, it can be determined that the mth frame image needs to be multi-scale coded, and thus the probability of successfully transmitting the code stream data of the lowest coding scale with the smallest data amount to a receiving device can be increased, and the fluency of a user watching a video at the receiving device can be ensured.

[0231] For example, the manner of judging whether the mth frame image needs to be multi-scale coded according to the impact degree of the code stream data corresponding to the mth frame image on the network can include at least one of the following:

[0232] (1) judging whether the data amount of the code stream data of the mth frame image is greater than a preset data amount threshold.

[0233] For example, the pre-processing module can estimate the data amount of the code stream data of the mth frame image according to the parameters of the encoder and the image complexity of the mth frame image, and then judge whether the data amount of the code stream data of the mth frame image is greater than a preset data amount threshold. The preset data amount threshold can be set as required, and embodiments of the present application do not limit this.

[0234] In a possible manner, the pre-processing module can send the mth frame image into the encoder for pre-coding to judge whether the data amount of the code stream data of the mth frame image is greater than a preset data amount threshold.

[0235] (2) judging whether the data amount of the code stream data of the mth frame image is greater than a network transmission capability.

[0236] Exemplarily, the pre-processing module can acquire the current network transmission capability (such as network bandwidth, i.e. the amount of data that can be transmitted per unit of time (generally, 1 second)) of the sending device, and estimate the data amount of the code stream data of the mth image, which can be referred to the related description above, and will not be described herein again. Then the pre-processing module compares the data amount of the code stream data of the mth image with the current network transmission capability of the sending device, to determine whether the data amount of the code stream data of the mth image is greater than the network transmission capability.

[0237] Exemplarily, when it is determined that the mth image needs to be multi-scale coded, S302 can be performed. When it is determined that the mth image does not need to be multi-scale coded, S307 can be performed.

[0238] S302, determining the coding scale N and the N groups of coding parameters for the mth image.

[0239] In a possible manner, the corresponding coding scale can be set for each frame in the video data in advance, which can be referred to Table 1:

[0240] Table 1

[0241] Frame number Coding scale 1 N＝4 2 N＝3 3 N＝2 4 No multiscale coding ... ... M (M is a positive integer) N＝2

[0242] In Table 1, M can be the total number of frames of the images contained in the video data, or the number of frames of the images contained in a GOP of the video data, which is not limited in the embodiments of the present application. Exemplarily, when M is the number of frames of the images contained in a GOP, the coding scales set for the frames at the same position in different GOPs can be the same or different, which is not limited in the embodiments of the present application.

[0243] Further, the coding scale corresponding to the mth image can be found from the coding scales set in advance for each frame in the video data. For example, M is the total number of frames of the images contained in the video data, if m = 2, the coding scale N = 3 corresponding to the mth image can be determined.

[0244] In a possible manner, the coding scales corresponding to various frame types can be set in advance, such as Table 2:

[0245] Table 2

[0246]

[0247]

[0248] Further, the coding scale corresponding to the mth image can be found from the coding scales set in advance for various frame types according to the frame type of the mth image. For example, the frame type of the mth image is a key frame, and the coding scale N = 3 corresponding to the mth image can be determined.

[0249] In a possible implementation, the encoding scales N corresponding to various network transmission capabilities can be preset, as shown in Table 3:

[0250] Table 3

[0251]

[0252] For example, D1 < D2 < D3 < D4 < D5 in Table 3. Wherein, D1, D2, D3, D4 and D5 can be set according to requirements, and embodiments of the present application do not limit this.

[0253] Further, the encoding scale corresponding to the mth image can be found from the preset encoding scales corresponding to various network transmission capabilities according to the data amount of the mth image. For example, the data amount of the mth image is 2 MB, and the data amount of the (m+1)th image is 4 MB. It can be determined that the encoding scale N corresponding to the mth image is 2.

[0254] In a possible implementation, the encoding scales N corresponding to the frame types of images and network transmission capabilities can be preset, as shown in Table 4:

[0255] Table 4

[0256]

[0257] Further, the encoding scale corresponding to the mth image can be found from the preset encoding scales corresponding to the frame types of images and network transmission capabilities according to the data amount of the code stream data of the mth image and the frame type. For example, the data amount of the code stream data of the mth image is 2 MB, and the data amount of the code stream data of the (m+1)th image is 4 MB. If the frame type is a scene switching frame, it can be determined that the encoding scale N corresponding to the mth image is 3.

[0258] It should be noted that Tables 1-4 are only examples of the encoding scale N, and do not represent the value of the encoding scale N in actual application.

[0259] In a possible implementation, for each N value, the encoding parameters of each encoding scale in N encoding scales can be preset, that is, N groups of encoding parameters corresponding to each N value are preset.

[0260] For example, for N=2 in Table 1, the encoding parameters corresponding to two encoding scales can be preset, as shown in Table 5:

[0261] Table 5

[0262]

[0263] For N=3 in Table 1, the encoding parameters corresponding to three encoding scales can be preset, as shown in Table 6:

[0264] Table 6

[0265]

[0266] For N=4 in Table 1, the encoding parameters corresponding to the four encoding scales can be preset, as shown in Table 7:

[0267] Table 7

[0268]

[0269] For example, in Tables 5-7, the greater the QP, the lower the encoding quality, and the smaller the QP, the higher the encoding quality.

[0270] It should be noted that although the encoding parameters corresponding to the first encoding scale in Tables 5, 6 and 7 are the same, it should be understood that the encoding parameters corresponding to the first encoding scale in Tables 5, 6 and 7 can be different. And although the encoding parameters corresponding to the second encoding scale in Tables 6 and 7 are the same, it should be understood that the encoding parameters corresponding to the second encoding scale in Tables 6 and 7 can be different, and so on. That is, when the value of the encoding scale N is different, the encoding parameters corresponding to the same encoding scale can be the same or different, and the embodiments of the present application do not limit this.

[0271] For example, the pre-processing module can find the encoding parameters corresponding to N from the preset encoding parameters according to the encoding scale N corresponding to the mth image.

[0272] For example, the encoding scale N corresponding to the mth image is 2, and the corresponding encoding parameters can be as shown in Table 5.

[0273] For example, the encoding scale N corresponding to the mth image is 3, and the corresponding encoding parameters can be as shown in Table 6.

[0274] In one possible manner, the encoding parameters corresponding to each encoding scale can be determined according to at least two of the encoding scale N, the image complexity of the mth image, the current network transmission capability and the parameter value range of the encoder.

[0275] For example, the encoding parameters corresponding to each encoding scale can be determined according to the encoding scale N and the parameter value range of the encoder.

[0276] For example, N=2, and the parameter value range of the encoder satisfies the first encoding condition, and the encoding parameters are as shown in Table 8:

[0277] Table 8

[0278]

[0279] For example, N=2, the parameter value range of the encoder satisfies the second encoding condition, and the encoding parameters are shown in Table 9:

[0280] Table 9

[0281]

[0282] The encoding quality of the encoder when the parameter value range of the encoder satisfies the second encoding condition is greater than the encoding quality of the encoder when the parameter value range of the encoder satisfies the first encoding condition.

[0283] For example, the first encoding condition and the second encoding condition can be set according to the encoding quality of the encoder when each parameter of the encoder takes a different range of values. Details are not described herein.

[0284] For example, N=3, the parameter value range of the encoder satisfies the first encoding condition, and the encoding parameters are shown in Table 10:

[0285] Table 10

[0286]

[0287] For example, the encoding parameters corresponding to each encoding scale can be determined according to the encoding scale N and the network transmission capability.

[0288] For example, N=2, the network data transmission amount is greater than the first data amount threshold, and the encoding parameters are shown in Table 11:

[0289] Table 11

[0290]

[0291] For example, N=2, the network data transmission amount is greater than the second data amount threshold, and the encoding parameters are shown in Table 12:

[0292] Table 12

[0293]

[0294]

[0295] The first amount threshold can be set as required, and embodiments of the present application do not limit this.

[0296] For example, N=3, the network data transmission amount is greater than the first data amount threshold, and the encoding parameters are shown in Table 13:

[0297] Table 13

[0298]

[0299] It should be understood that the encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N and the image complexity. The encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N, the image complexity and the parameter value range of the encoder. The encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N, the image complexity and the network transmission capability. The encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N, the image complexity, the parameter value range of the encoder and the network transmission capability, and the like, which will not be listed one by one here.

[0300] For example, the encoding parameters can also include reference frame information.

[0301] In one possible manner, the reference frame information can be pre-set. For example, the reference frame corresponding to each image in the video data can be pre-set. For example, for each image, the frame before the frame type of the preset frame type (such as I / IDR frame, scene switching frame, key frame, reference frame) can be determined as the reference frame of each image at one encoding scale.

[0302] For example, after determining the reference frame of each image, the reference frame information of each image at one encoding scale can be determined according to the sequence number (POC number) of the reference frame of each image entering the encoder.

[0303] For example, the POC number of the reference frame of each image entering the encoder can be taken as the reference frame information of each image at one encoding scale.

[0304] For example, for an image, the distance between the reference frame of the image and the image can be determined according to the POC number of the reference frame of the image entering the encoder and the POC number of the image input to the encoder when the image is encoded at one encoding scale; and the reference frame of the image and the distance between the reference frame of the image and the image can be determined as the reference frame information of the image at one encoding scale. For example, the reference frame of the image and the distance between the reference frame of the image and the image can refer to the difference between the POC number of the image input to the encoder when the image is encoded at one encoding scale and the POC number of the reference frame of the image entering the encoder.

[0305] Further, the pre-processing module can find the reference frame information of the mth image from the pre-set reference frame information.

[0306] In one possible manner, when the first feedback information is received, reference frame information is determined according to the first feedback information, the first feedback information indicates that the encoded frame corresponding to the code stream data that has been sent is successfully received, the encoded frame is confirmed as a reference frame, and the reference frame information is used to indicate the reference frame. In this way, when code stream data of a certain encoded frame is lost, only the decoding and display of the encoded frame are affected, and the decoding and display of other encoded frames are not affected, thereby limiting the influence of data loss on decoding and display.

[0307] For example, the first feedback information can include frame identification information and scale identification information. For example, the reference frame information of the mth image at a certain encoding scale can be determined according to the frame identification information and the scale identification information in the first feedback information.

[0308] For example, the POC number of the reference frame of the mth image entering the encoder can be used as the reference frame information of the mth image at a certain encoding scale.

[0309] For example, the distance between the reference frame of the mth image and the mth image can be determined according to the POC number of the reference frame of the mth image entering the encoder and the POC number of the mth image input to the encoder when the mth image is encoded at a certain encoding scale; and the reference frame of the mth image and the distance between the reference frame of the mth image and the mth image are used as the reference frame information of the mth image at a certain encoding scale. For example, the distance between the reference frame of the mth image and the mth image can be the difference between the POC number of the mth image input to the encoder when the mth image is encoded at a certain encoding scale and the POC number of the reference frame of the mth image entering the encoder.

[0310] For example, the reference frame information of the mth image at different encoding scales can be the same or different, and the specific manner can be set according to requirements, which is not limited in the embodiments of the present application.

[0311] Figure 3b A schematic diagram for selecting a reference frame is shown for example. Referring to Figure 3b For example, F1 to F7 are seven images. Among them, F1 and F4 need to be encoded at multiple scales, and F2, F3, F5, F6, and F7 do not need to be encoded at multiple scales.

[0312] Before F4 is encoded, the transmission result of the code stream data corresponding to any one of F1-F3 is not received, so F2-F4 can be encoded with the reconstructed image of the lowest encoding scale of F1 as the reference. Before F5 is encoded, the transmission result of the code stream data corresponding to the highest encoding scale of F1 is received, and the transmission is successful, so F5 is encoded with the reconstructed image of the highest encoding scale of F1 as the reference. Before F6 is encoded, the transmission result of the code stream data corresponding to F2 is received, and the transmission is successful, so F6 is encoded with the reconstructed image of F2 as the reference. Before F7 is encoded, the transmission result of the code stream data corresponding to F3 is received, and the transmission is failed, so F7 is encoded with the reconstructed image of F2 as the reference.

[0313] Exemplarily, in the embodiment of the present application, the encoding parameters include: quantization parameters and reference frame information.

[0314] Exemplarily, after the pre-processing module determines the encoding scale N and the N groups of encoding parameters, it can first send a group of encoding parameters and the mth frame of image into the encoder, and then the encoder encodes the mth frame of image according to the group of encoding parameters. Then the next group of encoding parameters and the mth frame of image are sent into the encoder, and the encoder encodes the mth frame of image according to the next group of encoding parameters, and so on, until the last group of encoding parameters and the mth frame of image are sent into the encoder.

[0315] In order to describe the above-mentioned cyclic process, a parameter i can be set, and i is equal to 1. Then i is increased by 1 each time the ith group of encoding parameters and the mth frame of image are sent into the encoder.

[0316] S303, i=1.

[0317] S304, the ith group of encoding parameters and the mth frame of image are input into the encoder.

[0318] Exemplarily, after the pre-processing module determines the ith group of encoding parameters, it can input the ith group of encoding parameters and the mth frame of image into the encoder, and then the encoder encodes the mth frame of image according to the ith group of encoding parameters to obtain the ith group of code stream data.

[0319] Exemplarily, the present application does not limit the order of inputting the ith group of encoding parameters and the mth frame of image into the encoder. For example, the ith group of encoding parameters and the mth frame of image can be input into the encoder at the same time. For another example, the ith group of encoding parameters can be input into the encoder first, and then the mth frame of image is input into the encoder. For another example, the mth frame of image is input into the encoder first, and then the ith group of encoding parameters is input into the encoder.

[0320] S305, whether i is equal to N is judged.

[0321] For example, the pre-processing module can determine whether i is equal to N after inputting the i-th set of encoding parameters into the encoder, that is, determine whether N sets of encoding parameters have been input into the encoder. If i is not equal to N, it means that part of the sets of encoding parameters have not been input into the encoder, and S306 can be performed. If i is equal to N, it means that N sets of encoding parameters have been input into the encoder, and S309 can be performed.

[0322] For example, the pre-processing module can also determine whether i is equal to N after receiving the encoding completion information fed back by the encoder, and the embodiments of the present application do not limit this.

[0323] S306, i is increased by 1.

[0324] For example, after S306 is performed, S304 can be performed again, that is, the next set of encoding parameters and the m-th frame of image are input into the encoder, and the m-th frame of image is encoded by the encoder at the next encoding scale.

[0325] S307, the encoding parameters corresponding to single-scale encoding are determined.

[0326] For example, when the pre-processing module determines that the m-th frame of image does not need to be multi-scale encoded, the encoding parameters corresponding to single-scale encoding can be determined.

[0327] For example, the encoding parameters corresponding to single-scale encoding can be pre-set, as shown in Table 14:

[0328] Table 14

[0329]

[0330] Further, the pre-set encoding parameters of single-scale encoding can be used as the encoding parameters corresponding to single-scale encoding of the m-th frame of image.

[0331] S308, the encoding parameters corresponding to single-scale encoding and the m-th frame of image are input into the encoder.

[0332] For example, the pre-processing module can input the encoding parameters corresponding to single-scale encoding and the m-th frame of image into the encoder, and the m-th frame of image is encoded by the encoder according to the encoding parameters corresponding to single-scale encoding, to obtain the single-scale encoded bitstream data.

[0333] S309, m is increased by 1.

[0334] For example, if i is equal to N or S308 is performed, m can be increased by 1, and the next frame of image can be input into the pre-processing module, and then the next frame of image is encoded according to S301-S308.

[0335] It should be noted that the coding scale N of different images in the video stream of the same video service can be the same or different, and part of the video data in the same video service can be multi-scale coded, and another part of the image can be single-scale coded.

[0336] For example, the pre-processing module can determine the frame type of the mth image while determining whether the mth image needs to be multi-scale coded according to the determination method described above. In addition, the pre-processing module can determine the scale identification information of the i-th coding scale while inputting the i-th set of coding parameters and the image into the encoder. Then, the pre-processing module can generate the i-th set of description information corresponding to the mth image according to the frame type of the mth image, the scale identification information of the i-th coding scale, and the frame identification, and output the i-th set of description information corresponding to the mth image to the data encapsulation module.

[0337] In this way, by combining the importance of the code stream data corresponding to each image to the codec and the impact on the network, the images in the video stream that need to be multi-scale coded are determined. For important and / or images that are easy to impact the network, multi-scale coding can be determined. When multi-scale coding is performed on the image, the image can be coded into a code stream data with a smaller data amount (i.e., a code stream data with a low coding scale), and one or more code stream data with a higher data amount (i.e., one or more code stream data with a higher coding scale). In this way, the code stream data with a low coding scale can have a higher probability of reaching the decoding end, ensuring smooth playback without screen tearing. For the code stream data with a high coding scale, if it is lost, it will not affect the normal decoding and playback of the playback end, and will not cause screen tearing. If it can reach the decoding end, it can also improve the picture quality of the playback. For images that are not important and / or have a small impact on the network, multi-scale coding is not performed. If the data is lost, it has a small impact on the image reconstruction in the decoding process and the user's viewing experience. In addition, since multi-scale coding can affect the coding compression efficiency and increase the coding calculation time, not performing multi-scale coding on part of the images can reduce the impact of multi-scale coding on the compression rate and calculation.

[0338] In one possible manner, the entire image can be coded with a low coding scale, and part of the image can be coded with a high coding scale, thereby reducing the calculation amount of the encoder for high coding scale coding and improving the coding efficiency.

[0339] Figure 4a For example, the data processing schematic diagram is shown in the figure. Referring to Figure 4a For example, the data processing process of the pre-processing module is as follows:

[0340] S401, determining whether the mth image needs to be multi-scale coded.

[0341] S402, determine the encoding scale N and N groups of encoding parameters for the mth frame image.

[0342] S403, let i = 1.

[0343] Exemplarily, S401-S403 can refer to S301-S303, which will not be repeated here.

[0344] S404, judge whether i is greater than the scale boundary threshold.

[0345] Exemplarily, the scale boundary threshold can be set in advance according to requirements, and the embodiments of the present application do not limit this.

[0346] Exemplarily, different scale boundary thresholds can be set in advance for different N values. For example, N = 1, and the scale boundary threshold can be set to 1. For another example, N = 3, and the scale boundary threshold can be set to 2. For another example, N = 4, and the scale boundary threshold can be set to 2.

[0347] Exemplarily, the scale boundary threshold can also be determined according to at least one of the encoding scale N, the image complexity of the mth frame image, the network transmission capability, and the parameters of the encoder. For example, assuming N = 4, if the network data transmission amount is less than a first data amount threshold, the scale boundary threshold can be determined to be 2. If the network data transmission amount is greater than the first data amount threshold, the scale boundary threshold can be determined to be 3. For another example, assuming N = 4, if the image complexity of the mth frame image is lower than a first complexity threshold, the scale boundary threshold can be determined to be 2. If the image complexity of the mth frame image is higher than the first complexity threshold, the scale boundary threshold can be determined to be 3. The first complexity threshold can be set according to requirements, and the embodiments of the present application do not limit this.

[0348] Exemplarily, when the current encoding scale is less than or equal to the scale boundary threshold, the entire frame image can be encoded, and when the current encoding scale is greater than the scale boundary threshold, part of the region in the entire frame image can be encoded.

[0349] Exemplarily, the i-th group of encoding parameters corresponds to the i-th encoding scale, and the larger i is, the higher the encoding scale is.

[0350] Exemplarily, when i is less than or equal to the scale boundary threshold, S405 can be executed. When i is greater than the scale boundary threshold, S406 can be executed.

[0351] S405, input the i-th group of encoding parameters and the mth frame image to the encoder.

[0352] Exemplarily, S405 can refer to S304 in the foregoing, which will not be repeated here.

[0353] S406, determine the target region in the mth frame image, and extract the image of the target region from the mth frame image.

[0354] For example, when it is determined that the quantization parameter in the ith group of encoding parameters is not the maximum value among the quantization parameters in the N groups of encoding parameters, it can be determined that the target region in the mth frame image is to be encoded.

[0355] For example, the pre-processing module can analyze the mth frame image, determine the region of interest of the user, and determine the region of interest of the user as the target region.

[0356] For example, the pre-processing module can perform subject detection on the mth frame image, and determine the region where the detected subject is located as the target region. For example, the region where the human face is located, the region where the animal is located, and the like.

[0357] For example, after the pre-processing module determines the target region, the pre-processing module can extract the image of the target region from the mth frame image.

[0358] For example, after the pre-processing module extracts the image of the target region from the mth frame image, the pre-processing module can discard the images of other regions of the mth frame image.

[0359] S407, input the ith group of encoding parameters and the image of the target region of the mth frame image to the encoder.

[0360] For example, the pre-processing module can input the ith group of encoding parameters and the image of the target region of the mth frame image to the encoder, and then the encoder can encode the image of the target region in the mth frame image at the ith encoding scale to obtain the code stream data corresponding to the image of the target region in the mth frame image.

[0361] S408, determine whether i is equal to N.

[0362] S409, add 1 to i.

[0363] S410, determine the encoding parameters corresponding to the single-scale encoding.

[0364] S411, input the encoding parameters corresponding to the single-scale encoding and the mth frame image to the encoder.

[0365] S412, add 1 to m.

[0366] For example, S408-S412 can refer to S305-S309 described above, and will not be described here.

[0367] For example, the method for determining the description information by the pre-processing module can refer to the description above, and will not be described here.

[0368] Figure 4bThe data processing schematic diagram is shown for example. Referring to Figure 4b , for example, the pre-processing module can include a multi-scale coding decision module and an image pre-processing module. After the multi-scale coding decision module performs S401, if it is determined that the mth frame of image needs to be multi-scale coded, S402-S404 can be performed. If i is less than or equal to the scale boundary threshold, on the one hand, the i-th group of coding parameters is input to the encoder in S407, and on the other hand, the mth frame of image is input to the image pre-processing module, and the mth frame of image is input to the encoder in S407 by the image pre-processing module. If i is greater than the scale boundary threshold, on the one hand, the multi-scale coding decision module can input the i-th group of coding parameters and the image to the image pre-processing module, and then the image pre-processing module can perform S405 and input the image of the target region in the mth frame of image to the encoder in S406. On the other hand, the multi-scale coding decision module can perform S406 to input the i-th group of coding parameters to the encoder. If the multi-scale coding decision module determines that the mth frame of image does not need to be multi-scale coded, S410 can be performed, and on the one hand, S411 can be performed to input the coding parameters corresponding to non-scale coding to the encoder, and on the other hand, the mth frame of image can be input to the image pre-processing module, and the mth frame of image is input to the encoder in S411 by the image pre-processing module. Then S408, S409 and S412 are performed by the multi-scale coding decision module. That is, the image pre-processing module is used to perform the steps of processing the mth frame of image in S401-S412 and inputting the processed mth frame of image or the mth frame of image to the encoder, and the multi-scale decision module is used to perform the steps in S401-S412 except the steps performed by the image pre-processing module, and perform the steps of inputting the i-th group of coding parameters and the mth frame of image to the image pre-processing module.

[0369] In addition, after the multi-scale coding decision module determines the i-th group of description information of the mth frame of image, the i-th group of description information of the mth frame of image can also be input to the data encapsulation module, which can be referred to as the description above, and will not be repeated here.

[0370] For example, S401-S412 can also be performed by the pre-processing module, that is, the pre-processing module is not divided into a multi-scale coding decision module and an image pre-processing module, as shown in Figure 2 .

[0371] Continuing to refer to Figure 4b , for example, if the mth frame of image is multi-scale coded, the decoder reconstructed image received by the post-processing module includes the reconstructed image of the mth frame of image at the lowest coding scale and the reconstructed image of the target region in the mth frame of image at other coding scales.

[0372] For example, when the post-processing module selects the reconstructed image in other coding scales to output to the display module, the reconstructed image of the target region in the mth frame image in other coding scales can be used to replace the image of the target region in the reconstructed image of the mth frame image in the lowest coding scale, to obtain the reconstructed image of the mth frame image in other coding scales; and then the reconstructed image of the mth frame image in other coding scales is output to the display module. Then the display module displays the reconstructed image of the mth frame image in other coding scales.

[0373] Figure 4c The data processing schematic diagram is exemplarily shown. Referring to Figure 4c , the video data includes e (e is a positive integer) frame images, such as image 1, image 2, …, image e.

[0374] Referring to Figure 4c (1), for example, the multi-scale coding module encodes the image 1 in three coding scales to obtain the code stream data in three coding scales.

[0375] Referring to Figure 4c (1), for example, the multi-scale coding module encodes the image 1 according to the coding parameters corresponding to the first coding scale to obtain the code stream data in the first coding scale. Referring to Figure 4c (2), for example, the code stream data corresponding to the first coding scale of the image 1 is decoded to obtain the image 1', which is the reconstructed image corresponding to the image 1.

[0376] Referring to Figure 4c (1), for example, the multi-scale coding module first extracts the target region R2 from the image 1, and then encodes R2 according to the coding parameters corresponding to the second coding scale to obtain the code stream data in the second coding scale. Referring to Figure 4c (2), for example, the code stream data corresponding to the second coding scale of the image 1 is decoded to obtain R2', which is the reconstructed image corresponding to R2.

[0377] Referring to Figure 4c (1), for example, the multi-scale coding module first extracts the target region R3 from the image 1, and then encodes R3 according to the coding parameters corresponding to the third coding scale to obtain the code stream data in the third coding scale. Referring to Figure 4c (2), for example, the code stream data corresponding to the third coding scale of the image 1 is decoded to obtain R3', which is the reconstructed image corresponding to R3.

[0378] Referring to Figure 4c(2), exemplary, for image 1, when the first encoding scale of the reconstructed image output is selected, the image 1' can be directly output to the display module for display. When the second encoding scale of the reconstructed image output is selected, the image 1' can be replaced by R2' in the target region of the image, and then the replaced image 1' is output. When the third encoding scale of the reconstructed image output is selected, the image 1' can be replaced by R3' in the target region of the image, and then the replaced image 1' is output.

[0379] Referring to Figure 4c (1), exemplary, for image 2, the multi-scale encoding module encodes the image 2 in two encoding scales to obtain two encoding scale of the code stream data.

[0380] Referring to Figure 4c (1), exemplary, the multi-scale encoding module encodes the image 2 according to the encoding parameters corresponding to the first encoding scale to obtain the first encoding scale of the code stream data. Referring to Figure 4c (2), exemplary, for image 2, the code stream data corresponding to the first encoding scale is decoded to obtain image 2', which is the reconstructed image corresponding to image 2.

[0381] Referring to Figure 5a (1), exemplary, the multi-scale encoding module first extracts the target region Q2 from the image 2, and then encodes Q2 according to the encoding parameters corresponding to the second encoding scale to obtain the second encoding scale of the code stream data. Referring to Figure 5a (2), exemplary, for image 2, the code stream data corresponding to the second encoding scale is decoded to obtain Q2', which is the reconstructed image corresponding to Q2.

[0382] Referring to Figure 4b (2), exemplary, for image 2, when the first encoding scale of the reconstructed image output is selected, the image 2' can be directly output to the display module for display. When the second encoding scale of the reconstructed image output is selected, the image 2' can be replaced by Q2' in the target region of the image, and then the replaced image 2' is output.

[0383] Exemplary, the multi-scale encoding module encodes the image e in a single scale to obtain the corresponding code stream data. Then the code stream data corresponding to the image e can be decoded to obtain the image e', which is the reconstructed image corresponding to the image e, and then the image e' can be directly output for display.

[0384] In one possible way, the encoding parameters can include: resolution, quantization parameter and reference frame information.

[0385] Figure 2 The processing process schematic diagram is shown for example. Referring toFigure 5b For example, when the existing encoder in the sending device supports the encoding of images with different resolutions, and the existing decoder in the receiving device supports the decoding of images with different resolutions, the data processing process of the pre-processing module is as follows:

[0386] S501, determine whether the mth frame of image needs to be multi-scale coded.

[0387] For example, after the pre-processing module receives the mth frame of image input by the image acquisition module, it can determine whether the image needs to be multi-scale coded according to the coding importance of other frames in the video data to which the mth frame of image belongs, and / or the impact of the code stream data corresponding to the mth frame of image on the network, and / or the predicted image quality of the mth frame of image. When the coding importance of other frames in the video data to which the mth frame of image belongs meets the first preset condition, and / or the impact of the code stream data corresponding to the mth frame of image on the network meets the second preset condition, and the predicted image quality of the mth frame of image meets the third preset condition, it is determined that the image needs to be multi-scale coded.

[0388] For example, the first preset condition includes at least one of the following:

[0389] The mth frame of image is an I frame or an IDR frame;

[0390] The mth frame of image is a scene switching frame;

[0391] The mth frame of image is a reference frame;

[0392] The mth frame of image is a key frame.

[0393] For example, the way of determining whether the mth frame of image needs to be multi-scale coded according to the coding importance of other frames in the video data to which the mth frame of image belongs can refer to the example described above, and will not be repeated here.

[0394] For example, the second preset condition includes at least one of the following:

[0395] The data amount of the code stream data of the mth frame of image is greater than a preset data amount threshold;

[0396] The data amount of the code stream data of the mth frame of image is greater than the network transmission capacity.

[0397] For example, the way of determining whether the mth frame of image needs to be multi-scale coded according to the impact of the code stream data corresponding to the mth frame of image on the network can refer to the example described above, and will not be repeated here.

[0398] For example, the predicted picture quality of the m-th frame image includes a first predicted picture quality and a second predicted picture quality, the first predicted picture quality is a picture quality of the m-th frame image encoded at a source resolution of the image, and the second predicted picture quality is a picture quality of the m-th frame image encoded at a resolution lower than the source resolution. The third preset condition includes that the first predicted picture quality is lower than or equal to the second predicted picture quality.

[0399] For example, the first predicted picture quality of the m-th frame image encoded at the source resolution can be predicted according to the image complexity of the m-th frame image and a first target data amount, and the second predicted picture quality of the m-th frame image encoded at the resolution lower than the source resolution can be predicted according to the image complexity of the m-th frame image and a second target data amount. For example, the first target data amount and the second target data amount can be two values close to each other. Alternatively, a difference between the first target data amount and the second target data amount is less than a preset data amount value, and the preset data amount value can be set as required.

[0400] For example, the first predicted picture quality of the m-th frame image encoded at the source resolution of 1920*1080 is 500K, and the second predicted picture quality of the m-th frame image encoded at the resolution of 1280*720 is 500K, which are predicted according to the image complexity of the m-th frame image. If the first predicted picture quality of the m-th frame image encoded at the source resolution of 1920*1080 is higher than the second predicted picture quality of the m-th frame image encoded at the resolution of 1280*720, it can be determined that the m-th frame image does not need to be encoded in multiple scales. If the first predicted picture quality of the m-th frame image encoded at the source resolution of 1920*1080 is lower than or equal to the second predicted picture quality of the m-th frame image encoded at the resolution of 1280*720, it can be determined that the m-th frame image needs to be encoded in multiple scales.

[0401] For example, when it is determined that the m-th frame image needs to be encoded in multiple scales, S502 can be performed. When it is determined that the m-th frame image does not need to be encoded in multiple scales, S510 can be performed.

[0402] S502, determining an encoding scale N and N sets of encoding parameters for the m-th frame image.

[0403] For example, the manner of determining the encoding scale N for the m-th frame image can refer to the description of S302 above, which will not be described here again.

[0404] In one possible manner, for each N value, the encoding parameters of each encoding scale in the N encoding scales can be preset. For example, for N=2 in Table 1, the encoding parameters corresponding to two encoding scales can be preset, as shown in Table 15:

[0405] Table 15

[0406]

[0407] For example, for N=3 in Table 1, three kinds of encoding parameters corresponding to three encoding scales can be preset, as shown in Table 16:

[0408] Table 16

[0409]

[0410] For example, for N=4 in Table 1, four kinds of encoding parameters corresponding to four encoding scales can be preset, as shown in Table 17:

[0411] Table 17

[0412]

[0413]

[0414] For example, in the resolution column of Table 15 to Table 17, "source resolution" refers to the resolution of the mth image. "Source resolution / 4" means that the resolution is one fourth of the source resolution, i.e., the horizontal pixels are one half of the horizontal pixels of the source resolution, and the vertical pixels are one half of the vertical pixels of the source resolution. "Source resolution / 8" means that the resolution is one eighth of the source resolution, i.e., the horizontal pixels are one fourth of the horizontal pixels of the source resolution, and the vertical pixels are one half of the vertical pixels of the source resolution. "Source resolution / 16" means that the resolution is one sixteenth of the source resolution, i.e., the horizontal pixels are one fourth of the horizontal pixels of the source resolution, and the vertical pixels are one fourth of the vertical pixels of the source resolution.

[0415] It should be noted that although the encoding parameters corresponding to the first encoding scale shown in Table 15, Table 16 and Table 17 are the same, it should be understood that the encoding parameters corresponding to the first encoding scale in Table 15, Table 16 and Table 17 can be different. And although the encoding parameters corresponding to the second encoding scale shown in Table 16 and Table 17 are the same, it should be understood that the encoding parameters corresponding to the second encoding scale in Table 16 and Table 17 can be different, and so on. That is, when the value of the encoding scale N is different, the encoding parameters corresponding to the same encoding scale can be the same or different, and the embodiments of the present application do not limit this.

[0416] For example, the pre-processing module can find the encoding parameters corresponding to N from the preset encoding parameters according to the encoding scale N corresponding to the mth image.

[0417] For example, the encoding scale N corresponding to the mth image is 2, and the corresponding encoding parameters can be as shown in Table 15.

[0418] For example, the encoding scale N corresponding to the m-th frame of image is 3, and the corresponding encoding parameters can be shown in Table 16.

[0419] In one possible manner, the encoding parameters corresponding to each encoding scale can be determined according to at least two of the encoding scale N, the image complexity of the m-th frame of image, the current network transmission capability, and the parameter value range of the encoder.

[0420] For example, the encoding scale N corresponding to the m-th frame of image is 3, and the corresponding encoding parameters can be shown in Table 16.

[0421] For example, N = 2, and the parameter value range of the encoder satisfies the first encoding condition, and the encoding parameters are shown in Table 18.

[0422] Table 18

[0423]

[0424] For example, N = 2, and the parameter value range of the encoder satisfies the second encoding condition, and the encoding parameters are shown in Table 19.

[0425] Table 19

[0426]

[0427] The encoding quality of the encoder when the parameter value range of the encoder satisfies the second encoding condition is greater than the encoding quality of the encoder when the parameter value range of the encoder satisfies the first encoding condition.

[0428] For example, the first encoding condition and the second encoding condition can be set according to the encoding quality of the encoder when each parameter of the encoder takes different range values, and details are not described herein.

[0429] For example, the encoding scale N corresponding to the m-th frame of image is 3, and the corresponding encoding parameters can be shown in Table 16.

[0430] Table 20

[0431]

[0432] For example, the encoding scale N corresponding to the m-th frame of image is 3, and the corresponding encoding parameters can be shown in Table 16.

[0433] For example, N = 2, and the network data transmission amount is greater than the first data amount threshold, and the encoding parameters are shown in Table 21.

[0434] Table 21

[0435]

[0436] For example, N=2, the network data transmission volume is greater than the second data volume threshold, and the encoding parameters are shown in Table 22:

[0437] Table 22

[0438]

[0439] The second data volume threshold is greater than the first data volume threshold, and the first quantity threshold and the second data volume threshold can be set as required, and the embodiments of the present application do not limit this.

[0440] For example, N=3, the network data transmission volume is greater than the first data volume threshold, and the encoding parameters are shown in Table 23:

[0441] Table 23

[0442]

[0443] It should be understood that the encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N and the image complexity. The encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N, the image complexity, and the parameter value range of the encoder. The encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N, the image complexity, and the network transmission capability. The encoding parameters corresponding to each encoding scale can also be determined according to the encoding scale N, the image complexity, the parameter value range of the encoder, and the network transmission capability; and the like, which will not be described one by one here.

[0444] For example, the way in which the pre-processing module determines the reference frame information of the mth image can also refer to the description in the foregoing, which will not be described here.

[0445] S503, let i=1.

[0446] S504, judge whether the resolution in the i-th group of encoding parameters is less than the source resolution of the mth image.

[0447] For example, when the resolution in the i-th group of encoding parameters is less than the source resolution of the mth image, the mth image can be down-sampled first and then input into the encoder, that is, S505 is executed first, and then S506 is executed.

[0448] For example, when the resolution in the i-th group of encoding parameters is equal to the source resolution of the mth image, the mth image does not need to be down-sampled, and the image can be directly input into the encoder, that is, S507 is executed.

[0449] S505, down-sample the mth image according to the resolution in the i-th group of encoding parameters.

[0450] For example, if the resolution in the i-th set of encoding parameters is less than the source resolution of the m-th frame of image, the pre-processing module can down-sample the m-th frame of image according to the resolution in the i-th set of encoding parameters.

[0451] For example, if the source resolution of the m-th frame of image is 1920*1080 and the resolution in the i-th set of encoding parameters is 1280*720, the pre-processing module can down-sample the m-th frame of image to obtain an image with a resolution of 1280*720.

[0452] S506, input the i-th set of encoding parameters and the down-sampled m-th frame of image to the encoder.

[0453] S507, input the i-th set of encoding parameters and the m-th frame of image to the encoder.

[0454] For example, if the resolution in the i-th set of encoding parameters is equal to the source resolution of the m-th frame of image, the i-th set of encoding parameters and the m-th frame of image can be directly input to the encoder. For details, refer to the description of S304, which will not be repeated here.

[0455] S508, determine whether i is equal to N.

[0456] For example, S508 can refer to the description of S305, which will not be repeated here.

[0457] S509, i is incremented by 1.

[0458] For example, after S509 is executed, S504-S507 can be executed again, that is, the encoding parameters of the next encoding scale are input to the encoder together with the down-sampled m-th frame of image or the m-th frame of image, and the m-th frame of image is encoded by the encoder at the next encoding scale.

[0459] S510, determine the encoding parameters corresponding to single-scale encoding.

[0460] S511, input the encoding parameters corresponding to single-scale encoding and the m-th frame of image to the encoder.

[0461] S512, m is incremented by 1.

[0462] For example, S510-S512 can refer to S307-S309 described above, which will not be repeated here.

[0463] It should be noted that the encoding scales N of different images in the video stream of the same video service can be the same or different, and part of the images in the video stream of the same video service can be multi-scale encoded, and another part of the images can be single-scale encoded. The embodiments of the present application do not limit this.

[0464] For example, the pre-processing module determines the i-th set of description information in the manner described above, which will not be repeated here.

[0465] With reference to Figure 5b , for example, the image pre-processing module is configured to perform the steps of S501-S512 on the m-th frame of image, and input the processed m-th frame of image or the m-th frame of image to the encoder, and the multi-scale decision module is configured to perform the steps of S501-S512 except the steps performed by the image pre-processing module, and perform the step of inputting the i-th set of encoding parameters and the m-th frame of image to the image pre-processing module, which can be referred to the description above, and will not be repeated here.

[0466] In addition, after the multi-scale encoding decision module determines the i-th set of description information of the m-th frame of image, the multi-scale encoding decision module can further input the i-th set of description information of the m-th frame of image to the data encapsulation module, which can be referred to the description above, and will not be repeated here.

[0467] For example, the steps of S501-S512 can also be performed by the pre-processing module, i.e. the pre-processing module is not divided into the multi-scale encoding decision module and the image pre-processing module, as shown in Figure 5b .

[0468] Figure 5b The encoding schematic diagram is shown for example. With reference to Figure 5b , the video stream includes e (e is a positive integer) frames of images, such as image 1, image 2, …, image e.

[0469] With reference to Figure 5b (1), for example, the multi-scale encoding module encodes the image 1 in three encoding scales to obtain three sets of code stream data of the three encoding scales.

[0470] With reference to Figure 5b (1), for example, the multi-scale encoding module can first sample the image 1 according to the resolution in the encoding parameters corresponding to the first encoding scale to obtain image 1A. Then, the multi-scale encoding module encodes the image 1A according to the encoding parameters corresponding to the first encoding scale to obtain the code stream data of the first encoding scale. With reference to Figure 5b (2), for example, the code stream data of the first encoding scale corresponding to the image 1 is decoded to obtain image 1a, which is the reconstructed image corresponding to the image 1A.

[0471] With reference to Figure 5b (1), for example, the multi-scale encoding module can first sample the image 1 according to the resolution in the encoding parameters corresponding to the second encoding scale to obtain image 1B. Then, the multi-scale encoding module encodes the image 1B according to the encoding parameters corresponding to the second encoding scale to obtain the code stream data of the first encoding scale. With reference toFigure 5b (2), exemplary, decode the bitstream data corresponding to the second encoding scale for image 1 to obtain image 1b, which is the reconstructed image corresponding to image 1B.

[0472] Referring to Figure 5b (1), exemplary, the multi-scale encoding module encodes image 1 according to the encoding parameters corresponding to the third encoding scale to obtain the bitstream data corresponding to the third encoding scale. Referring to Figure 5b (2), exemplary, decode the bitstream data corresponding to the third encoding scale for image 1 to obtain image 1', which is the reconstructed image corresponding to image 1.

[0473] Exemplary, one frame of image can be selected from image 1a, image 1b and image 1' for display.

[0474] Exemplary, the multi-scale encoding module encodes image 2 according to two encoding scales to obtain the bitstream data corresponding to the two encoding scales.

[0475] Referring to Figure 5b (1), exemplary, the multi-scale encoding module can first sample image 2 according to the resolution in the encoding parameters corresponding to the first encoding scale to obtain image 2A. Then encode image 2A according to the encoding parameters corresponding to the first encoding scale to obtain the bitstream data corresponding to the first encoding scale. Referring to Figure 6a (2), exemplary, decode the bitstream data corresponding to the first encoding scale for image 2 to obtain image 2a, which is the reconstructed image corresponding to image 2A.

[0476] Referring to Figure 6a (1), exemplary, the multi-scale encoding module encodes image 2 according to the encoding parameters corresponding to the second encoding scale to obtain the bitstream data corresponding to the second encoding scale. Referring to Figure 4b (2), exemplary, decode the bitstream data corresponding to the second encoding scale for image 2 to obtain image 2', which is the reconstructed image corresponding to image 2.

[0477] Exemplary, one frame of image can be selected from image 2a and image 2' for display.

[0478] Exemplary, the multi-scale encoding module encodes image e according to a single scale to obtain the corresponding bitstream data. Then the bitstream data corresponding to image e can be decoded to obtain image e', which is the reconstructed image corresponding to image e, and then image e' can be directly output for display.

[0479] In this way, by combining the importance in coding of the code stream data corresponding to each frame of image and the impact on the network, the image in the video stream that needs multi-scale coding is determined. For the important and / or easy to impact the network image, it can be determined that multi-scale coding is needed, wherein the image of this kind can be encoded into a code stream data with small data amount, and the image of this kind can be encoded into one or more data amount. Further, the code stream data of low coding scale can have a higher probability to reach the decoding end, ensuring the smoothness of the entire playback, and no screen tearing. For the code stream data of high coding scale, if lost, it will not affect the normal decoding and playback of the playback end, and no screen tearing will be generated; if it can reach the decoding end, it can also improve the picture quality of the playback. For the unimportant and / or small impact on the network image, no multi-scale coding is performed, and the influence of the data loss on the decoding reconstruction and the subjective influence of the user are small.

[0480] In addition, since multi-scale coding will affect the coding compression efficiency and increase the coding calculation time, not performing multi-scale coding on part of the images can reduce the influence of multi-scale coding on the compression rate and calculation.

[0481] In addition, when the picture quality of the image encoded at the source resolution is lower than or equal to the picture quality of the image encoded at a resolution lower than the source resolution, the resolution of the image encoding can be reduced to reduce the code rate of the corresponding code stream data, thereby reducing the blocking effect at low code rate, improving the picture quality, and also reducing the calculation amount of the encoder.

[0482] It should be noted that when the encoder has a downsampling function, the pre-processing module can not need to downsample the image, but can input the source resolution, the target coding resolution, the quantization parameter, and the reference frame information into the encoder, and the encoder can determine whether the image needs to be downsampled according to the source resolution and the target coding resolution. When the encoder determines that the image needs to be downsampled according to the source resolution and the target coding resolution, the image can be downsampled according to the target coding resolution, and then the encoder can encode the downsampled image according to other parameters in the coding parameter. When the encoder determines that the image does not need to be downsampled according to the source resolution and the target coding resolution, the image can be encoded according to other parameters in the coding parameter.

[0483] In one possible way, the entire frame of image can be encoded at a low coding scale, and part of the region in the image can be encoded at a high coding scale, thereby reducing the calculation amount of the encoder for high coding scale encoding and improving the coding efficiency.

[0484] Figure 4b To illustrate the processing process diagram. Referring to Figure 6bFor example, when the existing encoder in the sending device supports the encoding of images with different resolutions and the existing decoder in the receiving device supports the decoding of images with different resolutions, the data processing procedure of the pre-processing module is as follows:

[0485] S601: determining whether the mth image needs to be encoded in multiple scales.

[0486] S602: determining the encoding scale N and the N sets of encoding parameters for the mth image.

[0487] S603: setting i = 1.

[0488] For example, S601-S603 can refer to the description of S301-S303 above, and will not be described here again.

[0489] S604: determining whether the resolution in the ith set of encoding parameters is less than the source resolution of the mth image.

[0490] For example, if the resolution in the ith set of encoding parameters is less than the source resolution of the mth image, S605 is executed. If the resolution in the ith set of encoding parameters is equal to the source resolution of the mth image, S610 is executed.

[0491] S605: down-sampling the mth image according to the resolution in the ith set of encoding parameters.

[0492] For example, S605 can refer to the description of S505 above, and will not be described here again.

[0493] S606: determining whether i is greater than the scale boundary threshold.

[0494] For example, if i is less than or equal to the scale boundary threshold, S607 can be executed. If i is greater than the scale boundary threshold, S608 can be executed.

[0495] For example, S606 can refer to the description of S604, and will not be described here again.

[0496] For example, the embodiments of the present application do not limit the execution order of S604 and S606.

[0497] S607: inputting the ith set of encoding parameters and the down-sampled mth image to the encoder.

[0498] For example, S607 can refer to S506 above, and will not be described here again.

[0499] S608: determining a target region in the down-sampled mth image, and extracting the image of the target region from the down-sampled mth image.

[0500] S609, input the i-th group of encoding parameters and the image of the target region of the m-th frame image after down-sampling to the encoder.

[0501] For example, S608-S609 can refer to the description in S405-S406 above, and will not be repeated here.

[0502] S610, determine the target region in the m-th frame image, and extract the image of the target region from the m-th frame image.

[0503] S611, input the i-th group of encoding parameters and the image of the target region of the m-th frame image to the encoder.

[0504] For example, S610-S611 can refer to the description in S405-S406 above, and will not be repeated here.

[0505] S612, determine whether i is equal to N.

[0506] S613, add 1 to i.

[0507] S614, determine the encoding parameters corresponding to the single-scale encoding.

[0508] S615, input the encoding parameters corresponding to the single-scale encoding and the m-th frame image to the encoder.

[0509] S616, add 1 to m.

[0510] For example, S612-S616 can refer to S305-S309 above, and will not be repeated here.

[0511] With reference to Figure 6b , the image preprocessing module is configured to perform the steps of processing the m-th frame image in S601-S616 and inputting the processed m-th frame image or the m-th frame image to the encoder, and the multi-scale decision module is configured to perform the steps in S601-S616 other than the steps performed by the image preprocessing module and input the i-th group of encoding parameters and the m-th frame image to the image preprocessing module, which can refer to the description above, and will not be repeated here.

[0512] With reference to Figure 6b , for example, if the m-th frame image is multi-scale encoded, the decoder reconstructed image received by the post-processing module includes the reconstructed image of the m-th frame image at the lowest encoding scale and the reconstructed image of the target region at other encoding scales.

[0513] For example, when the post-processing module selects the m-th frame image to be reconstructed at the i-th encoding scale (any encoding scale except the lowest encoding scale) and output to the display module, the reconstructed image of the m-th frame image at the lowest encoding scale can be up-sampled to obtain an up-sampled image IMG UP0 according to the resolution of the i-th encoding scale. The reconstructed image of the target region at the i-th encoding scale is used to replace the image of the target region in the up-sampled image IMG UP0 to obtain the reconstructed image of the m-th frame image at the i-th encoding scale. Then the reconstructed image of the m-th frame image at the i-th encoding scale is output to the display module for display.

[0514] Figure 6b The processing process is schematically shown by way of example. Referring to Figure 6b , the video stream includes e (e is a positive integer) frame images, such as image 1, image 2, …, image e.

[0515] Referring to Figure 6b (1), for example, the multi-scale encoding module encodes image 1 at three encoding scales to obtain code stream data of the three encoding scales.

[0516] Referring to Figure 1 (1), for example, for image 1, the multi-scale encoding module can first down-sample image 1 according to the resolution in the encoding parameters corresponding to the first encoding scale to obtain image 1a. Then image 1a is encoded according to the encoding parameters corresponding to the first encoding scale to obtain code stream data of the first encoding scale. Referring to Figure 6b (2), for example, the code stream data of the first encoding scale corresponding to image 1 is decoded to obtain image 1a', which is the reconstructed image corresponding to image 1a.

[0517] Referring to Figure 6b (1), for example, for image 1, the multi-scale encoding module can first down-sample image 1 according to the resolution in the encoding parameters corresponding to the second encoding scale to obtain image 1b. Then Figure 6b b is extracted from image 1b, and R2 is encoded according to the encoding parameters corresponding to the second encoding scale to obtain code stream data of the second encoding scale. Referring to Figure 6b (2), for example, the code stream data of the second encoding scale corresponding to image 1 is decoded to obtain R2', which is the reconstructed image corresponding to R2.

[0518] Referring to Figure 6b(1), exemplary, for image 1, the multi-scale coding module can first extract the image R3 of the target region from image 1, and then encode R3 according to the coding parameters corresponding to the third coding scale to obtain the code stream data of the third coding scale. Referring to Figure 6b (2), exemplary, for image 1, the code stream data corresponding to the third coding scale is decoded to obtain R3', and R3' is the reconstructed image corresponding to R3.

[0519] Referring to Figure 6b (2), exemplary, for image 1, when the reconstructed image output of the first coding scale is selected, image 1a' can be directly output to the image display module for display. When the reconstructed image output of the second coding scale is selected, image 1a' can be up-sampled according to the resolution in the coding parameters corresponding to the second coding scale to obtain image 1b'. Wherein, the resolution of image 1b' is the same as that of image 1b. Then R2' is used to replace the image of the target region in image 1b', and the replaced image 1b' is output. When the reconstructed image output of the third coding scale is selected, image 1a' can be up-sampled according to the resolution in the coding parameters corresponding to the third coding scale to obtain image 1'. Wherein, the resolution of image 1' is the same as that of image 1. Then R3' is used to replace the image of the target region in image 1', and the replaced image 1' is output.

[0520] Referring to Figure 6b (1), exemplary, the multi-scale coding module encodes image 2 in two coding scales to obtain code stream data of two coding scales.

[0521] Referring to Figure 6b (1), exemplary, for image 2, the multi-scale coding module can first down-sample image 2 according to the resolution in the coding parameters corresponding to the first coding scale to obtain image 2a. Then image 2a is encoded according to the coding parameters corresponding to the first coding scale to obtain the code stream data of the first coding scale. Referring to Figure 6b (2), exemplary, for image 2, the code stream data corresponding to the first coding scale is decoded to obtain image 2a', and image 2a' is the reconstructed image corresponding to image 2a.

[0522] Referring to Figure 7a (1), exemplary, for image 2, the multi-scale coding module can first extract the target region Q2 from image 2, and then encode Q2 according to the coding parameters corresponding to the second coding scale to obtain the code stream data of the second coding scale. Referring to Figure 7a (2), exemplary, for image 2, the code stream data corresponding to the third coding scale is decoded to obtain Q2', and Q2' is the reconstructed image corresponding to Q2.

[0523] Referring to Figure 7b (2), for example, for image 2, when the reconstructed image output of the first encoding scale is selected, image 2a' can be directly output to the image display module for display. When the reconstructed image output of the second encoding scale is selected, image 2a' can be up-sampled according to the resolution in the corresponding encoding parameters of the second encoding scale to obtain image 2'. Wherein, the resolution of image 2' is the same as that of image 2. Then Q2' is used to replace the image in the target region of image 2', and the replaced image 2' is output.

[0524] For example, the multi-scale encoding module encodes the image e to obtain the corresponding code stream data. Then the code stream data corresponding to the image e can be decoded to obtain the image e', which is the reconstructed image corresponding to the image e, and then the image e' can be directly output for display.

[0525] For example, when the existing encoder in the sending device does not support the encoding of images with different resolutions, and the existing decoder in the receiving device does not support the decoding of images with different resolutions, the processing method of the multi-scale encoding module and the multi-scale decoding module can be as follows:

[0526] Figure 7b The device structure diagram shown is exemplary. Referring to Figure 4b , the sending device can include N encoders, and the receiving device can include N decoders. For example, each encoder in the sending device supports different resolutions, and the resolution supported by an encoder can correspond to the resolution in a set of encoding parameters. For example, each decoder in the receiving device supports different resolutions, and the resolution supported by a decoder can correspond to the resolution in a set of encoding parameters.

[0527] Figure 7a The processing process diagram shown is exemplary. Referring to Figure 7c , for example, when the existing encoder in the sending device does not support the encoding of images with different resolutions, and the existing decoder in the receiving device does not support the decoding of images with different resolutions, the data processing process of the pre-processing module is as follows:

[0528] S701, determine whether the mth frame of image needs to be multi-scale encoded.

[0529] S702, determine the encoding scale N and the N sets of encoding parameters for the mth frame of image.

[0530] For example, S701-S702 can refer to S501-S502 described above, which will not be repeated here.

[0531] S703, down-sampling the m-th image according to the resolution in the J-th group of encoding parameters to obtain a down-sampled image IMG DW1, the J-th group of encoding parameters corresponding to the lowest encoding scale.

[0532] For example, the encoding of the high encoding scale can be based on the encoding of the low encoding scale. For example, the m-th image can be encoded at the high encoding scale based on the code stream data obtained by encoding the m-th image at the lowest encoding scale.

[0533] For example, the group of encoding parameters with the lowest resolution can be searched from the N groups of encoding parameters as the encoding parameters corresponding to the lowest encoding scale. For the convenience of description, the encoding parameters corresponding to the lowest encoding scale can be referred to as the J-th group of encoding parameters. Then the m-th image can be down-sampled according to the resolution in the J-th group of encoding parameters to obtain a down-sampled image. For the convenience of distinguishing, the down-sampled image obtained by down-sampling the m-th image according to the resolution in the encoding parameters corresponding to the lowest encoding scale can be referred to as IMG DW1.

[0534] S704, inputting the J-th group of encoding parameters and the down-sampled image IMG DW1 to the J-th encoder.

[0535] For example, the resolution supported by the J-th encoder is the same as the resolution in the J-th group of encoding parameters.

[0536] For example, after the m-th image is encoded at the lowest encoding scale, the m-th image can be encoded at other encoding scales. S705-S712 can be referred to.

[0537] For example, in addition to the J-th group of encoding parameters, there are (N-1) groups of encoding parameters in the N groups of encoding parameters. S705-S712 can select one group of encoding parameters and the corresponding image to be encoded from the (N-1) groups of encoding parameters each time and input them to the corresponding encoder for encoding.

[0538] S705, let i be equal to 1.

[0539] S706, up-sampling the down-sampled image IMG DW1 according to the resolution in the i-th group of encoding parameters to obtain an up-sampled image IMG UP1.

[0540] For example, the down-sampled image IMG DW1 can be up-sampled by interpolation according to the resolution in the i-th group of encoding parameters to obtain the up-sampled image IMG UP1. The application does not limit the up-sampling method.

[0541] S707, determining whether the resolution in the i-th group of encoding parameters is less than the source resolution of the m-th image.

[0542] For example, if the resolution in the i-th set of encoding parameters is less than the source resolution of the m-th frame of image, S708 is performed. If the resolution in the i-th set of encoding parameters is equal to the source resolution of the m-th frame of image, S710 is performed.

[0543] S708, down-sampling the m-th frame of image according to the resolution in the i-th set of encoding parameters to obtain a down-sampled image IMG_DW2.

[0544] For example, the m-th frame of image can be down-sampled according to the resolution in the i-th set of encoding parameters to obtain a down-sampled image IMG_DW2.

[0545] S709, determining a difference image of the up-sampled image IMG_UP1 and the down-sampled image IMG_DW2.

[0546] For example, the resolution of the up-sampled image IMG_UP1 and the down-sampled image IMG_DW2 is the same, and the pixel values of the corresponding pixels of the up-sampled image IMG_UP1 and the down-sampled image IMG_DW2 can be subtracted to obtain a difference image.

[0547] S710, determining a difference image of the up-sampled image IMG_UP1 and the m-th frame of image.

[0548] For example, S710 can be performed according to the description of S709, which will not be repeated here.

[0549] S711, inputting the i-th set of encoding parameters and the difference image to the i-th encoder.

[0550] For example, the resolution supported by the i-th encoder is the same as the resolution in the i-th set of encoding parameters.

[0551] S712, determining whether i is equal to (N-1).

[0552] For example, S712 can be performed according to the description of S305, which will not be repeated here.

[0553] For example, if i is equal to (N-1), S716 is performed. If i is not equal to (N-1), S706 is performed.

[0554] S713, adding 1 to i.

[0555] S714, determining the encoding parameters corresponding to the single-scale encoding.

[0556] S715, inputting the encoding parameters corresponding to the single-scale encoding and the m-th frame of image to the encoder.

[0557] S716, adding 1 to m.

[0558] For example, S713-S716 can refer to S306-S309 described above, and details are not repeated here.

[0559] It should be noted that the coding scale N of different images in the video stream of the same video service can be the same or different. Part of the images in the video stream of the same video service can be multi-scale coded, and another part of the images can be single-scale coded, and the embodiments of the present application do not limit this.

[0560] With reference to Figure 7c , the image preprocessing module is configured to perform the steps of processing the mth image in S701-S716 and inputting the processed mth image or the mth image to the encoder, and the multi-scale decision module is configured to perform the steps in S701-S716 except the steps performed by the image preprocessing module, and perform the step of inputting the ith set of coding parameters and the mth image to the image preprocessing module. For details, please refer to the description above, and details are not repeated here.

[0561] With reference to Figure 7c , after the data receiving module of the receiving device receives the encapsulated description data and the code stream data, the encapsulated description data and the code stream data can be input to the data parsing module, and the data parsing module can be used to unpack the encapsulated description data and the code stream data to obtain the code stream data and the description data.

[0562] For example, the data parsing module can input the ith set of code stream data into the ith decoder, and the ith decoder can reconstruct an image based on the ith set of code stream data to obtain a reconstructed image.

[0563] For example, after each decoder obtains a reconstructed image, the reconstructed image can be input to the post-processing module.

[0564] For example, the reconstructed image received by the post-processing module includes the reconstructed image of the down-sampled image IMG_DW1 and the reconstructed image of the difference image.

[0565] For example, when the post-processing module selects the reconstructed image corresponding to the Jth coding scale (i.e., the lowest coding scale) for display, the reconstructed image of the down-sampled image IMG_DW1 can be directly output to the display module.

[0566] For example, when the post-processing module selects the reconstructed image corresponding to the i-th encoding scale (any other encoding scale except the lowest encoding scale) to output to the display module, the reconstructed image of the down-sampled image IMG DW1 can be up-sampled according to the resolution of the i-th encoding scale to obtain an up-sampled image IMG UP2. Then the up-sampled image IMG UP2 can be superimposed with the reconstructed image of the i-th encoding scale to obtain the reconstructed image of the m-th frame of image at the i-th encoding scale. The reconstructed image of the m-th frame of image at the i-th encoding scale is then input to the display module.

[0567] For example, for other encoding scales except the lowest encoding scale, the image of the target region of the difference image can be input to the corresponding encoder for encoding, further reducing the computational load of the encoder. This can be referred to in the foregoing description and will not be described here.

[0568] For example, the reconstructed image received by the post-processing module includes the reconstructed image of the down-sampled image IMG DW1 and the reconstructed image of the image of the target region of the difference image.

[0569] For example, when the post-processing module selects the reconstructed image corresponding to the i-th encoding scale (any other encoding scale except the lowest encoding scale) to output to the display module, the reconstructed image of the down-sampled image IMG DW1 can be up-sampled according to the resolution of the i-th encoding scale to obtain an up-sampled image IMG UP2. Then the up-sampled image IMG UP2 can be superimposed with the reconstructed image of the i-th encoding scale to obtain the reconstructed image of the m-th frame of image at the i-th encoding scale. The reconstructed image of the m-th frame of image at the i-th encoding scale is then input to the display module.

[0570] Figure 7c The processing process is schematically shown for example. Referring to Figure 7c , the video stream includes e (e is a positive integer) frames of images, such as image 1, image 2, …, image e.

[0571] Referring to Figure 7c (1), for example, the multi-scale encoding module encodes image 1 at three encoding scales to obtain code stream data of the three encoding scales.

[0572] Referring to Figure 7c (1), for example, for image 1, the multi-scale encoding module can first down-sample image 1 according to the resolution in the encoding parameters corresponding to the first encoding scale to obtain image 1A. Then image 1A is encoded according to the encoding parameters corresponding to the first encoding scale to obtain code stream data of the first encoding scale. Referring to Figure 7c(2), exemplary, decode the code stream data corresponding to the first coding scale for image 1 to obtain image 1a, which is the reconstructed image corresponding to image 1A.

[0573] Referring to Figure 7c (1), exemplary, for image 1, the multi-scale coding module can first down-sample image 1 according to the resolution in the coding parameters corresponding to the second coding scale to obtain image 1B. Then up-sample image 1A according to the resolution in the coding parameters corresponding to the second coding scale to obtain image 1B'. Then determine the difference image 1B between image 1B and image 1B'. Next, encode the difference image 1B according to the coding parameters corresponding to the second coding scale to obtain the code stream data of the second coding scale. Referring to Figure 7c (2), exemplary, decode the code stream data corresponding to the second coding scale for image 1 to obtain difference image 1b, which is the reconstructed image corresponding to difference image 1B.

[0574] Referring to Figure 1 (1), exemplary, for image 1, the multi-scale coding module can first up-sample image 1A according to the resolution in the coding parameters corresponding to the third coding scale to obtain image 1C', and then determine the difference image 1C between image 1 and image 1C'. Next, encode the difference image 1C according to the coding parameters corresponding to the third coding scale to obtain the code stream data of the third coding scale. Referring to Figure 7c (2), exemplary, decode the code stream data corresponding to the third coding scale for image 1 to obtain difference image 1c, which is the reconstructed image corresponding to difference image 1C.

[0575] Referring to Figure 7c (2), exemplary, for image 1, when the reconstructed image output of the first coding scale is selected, image 1a can be directly output to the image display module for display. When the reconstructed image output of the second coding scale is selected, image 1a can be up-sampled according to the resolution in the coding parameters corresponding to the second coding scale to obtain image 1a'. Wherein, the resolution of image 1a' is the same as that of difference image 1b. Then superimpose difference image 1b and image 1a' to obtain image 1b' and output. When the reconstructed image output of the third coding scale is selected, image 1a can be up-sampled according to the resolution in the coding parameters corresponding to the third coding scale to obtain image 1a". Wherein, the resolution of image 1a" is the same as that of difference image 1c. Then superimpose difference image 1c and image 1a" to obtain image 1c' and output. Figure 7c (2), exemplary, decode the code stream data corresponding to the third coding scale for image 1 to obtain difference image 1c, which is the reconstructed image corresponding to difference image 1C.

[0576] Referring to Figure 7c(1), exemplary, the multi-scale coding module encodes the image 2 in two coding scales to obtain code stream data corresponding to the two coding scales.

[0577] Referring to Figure 7c (1), exemplary, for the image 2, the multi-scale coding module can first down-sample the image 2 according to the resolution in the coding parameters corresponding to the first coding scale to obtain the image 2A. Then the image 2A is encoded according to the coding parameters corresponding to the first coding scale to obtain the code stream data corresponding to the first coding scale. Referring to Figure 7c (2), exemplary, the code stream data corresponding to the first coding scale of the image 2 is decoded to obtain the image 2a, which is the reconstructed image corresponding to the image 2A.

[0578] Referring to Figure 8 (1), exemplary, for the image 2, the multi-scale coding module can first up-sample the image 2A according to the resolution in the coding parameters corresponding to the second coding scale to obtain the image 2B, and then determine the difference image 2B between the image 2 and the image 2B. Next, the difference image 2B is encoded according to the coding parameters corresponding to the second coding scale to obtain the code stream data corresponding to the third coding scale. Referring to ​ (2), exemplary, the code stream data corresponding to the third coding scale of the image 2 is decoded to obtain the difference image 2b, which is the reconstructed image corresponding to the difference image 2B.

[0579] Referring to ​ (2), exemplary, for the image 2, when the reconstructed image output of the first coding scale is selected, the image 2a can be directly output to the image display module for display. When the reconstructed image output of the second coding scale is selected, the image 2a can be up-sampled according to the resolution in the coding parameters corresponding to the second coding scale to obtain the image 2a'. Wherein, the resolution of the image 2a' is the same as that of the difference image 2b. Then the difference image 2b and the image 2a' are superimposed to obtain the image 2b' and output.

[0580] Exemplary, the multi-scale coding module encodes the image e in a single scale to obtain corresponding code stream data. Then the code stream data corresponding to the image e can be decoded to obtain the image e', which is the reconstructed image corresponding to the image e, and then the image e' can be directly output for display.

[0581] It should be noted that the above table 15 to table 17 show that when the coding parameters corresponding to each coding scale are preset, only one quantization parameter is set for one resolution; it should be understood that when the coding parameters corresponding to each coding scale are preset, multiple quantization parameters can be set for one resolution, which can be set according to the demand, and the embodiments of the present application do not limit this.

[0582] For example, for N=2 in Table 1, two kinds of coding parameters corresponding to coding scales can be preset, as shown in Table 24:

[0583] Table 24

[0584]

[0585]

[0586] In Table 24, the resolutions of different coding scales are the same, and the quantization parameters are different.

[0587] For example, for N=3 in Table 1, three kinds of coding parameters corresponding to coding scales can be preset, as shown in Table 25:

[0588] Table 25

[0589]

[0590] In Table 25, the resolutions of different coding scales are the same, and the quantization parameters are different.

[0591] For example, for N=4 in Table 1, four kinds of coding parameters corresponding to coding scales can be preset, as shown in Table 26:

[0592] Table 26

[0593]

[0594] In Table 26, the resolutions of different coding scales are the same, and the quantization parameters are different.

[0595] It should be noted that Tables 18-23 above show that, when determining the coding parameters corresponding to each coding scale according to at least two of the coding scale N, the image complexity of the mth frame of image, the current network transmission capability and the parameter value range of the encoder, only one quantization parameter is determined for one resolution. However, it should be understood that when determining the coding parameters corresponding to each coding scale according to at least two of the coding scale N, the image complexity of the mth frame of image, the current network transmission capability and the parameter value range of the encoder, multiple quantization parameters can also be determined for each resolution, which will not be illustrated one by one here.

[0596] It should be understood that, for different images belonging to the same video data, when it is determined that a certain frame image needs to be encoded in multiple scales, each set of determined encoding parameters can include a quantization parameter and reference frame information; when it is determined that another frame image needs to be encoded in multiple scales, each set of determined encoding parameters can include a resolution, a quantization parameter and reference frame information. That is, for different images belonging to the same frame of video data and needing to be encoded in multiple scales, the types of parameters included in the determined encoding parameters can be the same or different, and the embodiments of the present application do not limit this.

[0597] In one example, ​ A schematic block diagram of an apparatus 800 is shown, which can include a processor 801 and a transceiver / transceiver pin 802, and optionally a memory 803.

[0598] The various components of the apparatus 800 are coupled together by a bus 804, which can include a data bus, a power bus, a control bus, and a state signal bus. However, for the sake of clarity, the various buses are shown as a bus 804.

[0599] Optionally, the memory 803 can be used for instructions in the foregoing method embodiments. The processor 801 can be used to execute the instructions in the memory 803, and control the receiving pin to receive signals and the transmitting pin to transmit signals.

[0600] The apparatus 800 can be an electronic device or a chip of an electronic device in the above method embodiments.

[0601] All relevant content of each step involved in the above method embodiments can be cited from the function description of the corresponding function module, and will not be repeated here.

[0602] The embodiment also provides a computer storage medium, which stores computer instructions, and when the computer instructions run on an electronic device, the electronic device executes the above related method steps to implement the method in the above embodiment.

[0603] The embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the above related steps to implement the method in the above embodiment.

[0604] In addition, the embodiments of the present application also provide an apparatus, which can be a chip, a component or a module, and the apparatus can include a processor and a memory connected thereto; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the method in each of the above method embodiments.

[0605] The electronic device, the computer storage medium, the computer program product or the chip provided in the embodiment are used for executing the corresponding method provided above, and thus the beneficial effects achieved by the electronic device, the computer storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be described here again.

[0606] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0607] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0608] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0609] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0610] Any content of each embodiment of the present application, and any content of the same embodiment, can be freely combined. Any combination of the above is within the scope of the present application.

[0611] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes several instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0612] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims, and all of them belong to the protection of the present application.

[0613] The steps of the method or algorithm described in combination with the disclosure of the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disc (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0614] Those skilled in the art can understand that the functions described in the embodiments of the present application in the one or more examples above can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer program from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0615] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the specific embodiments described above, which are merely illustrative rather than restrictive, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. An encoding method characterized by comprising: The method comprises: acquiring an image to be encoded; determining that the image needs to be encoded in multiple scales according to a first preset condition that a coding importance of the image to other images in video data to which the image belongs meets, and / or a second preset condition that an impact of code stream data corresponding to the image on a network meets, and / or a third preset condition that a predicted image quality of the image meets; determining an encoding scale N for the image, wherein N is an integer greater than 1; determining N sets of encoding parameters corresponding to the N encoding scales, wherein each encoding scale corresponds to a set of encoding parameters; encoding the image N times by using a pre-set single-scale encoder according to the N sets of encoding parameters to obtain N sets of code stream data.

2. The method of claim 1, wherein, The first preset condition comprises at least one of the following: The image is an I frame or an IDR frame; The image is a scene switching frame; The image is a reference frame; The image is a key frame.

3. The method of claim 1, wherein, The second preset condition comprises at least one of the following: A data amount of the code stream data corresponding to the image is greater than a preset data amount threshold; The data amount of the code stream data corresponding to the image is greater than a network transmission capability.

4. The method of claim 1, wherein The predicted image quality of the image comprises a first predicted image quality and a second predicted image quality, the first predicted image quality is a predicted image quality of the image encoded at a source resolution of the image, and the second predicted image quality is a predicted image quality of the image encoded at a resolution lower than the source resolution; The third preset condition comprises: The first predicted image quality is lower than or equal to the second predicted image quality.

5. The method according to any one of claims 1 to 4, characterized in that, The encoding parameters comprise at least one of the following: a resolution, a quantization parameter, reference frame information, and a target code rate.

6. The method according to any one of claims 1 to 5, characterized in that, The determination of the encoding scale N for the image comprises: determining the encoding scale N according to an image type of the image and a preset correspondence between image types and encoding scales.

7. The method according to any one of claims 1 to 5, characterized in that, The determination of the encoding scale N for the image comprises: determining the encoding scale N for the image according to an encoding scale pre-set for each image in video data to which the image belongs.

8. The method according to any one of claims 1 to 7, characterized in that, The determination of the N sets of encoding parameters corresponding to the N encoding scales comprises: determining the N sets of encoding parameters according to the encoding scale N and a parameter value range of the encoder.

9. The method according to any one of claims 1 to 7, characterized in that, The encoding parameters comprise reference frame information, and when first feedback information is received, the reference frame information is determined according to the first feedback information, the first feedback information indicates that an encoded frame corresponding to transmitted code stream data is successfully received, the encoded frame is confirmed as the reference frame, and the reference frame information is used to indicate the reference frame.

10. The method according to any one of claims 1 to 9, characterized in that, The encoding parameters comprise a resolution, and in the N times of encoding of the image by using the pre-set single-scale encoder according to the N sets of encoding parameters, the i-th time of encoding comprises: when the resolution in the i-th set of encoding parameters is equal to the source resolution of the image, inputting the i-th set of encoding parameters and the image into the pre-set single-scale encoder for encoding; when the resolution in the i-th set of encoding parameters is less than the source resolution of the image, down-sampling the image to the resolution in the i-th set of encoding parameters, inputting the i-th set of encoding parameters and the image after the down-sampling into the pre-set single-scale encoder for encoding; wherein the encoding parameters corresponding to the i-th encoding are the i-th set of encoding parameters, and i is any integer between 1 and N.

11. A data encapsulation method, characterized by, The method comprises: obtaining code stream data obtained by encoding the image by an encoder, and description information of the code stream data; packaging the code stream data and the description information of the code stream data to obtain packaged data, and storing and / or transmitting the packaged data; wherein the code stream data comprises N sets of sub-code stream data, the N sets of sub-code stream data being obtained by encoding the image N times by a pre-set single-scale encoder according to N sets of encoding parameters; each set of encoding parameters corresponds to an encoding scale, wherein N is an integer greater than 1; the image satisfies a first preset condition in terms of the importance of the encoding and decoding of the image to other images in the video data, and / or the code stream data corresponding to the image satisfies a second preset condition in terms of the impact of the code stream data on the network, and / or the predicted image quality of the image satisfies a third preset condition.

12. The method of claim 11, wherein, The description information comprises scale identification information, and the method further comprises: when it is determined according to the scale identification information that the encoding scale corresponding to the code stream data is the lowest encoding scale, determining that the transmission strategy of the code stream data is a first transmission strategy; when it is determined according to the scale identification information that the encoding scale corresponding to the code stream data is other encoding scales, determining that the transmission strategy of the code stream data is a second transmission strategy; wherein the first transmission strategy is superior to the second transmission strategy.

13. The method according to claim 11 or 12, characterized in that, The description information comprises at least one of the following: scale identification information, frame type, and frame identification information.

14. A display method characterized by comprising: The method comprises: obtaining a plurality of reconstructed images obtained by decoding the code stream data by a decoder, and obtaining a plurality of sets of description information corresponding to the plurality of reconstructed images, at least one set of description information in the plurality of sets of description information comprising first frame identification information; if only one set of description information in the plurality of sets of description information comprises the first frame identification information, selecting the reconstructed image corresponding to the first frame identification information for display; if at least two sets of description information in the plurality of sets of description information comprise the first frame identification information, selecting one reconstructed image from the at least two reconstructed images corresponding to the first frame identification information, and displaying the selected reconstructed image; wherein the code stream data comprises at least two sets of sub-code stream data, and the at least two reconstructed images are obtained by decoding the at least two sets of sub-code stream data; the at least two sets of sub-code stream data are obtained by encoding the first image at least twice by a pre-set single-scale encoder according to at least two sets of encoding parameters. Each set of coding parameters corresponds to a coding scale, the first image pair satisfies a first preset condition in terms of coding importance of other images in the video data to which the image pair belongs, and / or, the first image pair satisfies a second preset condition in terms of impact of the stream data corresponding to the image pair on a network, and / or, the first image pair satisfies a third preset condition in terms of prediction quality of the image.

15. The method of claim 14, wherein, The description information further includes scale identification information, and the selecting one of the multiple reconstructed images corresponding to the first frame identification information includes: According to the scale identification information of the reconstructed image corresponding to the first frame identification information, selecting the reconstructed image with the highest coding scale.

16. The method of claim 14, wherein, The description information further includes scale identification information, and the selecting one of the multiple reconstructed images corresponding to the first frame identification information includes According to the scale identification information of the reconstructed image corresponding to the first frame identification information, selecting the reconstructed image with the scale identification information closest to the scale identification information of the previously displayed reconstructed image.

17. The method of any one of claims 14-16, wherein: The description information is obtained by unpackaging the received data, and the data obtained by unpackaging the received data further includes the stream data.

18. A multi-scale encoding apparatus characterized by comprising: The multi-scale coding device includes a pre-processing module and a single-scale encoder, The pre-processing module is configured to acquire an image to be coded, determine that the image needs to be coded in multiple scales according to a first preset condition in terms of coding importance of other images in the video data to which the image belongs, and / or, a second preset condition in terms of impact of the stream data corresponding to the image on a network, and / or, a third preset condition in terms of prediction quality of the image; determine a coding scale N for the image, wherein N is an integer greater than 1; determine N sets of coding parameters corresponding to the N coding scales, wherein each coding scale corresponds to a set of coding parameters; and input the N sets of coding parameters and the image to the single-scale encoder; The single-scale encoder is configured to code the image N times by using a pre-set single-scale encoder according to the N sets of coding parameters, to obtain N sets of stream data.

19. The apparatus of claim 18, wherein, The first preset condition includes at least one of the following: The image is an I frame or an IDR frame; The image is a scene switching frame; The image is a reference frame; The image is a key frame.

20. The apparatus of claim 18, wherein, The second preset condition includes at least one of the following: The data amount of the stream data corresponding to the image is greater than a preset data amount threshold; The data amount of the stream data corresponding to the image is greater than the network transmission capacity.

21. The apparatus of claim 18, wherein, The prediction quality of the image includes a first prediction quality and a second prediction quality, the first prediction quality is the quality of the image coded at a source resolution of the image, and the second prediction quality is the quality of the image coded at a resolution lower than the source resolution; The third preset condition includes that the first prediction quality is lower than or equal to the second prediction quality.

22. The apparatus of any one of claims 18-21, wherein: The pre-processing module is configured to determine the encoding scale N according to the image type of the image and a pre-set correspondence between image types and encoding scales.

23. The apparatus of any one of claims 18 to 22, wherein, The pre-processing module is configured to determine the N sets of encoding parameters according to the encoding scale N and a parameter value range of the encoder.

24. The apparatus of any one of claims 18 to 23, wherein, The encoding parameters include reference frame information, and the reference frame information is determined according to first feedback information when the first feedback information is received, the first feedback information indicating that an encoding frame corresponding to transmitted code stream data is successfully received, the encoding frame being confirmed as the reference frame, and the reference frame information being used to indicate the reference frame.

25. A multi-scale decoding apparatus, comprising: The multi-scale decoding apparatus includes a decoder and a post-processing module, The decoder is configured to input a plurality of reconstructed images obtained by decoding code stream data to the post-processing module. The post-processing module is configured to obtain the plurality of reconstructed images obtained by decoding the code stream data by the decoder, and obtain a plurality of sets of description information corresponding to the plurality of reconstructed images, at least one set of description information in the plurality of sets of description information including first frame identification information; if only one set of description information in the plurality of sets of description information includes the first frame identification information, a reconstructed image corresponding to the first frame identification information is selected to be input to a display module for display; if at least two sets of description information in the plurality of sets of description information include the first frame identification information, one reconstructed image is selected from the at least two reconstructed images corresponding to the first frame identification information, and the selected reconstructed image is input to the display module for display. The code stream data includes at least two sets of sub-code stream data, and the at least two reconstructed images are obtained by decoding the at least two sets of sub-code stream data; the at least two sets of sub-code stream data are obtained by encoding a first image at least twice using a pre-set single-scale encoder according to at least two sets of encoding parameters; each set of encoding parameters corresponds to an encoding scale, the first image satisfies a first preset condition in terms of the importance of the first image to the encoding and decoding of other images in video data, and / or the impact of the code stream data corresponding to the image on a network satisfies a second preset condition, and / or the prediction quality of the image satisfies a third preset condition.

26. An electronic device, comprising: The memory and the processor are coupled; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device performs the method in any one of claims 1 to 17. The interface circuit is configured to receive a signal from a memory of an electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method in any one of claims 1 to 17.

27. A chip, characterized by ​ 28. A computer storage medium, comprising, The computer readable storage medium stores a computer program which, when executed on a computer or processor, causes the computer or processor to perform the method of any one of claims 1 to 17.

29. A computer program product, characterised in that, The computer program product comprises a software program which, when executed by a computer or processor, causes the steps of the method of any one of claims 1 to 17 to be performed.

Images