Multimedia data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a multimedia data processing method and device, equipment and a storage medium. The method comprises the following steps: a sending end device obtains target multimedia data; inputting the target multimedia data into a preset coding model for coding processing to obtain an output code stream; wherein the output code stream comprises a basic coding code stream and N sub-coding code streams, and N is an integer greater than or equal to 1; according to the output code stream, obtaining processed multimedia data, and sending the processed multimedia data to a receiving end device; wherein the processed multimedia data comprises a basic coding code stream and P sub-coding code streams, and P is an integer greater than or equal to 1 and less than or equal to N. According to the invention, the storage resource occupancy rate of the sending end equipment can be reduced.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to a multimedia data processing method, apparatus, device and storage medium. Background Technology

[0002] With the development of science and technology, users can now watch pre-recorded audio and video programs on electronic devices. The playback of audio and video programs relies on the encoding and decoding process of multimedia data.

[0003] Currently, for multimedia data, multiple bitrate encoding data are typically pre-made. For example, for the same video data, standard definition, high definition, and ultra-high definition encoding data are pre-made, and then the bitrate encoding data to be sent is determined according to the user's needs. However, this method requires pre-storing encoding data of multiple bitrates, which occupies a large amount of storage space on the sending device. Summary of the Invention

[0004] This application provides a multimedia data processing method, apparatus, device, and storage medium to reduce the storage resource occupancy rate of the transmitting device.

[0005] In a first aspect, embodiments of this application provide a multimedia data processing method, the method being applied to a transmitting device, comprising:

[0006] Acquire target multimedia data;

[0007] The target multimedia data is input into a preset encoding model for encoding processing to obtain an output bitstream; wherein, the output bitstream includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1;

[0008] Based on the output bitstream, processed multimedia data is obtained and sent to the receiving device; wherein, the processed multimedia data includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N.

[0009] In one possible implementation, the encoding model includes a single main encoding structure and N residual encoding structures; the target multimedia data is input into the preset encoding model for encoding processing to obtain an output bitstream, including:

[0010] Based on the main coding structure, the target multimedia data is encoded to obtain the basic encoded bitstream;

[0011] Based on each of the residual coding structures, the target multimedia data is encoded to obtain each of the sub-coded bitstreams.

[0012] In one possible implementation, the main coding structure includes a first encoder network and a first quantization network; based on the main coding structure, the target multimedia data is encoded to obtain the basic coded bitstream, including:

[0013] The first encoder network based on the main coding structure performs dimensionality reduction and coding processing on the target multimedia data to obtain basic coded data, and outputs the basic coded data to the first quantization network of the main coding structure.

[0014] Based on the first quantization network of the main coding structure, the codewords and their indices corresponding to the basic coding data are determined in a preset code table; and based on the first quantization network of the main coding structure, the basic coding bitstream is determined according to the index of the codewords corresponding to the basic coding data; wherein, the code table includes codewords with multiple preset indices.

[0015] In one possible implementation, the main coding structure further includes a first mapping network; based on the first encoder network of the main coding structure, the target multimedia data undergoes dimensionality reduction and encoding processing to obtain basic encoded data, including:

[0016] The first encoder network based on the main coding structure performs dimensionality reduction and encoding processing on the target multimedia data to obtain initial intermediate encoded data, and outputs the initial intermediate encoded data to the first mapping network of the main coding structure.

[0017] The first mapping network based on the main coding structure performs dimensionality reduction processing on the initial intermediate coding data to obtain basic coding data, and outputs the basic coding data to the first quantization network of the main coding structure.

[0018] In one possible implementation, after determining the codewords and their indices corresponding to the basic encoded data in a preset code table within a first quantization network based on the master coding structure, the method includes:

[0019] The first quantization network based on the master coding structure determines the first residual data according to the basic coding data and the codewords corresponding to the basic coding data, and outputs the first residual data to the first layer residual coding structure.

[0020] In one possible implementation, the i-th layer residual coding structure in the N-layer residual coding structure includes an (i+1)-th encoder network and an (i+1)-th quantization network; where i is an integer greater than or equal to 1 and less than or equal to N; based on each residual coding structure, the target multimedia data is encoded to obtain each sub-coded bitstream, including:

[0021] Based on the (i+1)th encoder network of the i-th layer residual coding structure, the i-th residual data is subjected to dimensionality reduction and coding processing to obtain the i-th coded data, and the i-th coded data is output to the (i+1)th quantization network of the i-th layer residual coding structure; wherein, when i is 1, the first residual data is obtained by the main coding structure processing the target multimedia data; when i is not 1, the i-th residual data is obtained by the (i-1)th layer residual coding structure processing the (i-1)th residual data;

[0022] Based on the (i+1)th quantization network of the i-th layer residual coding structure, the codeword and its index corresponding to the i-th encoded data are determined in a preset code table, and the i-th sub-coded bitstream is determined based on the (i+1)th quantization network of the i-th layer residual coding structure and the index of the codeword corresponding to the i-th encoded data.

[0023] In one possible implementation, the i-th layer residual coding structure further includes an (i+1)-th mapping network; based on the (i+1)-th encoder network of the i-th layer residual coding structure, the i-th residual data undergoes dimensionality reduction and coding processing to obtain the i-th coded data, including:

[0024] Based on the (i+1)th encoder network of the i-th layer residual coding structure, the i-th residual data is subjected to dimensionality reduction and coding processing to obtain the i-th intermediate coding data, and the i-th intermediate coding data is output to the (i+1)th mapping network of the i-th layer residual coding structure.

[0025] Based on the (i+1)th mapping network of the i-th layer residual coding structure, the i-th intermediate coding data is dimensionality reduced to obtain the i-th coding data, and the i-th coding data is output to the (i+1)th quantization network of the i-th layer residual coding structure.

[0026] In one possible implementation, after determining the codeword and its index corresponding to the i-th encoded data in a preset code table within the (i+1)-th quantization network based on the i-th layer residual coding structure, the method includes:

[0027] Based on the (i+1)th quantization network of the i-th layer residual coding structure, the (i+1)th residual data is determined according to the i-th encoded data and the codeword corresponding to the i-th encoded data, and the (i+1)th residual data is output to the (i+2)th encoder network of the (i+1)th layer residual coding structure.

[0028] In one possible implementation, the step of inputting the target multimedia data into a preset encoding model to obtain an output bitstream includes:

[0029] Determine the network bandwidth between the sending and receiving devices;

[0030] The number of levels N of the coding model is determined based on the network bandwidth; wherein the number of levels is used to indicate the number of sub-coded bitstreams included in the output bitstream;

[0031] Based on the number of levels N, the encoding model corresponding to the number of levels N is determined from a preset model library; wherein, the model library includes encoding models with different numbers of levels N;

[0032] The target multimedia data is input into the encoding model corresponding to the number of layers N to obtain the output bitstream.

[0033] In one possible implementation, obtaining the processed multimedia data based on the output bitstream includes:

[0034] Determine the network bandwidth between the sending and receiving devices;

[0035] Based on the network bandwidth, determine the number P of sub-coded bitstreams; and based on the number P of sub-coded bitstreams, determine the output bitstream to be sent from the output bitstreams;

[0036] Based on the output bitstream to be sent, the processed multimedia data is obtained.

[0037] Secondly, embodiments of this application provide a multimedia data processing method, the method being applied to a receiving device, the method comprising:

[0038] The system receives processed multimedia data sent by a transmitting device; wherein the processed multimedia data is obtained by the transmitting device based on an output bitstream, and the output bitstream is obtained by the transmitting device based on a preset encoding model; the output bitstream includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1; the processed multimedia data includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N;

[0039] Based on the processed multimedia data, encoding information is determined; wherein the encoding information is used to indicate the bitstream information included in the processed multimedia data;

[0040] Based on the encoding information, the processed multimedia data is input into a preset decoding model for decoding to obtain target multimedia data, and then the target multimedia data is played.

[0041] In one possible implementation, the step of inputting the processed multimedia data into a preset decoding model for decoding processing according to the encoding information to obtain multimedia data includes:

[0042] Based on the encoding information, determine the number P of sub-coded bitstreams included in the processed multimedia data;

[0043] Based on the number P of the sub-coded bitstreams, a decoding model corresponding to the number P of the sub-coded bitstreams is determined from a preset model library; wherein, the model library includes multiple decoding models, and each decoding model can decode and process a different number of sub-coded bitstreams;

[0044] The processed multimedia data is input into the decoding model corresponding to the number P of the sub-encoded bitstreams for decoding processing to obtain the target multimedia data.

[0045] In one possible implementation, the decoding model includes a single main decoding structure and a P-layer residual decoding structure; the processed multimedia data is input into the decoding model corresponding to the number P of the sub-coded bitstreams for decoding processing to obtain the target multimedia data, including:

[0046] Based on the encoding information, determine the main decoding structure in the decoding model corresponding to the basic encoded bitstream, and the residual decoding structure in the decoding model corresponding to each sub-encoded bitstream;

[0047] The sub-coded bitstream is input into the residual decoding structure in the decoding model corresponding to the sub-coded bitstream for decoding processing to obtain the first sub-decoded bitstream;

[0048] The basic encoded bitstream and the first sub-decoded bitstream are input into the main decoding structure in the decoding model corresponding to the basic encoded bitstream for decoding processing to obtain the target multimedia data.

[0049] In one possible implementation, the j-th layer residual decoding structure in the P-layer residual decoding structure includes a second inverse quantization network, a second feature concatenation network, and a second decoder network; wherein j is 1, and P is greater than 1;

[0050] The sub-coded bitstream is input into the residual decoding structure in the decoding model corresponding to the sub-coded bitstream for processing to obtain the first sub-decoded bitstream, including:

[0051] The second inverse quantization network based on the first layer residual decoding structure determines the codeword corresponding to the index in the first sub-encoded bitstream in the preset code table, obtains the first decoded data, and outputs the first decoded data to the second feature splicing network of the first layer residual decoding structure.

[0052] The second feature concatenation network based on the first layer residual decoding structure performs feature concatenation on the first decoded data output by the second inverse quantization network of the first layer residual decoding structure and the second sub-decoded bitstream output by the third decoder network of the second layer residual decoding structure to obtain the first intermediate decoded data, and outputs the first intermediate decoded data to the second decoder network of the first layer residual decoding structure.

[0053] The second decoder network, based on the first-layer residual decoding structure, performs dimensionality upscaling and decoding on the first intermediate decoded data to obtain the first sub-decoded bitstream.

[0054] In one possible implementation, the j-th layer residual decoding structure in the P-layer residual decoding structure includes a (j+1)-th dequantization network, a (j+1)-th feature concatenation network, and a (j+1)-th decoder network; wherein j is an integer greater than 1 and less than P; the method further includes:

[0055] Based on the j+1 inverse quantization network of the j-th layer residual decoding structure, the codeword corresponding to the index in the j-th sub-encoded bitstream is determined in the preset code table to obtain the j-th decoded data, and the j-th decoded data is output to the j+1 feature splicing network of the j-th layer residual decoding structure.

[0056] Based on the j+1 feature concatenation network of the j-th residual decoding structure, the j-th decoded data output by the j+1 inverse quantization network of the j-th residual decoding structure and the j+1 sub-decoded bitstream output by the j+2 decoder network of the j+1 residual decoding structure are concatenated to obtain the j-th intermediate decoded data, and the j-th intermediate decoded data is output to the j+1 decoder network of the j-th residual decoding structure.

[0057] Based on the j+1 decoder network of the j-th layer residual decoding structure, the j-th intermediate decoded data is subjected to dimensionality upscaling and decoding to obtain the j-th sub-decoded bitstream, and the j-th sub-decoded bitstream is output to the j-th feature splicing network of the j-1 layer residual decoding structure.

[0058] In one possible implementation, the j-th layer residual decoding structure in the P-layer residual decoding structure includes a P+1-th inverse quantization network and a P+1-th decoder network; where j is P; the method further includes:

[0059] The P+1 dequantization network based on the P-th layer residual decoding structure determines the codeword corresponding to the index in the P-th sub-encoded bitstream in the preset code table, obtains the P-th decoded data, and outputs the P-th decoded data to the P+1 decoder network of the P-th layer residual decoding structure.

[0060] Based on the P+1 decoder network of the P-th layer residual decoding structure, the P-th decoded data is subjected to dimensionality upscaling and decoding to obtain the P-th sub-decoded bitstream, and the P-th sub-decoded bitstream is output to the P-th feature splicing network of the P-1 layer residual decoding structure.

[0061] In one possible implementation, the main decoding structure includes a first inverse quantization network, a first feature concatenation layer, and a first decoder network; the basic encoded bitstream and the first sub-decoded bitstream are input into the main decoding structure of the decoding model corresponding to the basic encoded bitstream for processing to obtain the target multimedia data, including:

[0062] The first inverse quantization network based on the main decoding structure determines the codeword corresponding to the index in the basic encoded bitstream in the preset code table, obtains the basic decoding data, and outputs the basic decoding data to the first feature concatenation network of the main decoding structure.

[0063] Based on the first feature concatenation network of the main decoding structure, the basic decoded data output by the first inverse quantization network of the received main decoding structure and the first sub-decoded bitstream output by the second decoder network of the received first layer residual decoding structure are concatenated to obtain the final intermediate decoded data, and the final intermediate decoded data is output to the first decoder network of the main decoding structure.

[0064] The first decoder network based on the main decoding structure performs dimensionality upscaling and decoding on the final intermediate decoded data to obtain the target multimedia data.

[0065] Thirdly, embodiments of this application provide a method for training an encoding / decoding model, wherein the encoding / decoding model includes a single initial master coding structure and N initial residual coding structures, as well as a single initial master decoding structure and N initial residual decoding structures; including:

[0066] Acquire training data; wherein the training data includes sample multimedia data;

[0067] Based on the training data, the initial master coding structure and the initial master decoding structure are trained to obtain the master coding structure and the master decoding structure;

[0068] For the initial residual coding structure and the initial residual decoding structure of the Lth layer, based on the main coding structure, the main decoding structure, and the training data, the initial residual coding structure and the initial residual decoding structure of the Lth layer are trained to obtain the residual coding structure and the residual decoding structure of the Lth layer; where L is an integer greater than or equal to 1 and less than or equal to N;

[0069] Based on the main coding structure and the N-layer residual coding structure, a preset coding model is obtained; and based on the main decoding structure and the N-layer residual decoding structure, a preset decoding model is obtained.

[0070] The preset encoding model is used to encode the target multimedia data as described in any one of the first aspects; the preset decoding model is used to decode the processed multimedia data as described in any one of the second aspects.

[0071] In one possible implementation, the initial master coding structure and the initial master decoding structure are trained based on the training data to obtain the master coding structure and the master decoding structure, including:

[0072] Based on the initial master encoding structure, the training data is subjected to dimensionality reduction and encoding processing to obtain basic encoded training data; and based on the initial master decoding structure, the basic encoded training data is subjected to dimensionality increase and decoding processing to obtain basic decoded training data.

[0073] Based on the training data, the basic encoding training data, and the basic decoding training data, the model parameters of the initial master encoding structure and the initial master decoding structure are adjusted to obtain the master encoding structure and the master decoding structure; wherein, the model parameters include weight values ​​and / or bias values.

[0074] In one possible implementation, the initial residual coding structure and the initial residual decoding structure of layer L are trained based on the main coding structure, the main decoding structure, and the training data to obtain the initial residual coding structure and the initial residual decoding structure of layer L, including:

[0075] For the initial residual coding structure and the initial residual decoding structure of the Lth layer, the training data is subjected to dimensionality reduction and encoding based on the main coding structure and the (L-1)th layer residual coding structure to obtain the Lth residual training data; wherein, the (L-1)th layer residual coding structure is trained based on the (L-1)th layer initial residual coding structure.

[0076] Based on the main decoding structure, the training data, and the Lth residual training data, the model parameters of the Lth layer initial residual coding structure and the Lth layer initial residual decoding structure are adjusted to obtain the Lth layer residual coding structure and the Lth layer residual decoding structure.

[0077] Fourthly, embodiments of this application provide a multimedia data processing apparatus, which is applied to a transmitting end device and includes:

[0078] The acquisition module is used to acquire target multimedia data;

[0079] The encoding module is used to input the target multimedia data into a preset encoding model for encoding processing to obtain an output bitstream; wherein, the output bitstream includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1;

[0080] The output module is used to obtain processed multimedia data based on the output bitstream and send the processed multimedia data to the receiving device; wherein the processed multimedia data includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N.

[0081] Fifthly, embodiments of this application provide a multimedia data processing apparatus, which is applied to a receiving end device and includes:

[0082] A receiving module is used to receive processed multimedia data sent by a sending device; wherein the processed multimedia data is obtained by the sending device based on an output bitstream, and the output bitstream is obtained by the sending device based on a preset encoding model; the output bitstream includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1; the processed multimedia data includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N;

[0083] A determining module is used to determine encoding information based on the processed multimedia data; wherein the encoding information is used to indicate the bitstream information included in the processed multimedia data;

[0084] The decoding module is used to input the processed multimedia data into a preset decoding model for decoding processing according to the encoding information, to obtain target multimedia data, and to play the target multimedia data.

[0085] Sixthly, embodiments of this application provide a codec model training apparatus, the apparatus comprising:

[0086] An acquisition module is used to acquire training data; wherein, the training data includes sample multimedia data;

[0087] The first training module is used to train the initial master coding structure and the initial master decoding structure based on the training data to obtain the master coding structure and the master decoding structure.

[0088] The second training module is used to train the initial residual coding structure and the initial residual decoding structure of the Lth layer based on the main coding structure, the main decoding structure, and the training data to obtain the initial residual coding structure and the initial residual decoding structure of the Lth layer; wherein, L is an integer greater than or equal to 1 and less than or equal to N;

[0089] The combination module is used to obtain a preset encoding model based on the main encoding structure and the N-layer residual encoding structure; and to obtain a preset decoding model based on the main decoding structure and the N-layer residual decoding structure.

[0090] The preset encoding model is used to encode the target multimedia data as described in any one of the first aspects; the preset decoding model is used to decode the processed multimedia data as described in any one of the second aspects.

[0091] In a seventh aspect, embodiments of this application provide an electronic device, including: a memory and a processor;

[0092] The memory stores computer-executed instructions;

[0093] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any one of the first, second, or third aspects above.

[0094] Eighthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any one of the first, second, or third aspects above.

[0095] Ninthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any one of the first, second, or third aspects above.

[0096] The multimedia data processing method, apparatus, device, and storage medium provided in this application embodiment enable a transmitting device to encode target multimedia data using a preset encoding model, obtaining an output bitstream comprising a basic encoded bitstream and N sub-encoded bitstreams. The processed multimedia data can then be determined based on the output bitstream and sent to a receiving device. In this manner, the transmitting device can perform any combination based on the output bitstream to obtain the desired processed multimedia data. Furthermore, the transmitting device only needs to store an output bitstream comprising a maximum of one basic encoded bitstream and N sub-encoded bitstreams. Compared to the prior art, which requires storing bitstreams of multiple bitrates, this reduces the storage resource occupancy of the transmitting device and saves storage resources. Attached Figure Description

[0097] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0098] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0099] Figure 2 A flowchart illustrating a multimedia data processing method provided in an embodiment of this application;

[0100] Figure 3 A flowchart illustrating another multimedia data processing method provided in an embodiment of this application;

[0101] Figure 4 A flowchart illustrating another multimedia data processing method provided in an embodiment of this application;

[0102] Figure 5 This is a schematic diagram of the structure of an encoding model provided in an embodiment of this application;

[0103] Figure 6 This is a schematic diagram of a main coding structure provided in an embodiment of this application;

[0104] Figure 7 This is a schematic diagram of another main coding structure provided in an embodiment of this application;

[0105] Figure 8 This is a schematic diagram of another encoding model provided in an embodiment of this application;

[0106] Figure 9 This is a schematic diagram of another main coding structure provided in an embodiment of this application;

[0107] Figure 10 A flowchart illustrating a multimedia data processing method provided in an embodiment of this application;

[0108] Figure 11 A flowchart illustrating a multimedia data processing method provided in an embodiment of this application;

[0109] Figure 12 This is a schematic diagram of the structure of a decoding model provided in an embodiment of this application;

[0110] Figure 13 This is a schematic diagram of a residual decoding structure provided in an embodiment of this application;

[0111] Figure 14 This is a schematic diagram of a main decoding structure provided in an embodiment of this application;

[0112] Figure 15 A flowchart illustrating a method for training an encoding / decoding model provided in an embodiment of this application;

[0113] Figure 16 This is a schematic diagram of the structure of a multimedia data processing device provided in an embodiment of this application;

[0114] Figure 17 This is a schematic diagram of another multimedia data processing device provided in an embodiment of this application;

[0115] Figure 18 This is a schematic diagram of the structure of a codec model training device provided in an embodiment of this application;

[0116] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0117] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0118] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0119] First, the terms used in this application will be explained.

[0120] Bitrate: refers to the number of bits of data transmitted per unit time during data transmission.

[0121] When users watch pre-recorded audio and video programs on electronic devices, they can typically choose from multiple bitrates, such as standard definition, high definition, or ultra-high definition video. To meet the needs of all users, the transmitting device needs to pre-encode the audio and video at various bitrates and pre-store the encoded data for each bitrate, thus providing users with audio and video data at multiple bitrates.

[0122] However, the same audio and video data corresponds to multiple encoded data with different bitrates. That is, the same audio data needs to be stored in multiple encoded data, resulting in high storage resource utilization of the sending device.

[0123] In view of this, embodiments of this application provide a multimedia data processing method. A transmitting device performs layered encoding processing on multimedia data to be sent to a receiving device, obtaining multiple bitstreams after layered encoding. These multiple bitstreams can be combined to form output bitstreams with various bitrates. In this way, even if the multimedia data to be sent to the receiving device undergoes the highest level of layered encoding processing, the transmitting device only needs to store one output bitstream with the highest bitrate, eliminating the need to store multiple bitstreams with different bitrates, thus reducing the storage space occupied by the transmitting device.

[0124] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, the sending device encodes the multimedia data to be sent to the receiving device, sends the encoded multimedia data to the receiving device, and the receiving device decodes the data for playback.

[0125] The aforementioned transmitting and receiving devices can be electronic devices with processing capabilities, such as servers, computers, mobile phones, tablets, or smart wearable electronic devices, etc., and the embodiments of this application are not limited to these.

[0126] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0127] Figure 2 This is a flowchart illustrating a multimedia data processing method provided in an embodiment of this application, as shown below. Figure 2 As shown, the method includes:

[0128] S101, The sending device acquires the target multimedia data.

[0129] For example, the target multimedia data mentioned above is multimedia data to be sent to the receiving device. This multimedia data may be, for example, audio, video, or images. It should be noted that this application embodiment does not limit the type of multimedia data.

[0130] In one possible implementation, the sending device may communicate with other electronic devices to receive the target multimedia data sent by those devices; or the sending device may receive target multimedia data imported from an external device, such as a USB flash drive; or the sending device may retrieve data from a preset database to obtain the target multimedia data.

[0131] S102. The transmitting device inputs the target multimedia data into a preset encoding model for encoding processing to obtain the output bitstream.

[0132] The aforementioned output bitstream refers to the multimedia data obtained after encoding the target multimedia data. The output bitstream includes one basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1. Specifically, the basic encoded bitstream refers to the data obtained after performing basic encoding processing on the target multimedia data, and the sub-encoded bitstreams refer to the data obtained after performing deep encoding processing on the target multimedia data.

[0133] The combination of the aforementioned basic encoded bitstream and different numbers of sub-encoded bitstreams can yield output bitstreams with different bitrates. For example, taking multimedia data as video data, when the output bitstream includes the basic encoded bitstream + sub-encoded bitstream 1, it can correspond to standard definition video (e.g., bitrate of 3Mb / s); when the output bitstream includes the basic encoded bitstream + sub-encoded bitstream 1 + sub-encoded bitstream 2, it can correspond to high definition video (e.g., bitrate of 3.5Mb / s).

[0134] The aforementioned preset encoding model is used to perform layered encoding processing on the target multimedia data to facilitate subsequent transmission.

[0135] In one example, the above encoding model may include N+1 encoding structures, which are interconnected to output a basic encoded bitstream and various sub-encoded bitstreams. These encoding structures may, for example, include Convolutional Neural Networks (CNNs). It should be noted that different encoding structures have different CNN parameters.

[0136] In this implementation, the transmitting device inputs the target multimedia data into a preset encoding model, summarizes the outputs of each encoding structure, and obtains the output bitstream. For example, the basic encoded bitstream is obtained using encoding structure 1, and the sub-encoded bitstream 1 is obtained using encoding structure 2. The outputs of the two encoding structures are summarized to obtain an output bitstream that includes the basic encoded bitstream and the sub-encoded bitstream 1.

[0137] Regarding how to determine the value of N in the output bitstream, one possible approach is to use a preset value for N; another possible approach is to determine the value of N based on the real-time network bandwidth between the sending and receiving devices. For example, a mapping relationship between network bandwidth and the value of N can be preset, and then the value of N can be determined based on the network bandwidth.

[0138] S103. The transmitting device obtains the processed multimedia data based on the output bitstream and sends the processed multimedia data to the receiving device.

[0139] It should be understood that the aforementioned receiving device is equipped with a preset decoding model for decoding processing. The processed multimedia data is then used for decoding based on the preset decoding model to obtain the target multimedia data. The processed multimedia data includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N. The processed multimedia data has encoding information, which may include, for example, the position, identifier, and quantity of the bitstreams. It should be noted that the embodiments of this application do not limit the content of the encoding information; it can be set according to actual needs.

[0140] In one possible implementation, the transmitting device directly sends the above-mentioned output bitstream as processed multimedia data, i.e., P=N, to the receiving device.

[0141] In another possible implementation, the transmitting device filters the processed multimedia data from the output bitstream and sends it to the receiving device. For example, the output bitstream includes a basic encoded bitstream and five sub-encoded bitstreams, and the transmitting device uses the basic encoded bitstream and three sub-encoded bitstreams as the processed multimedia data.

[0142] In this embodiment, the transmitting device encodes the target multimedia data using a preset encoding model to obtain an output bitstream comprising a basic encoded bitstream and N sub-encoded bitstreams. The processed multimedia data can then be determined based on the output bitstream and sent to the receiving device. In this way, the transmitting device can perform any combination based on the output bitstream to obtain the desired processed multimedia data. Furthermore, the transmitting device only needs to store an output bitstream comprising a maximum of one basic encoded bitstream and N sub-encoded bitstreams. Compared to the prior art, which requires storing multiple bitstreams at different bitrates, this reduces the storage resource occupancy of the transmitting device and saves storage resources.

[0143] Figure 3 A flowchart illustrating another multimedia data processing method provided in this application embodiment is shown below. Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the implementation of the multimedia data processing method in the application scenario where the sending device transmits multimedia data to the receiving device in real time is described in detail. The method includes:

[0144] S201, The sending device acquires the target multimedia data.

[0145] The aforementioned target multimedia data refers to multimedia data sent to the receiving device in real time, such as multimedia data during a live broadcast.

[0146] It should be noted that this step is similar to the aforementioned step S101. For details, please refer to the aforementioned step S101. It will not be repeated here.

[0147] S202. The sending device determines the network bandwidth between the sending device and the receiving device.

[0148] The aforementioned network bandwidth refers to the network's data transmission capacity or rate, usually measured in the amount of data transmitted per second.

[0149] In one possible implementation, the sending device can use a speed testing tool to measure the network bandwidth between the sending and receiving devices. For example, the sending device can use the speed testing tool to send test data to the receiving device to measure the transmission speed of the test data and determine the network bandwidth between them. It should be noted that the embodiments of this application do not limit the type of speed testing tool.

[0150] The sending device can also use network performance monitoring tools to monitor network bandwidth usage in real time, thereby determining the current network bandwidth between the sending and receiving devices.

[0151] S203. The sending device determines the number of levels N of the encoding model based on the network bandwidth.

[0152] The number of levels mentioned above is used to indicate the number of sub-coded bitstreams included in the output bitstream.

[0153] It should be understood that processed multimedia data needs to be transmitted from the sending device to the receiving device via the network, and the bitrate of the processed multimedia data is affected by network bandwidth. For example, when the network bandwidth is insufficient to support the bitrate of the processed multimedia data, problems such as data congestion and packet loss will occur, resulting in stuttering, blurry images, or pauses during playback. Therefore, the bitrate of the processed multimedia data needs to be determined based on the network bandwidth. The bitrate of the processed multimedia data is determined by the number of levels N in the encoding model; therefore, the number of levels N in the encoding model can be determined based on the network bandwidth.

[0154] In one example, the sending device can preset the mapping relationship between network bandwidth and the number of coding model levels N, and determine the number of coding model levels N based on the network bandwidth and the mapping relationship.

[0155] S204. The transmitting device determines the encoding model corresponding to the number of levels N from the preset model library.

[0156] The aforementioned model library includes encoding models with different levels N.

[0157] For example, after determining the number of levels N of the encoding model, the transmitting device can determine the encoding model corresponding to the number of levels N from the model library.

[0158] S205. The transmitting device inputs the target multimedia data into the encoding model corresponding to the number of layers N to obtain the output bitstream.

[0159] For example, as mentioned above, the encoding model can include an N+1 layer encoding structure, where one layer is used to output a basic encoded bitstream and N layers are used to output N sub-encoded bitstreams. Therefore, after determining the encoding model corresponding to the number of layers N, inputting the target multimedia data into the encoding model corresponding to the number of layers N can yield an output bitstream including a basic encoded bitstream and N sub-encoded bitstreams.

[0160] S206. The transmitting device obtains the processed multimedia data based on the output bitstream and sends the processed multimedia data to the receiving device.

[0161] For example, since the aforementioned steps have determined that the output stream can be used for transmission based on the network bandwidth, the output stream can be directly sent to the receiving device as processed multimedia data.

[0162] The multimedia data processing method provided in this application allows the sending device to determine the number of encoding model levels based on the network bandwidth between the sending and receiving devices. When network bandwidth is high, a higher number of encoding levels can be used to process the target multimedia data, resulting in higher bitrate processed multimedia data being sent to the receiving device, thus improving the clarity and image quality of the multimedia data. Conversely, when network bandwidth is low, to avoid data congestion during data transmission, the sending device can use a lower number of encoding levels to process the target multimedia data, resulting in lower bitrate processed multimedia data being sent to the receiving device, ensuring smooth playback of the multimedia data and minimizing stuttering and image quality loss.

[0163] This method utilizes network bandwidth to determine the number of layers in the encoding model, thereby directly determining the number of sub-encoded bitstreams included in the output bitstream, which in turn determines the bitrate of the processed multimedia data. This not only reduces the occupancy of storage resources at the sending end, but also allows for real-time adjustment of the bitrate of multimedia data sent to the receiving end device based on network bandwidth, ensuring high-quality playback of multimedia data.

[0164] Figure 4 A flowchart illustrating another multimedia data processing method provided in this application embodiment is shown below. Figure 4 As shown, in this embodiment... Figure 2 Based on the embodiments, the application of the multimedia data processing method to pre-recorded multimedia data is described in detail. The method includes:

[0165] S301, The sending device acquires the target multimedia data.

[0166] The aforementioned target multimedia data refers to pre-recorded multimedia data, such as pre-produced video programs.

[0167] It should be noted that this step is similar to the aforementioned step S101. For details, please refer to the aforementioned step S101. It will not be repeated here.

[0168] S302. The transmitting device inputs the target multimedia data into a preset encoding model for encoding processing to obtain the output bitstream.

[0169] For example, the output bitstream described above includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1. The value of N is a preset value and is independent of network bandwidth.

[0170] In one example, the above coding model includes a single main coding structure and N residual coding structures. The main coding structure performs basic coding processing on the target multimedia data to obtain a basic coded bitstream. The N residual coding structures perform deeper coding processing on the target multimedia data to obtain N sub-coded bitstreams. The basic coded bitstream and the N sub-coded bitstreams are then combined to obtain the output bitstream.

[0171] S303. The sending device determines the network bandwidth between the sending device and the receiving device.

[0172] It should be noted that this step is similar to the aforementioned step S202, and the details can be found in the aforementioned step S202, which will not be repeated here.

[0173] S304. The transmitting device determines the number of sub-coded bitstreams based on the network bandwidth; and determines the output bitstream to be sent from the output bitstream based on the number of sub-coded bitstreams.

[0174] For example, the output stream to be sent mentioned above refers to the stream after filtering the output stream.

[0175] In one example, as mentioned above, network bandwidth is related to the bitrate of the processed multimedia data, and the number of sub-coded bitstreams is related to the bitrate of the processed multimedia data. Therefore, the number of sub-coded bitstreams can be determined based on the network bandwidth. For example, the transmitting device can preset a mapping relationship between network bandwidth and the number of sub-coded bitstreams, and determine the number of sub-coded bitstreams based on the network bandwidth and this mapping relationship.

[0176] After determining the number of sub-coded bitstreams, the output bitstream to be sent can be selected from the above output bitstreams based on this number. For example, if the number of sub-coded bitstreams is 3, the output bitstream to be sent is the basic coded bitstream + sub-coded bitstream 1 + sub-coded bitstream 2 + sub-coded bitstream 3.

[0177] S305. The transmitting device obtains the processed multimedia data based on the output bitstream to be transmitted.

[0178] For example, the sending device can use the output bitstream to be sent as processed multimedia data.

[0179] The multimedia data processing method provided in this application embodiment allows the sending device to pre-encode target multimedia data by a preset value N, obtaining an output bitstream including a basic encoded bitstream and N sub-encoded bitstreams. When the processed multimedia data corresponding to the target multimedia data needs to be sent to the receiving device, the bitrate of the output bitstream to be sent is determined based on the current network bandwidth, i.e., the number of sub-encoded bitstreams included in the output bitstream to be sent, thereby obtaining the processed multimedia data. In this way, the sending device occupies at most the storage space of the output bitstream including the preset value N sub-encoded bitstreams and the basic encoded bitstream, reducing the occupancy rate of storage resources.

[0180] In summary, whether in applications involving pre-recorded multimedia data or applications requiring real-time transmission of multimedia data, the transmitting device can save storage resources and reduce storage resource occupancy by generating an output stream including a basic encoded bitstream and N sub-encoded bitstreams based on a preset encoding model. The structure of the encoding model provided in the embodiments of this application will be described in detail below.

[0181] Figure 5 This is a schematic diagram of the structure of an encoding model provided in an embodiment of this application, such as... Figure 5 As shown, the coding model includes a single main coding structure and N residual coding structures.

[0182] The main coding structure and each residual coding structure described above can be a CNN, but the parameters of CNNs with different coding structures are different.

[0183] The transmitting device can encode the target multimedia data based on the above main coding structure to obtain the basic coded bitstream; and encode the target multimedia data based on each residual coding structure to obtain each sub-coded bitstream.

[0184] (I) Main Encoding Structure

[0185] In some embodiments, reference Figure 6 , Figure 6 This is a schematic diagram of a master coding structure provided in an embodiment of this application. The master coding structure may include a first encoder network and a first quantization network.

[0186] The first encoder network mentioned above, such as a CNN, is used to perform dimensionality reduction and encoding on the target multimedia data.

[0187] The aforementioned first quantization network, such as a CNN, is used to quantize multimedia data after dimensionality reduction and encoding to reduce the amount of data.

[0188] Under this architecture, the aforementioned transmitting device encodes the target multimedia data based on the aforementioned main coding structure to obtain a basic encoded bitstream, which may include the following steps:

[0189] S4011. The transmitting device performs dimensionality reduction and encoding processing on the target multimedia data based on the first encoder network of the main coding structure to obtain basic encoded data, and outputs the basic encoded data to the first quantization network of the main coding structure.

[0190] The aforementioned basic encoded data refers to the convolutional vector obtained after dimensionality reduction and encoding of the target multimedia data. The methods of dimensionality reduction and encoding are not limited in this embodiment.

[0191] S4012. The transmitting device can determine the codewords and their indices corresponding to the basic encoded data in a preset code table based on the first quantization network of the main encoding structure; and determine the basic encoded bitstream based on the index of the codewords corresponding to the basic encoded data according to the first quantization network of the main encoding structure; wherein the code table includes codewords with multiple preset indices.

[0192] The aforementioned code table is a pre-defined index codeword table within the first quantization network. The index codeword table contains at least one level of indexes and the codewords corresponding to each index. An index is a quantization value used to describe a codeword; a codeword is a multi-dimensional vector used for comparison with the basic encoded data. The aforementioned basic encoded bitstream can be viewed as a convolutional vector represented by an index based on the code table.

[0193] For example, the transmitting device can calculate the similarity between the basic encoded data and each codeword in the code table through the first quantization network of the main coding structure, and determine the codeword and its index corresponding to the basic encoded data. Then, the first quantization network of the main coding structure can combine the indices of the codewords corresponding to the basic encoded data to obtain the basic encoded bitstream.

[0194] It should be noted that the settings of the first encoder network parameters mentioned above need to take into account the dimension of the preset code table in the first quantization network, that is, the dimension of the basic encoded data output by the first encoder network is the same as the dimension of the code table.

[0195] In one possible implementation, the first quantization network can also be used to determine the residual data of the basic encoded data and the codewords corresponding to the basic encoded data. Specifically, the transmitting device can determine the first residual data based on the first quantization network of the main coding structure, according to the basic encoded data and the codewords corresponding to the basic encoded data, and output the first residual data to the first layer residual coding structure.

[0196] For example, the first residual data is used to characterize the difference between the basic encoded data and the codeword corresponding to the basic encoded data determined based on the code table. For example, the difference between each element in the basic encoded data and the codeword corresponding to that element can be calculated and summed to obtain the first residual data.

[0197] In this embodiment, the main coding structure includes a first encoder network and a first quantization network. The transmitting device can perform dimensionality reduction and coding processing on the target multimedia data based on the first encoder network of the main coding structure to obtain basic coded data. The first quantization network of the main coding structure performs quantization processing on the basic coded data to obtain a basic coded bitstream, further reducing the amount of multimedia data and improving the subsequent transmission speed.

[0198] In some embodiments, reference Figure 7 , Figure 7 This is a schematic diagram of another main coding structure provided in an embodiment of this application. Figure 6 Based on the embodiments, the main encoding structure described above may further include a first mapping network. This first mapping network may be a CNN, used to ensure that the dimension of the basic encoded data input to the first quantization network is consistent with the dimension of the preset code table in the first quantization network.

[0199] Under this architecture, the aforementioned transmitting device performs dimensionality reduction and encoding processing on the target multimedia data based on the first encoder network of the main coding structure to obtain basic encoded data, which may include the following steps:

[0200] S4021. The transmitting device performs dimensionality reduction and encoding processing on the target multimedia data based on the first encoder network of the main encoding structure to obtain initial intermediate encoded data, and outputs the initial intermediate encoded data to the first mapping network of the main encoding structure.

[0201] For example, the aforementioned initial intermediate encoded data refers to the convolutional vector after dimensionality reduction and encoding of the target multimedia data.

[0202] S4022. The transmitting device performs dimensionality reduction processing on the initial intermediate encoded data based on the first mapping network of the main coding structure to obtain basic encoded data, and outputs the basic encoded data to the first quantization network of the main coding structure.

[0203] In this embodiment, the main coding structure includes a first encoder network, a first mapping network, and a first quantization network. The transmitting device can perform dimensionality reduction and encoding processing on the target multimedia data based on the first encoder network of the main coding structure to obtain initial intermediate coded data. The first mapping network of the main coding structure can perform dimensionality reduction processing on the initial intermediate coded data to ensure that the dimension of the output basic coded data is the same as the dimension of the preset code table of the first quantization network. The first quantization network of the main coding structure performs quantization processing on the basic coded data. In this way, the architecture of the first encoder network does not need to consider the dimension of the code table, reducing the complexity of the first encoder network.

[0204] (II) Residual Coding Structure

[0205] In some embodiments, the i-th layer residual coding structure in the N-layer residual coding structure includes the (i+1)-th encoder network and the (i+1)-th quantization network.

[0206] The i+1th encoder network mentioned above, for example, can be a CNN, used for dimensionality reduction and encoding of the residual data.

[0207] The i+1th quantization network mentioned above, such as a CNN, is used to quantize the residual data after dimensionality reduction and encoding to reduce the amount of data.

[0208] It should be noted that the model parameters of the encoder network and quantization network in each residual coding structure are different.

[0209] Under this architecture, the following section provides a detailed explanation of how the transmitting device encodes the target multimedia data based on each layer of residual coding structure to obtain each sub-coded bitstream.

[0210] refer to Figure 8 , Figure 8 This is a schematic diagram of another encoding model provided in an embodiment of this application. It should be noted that... Figure 8 The main coding structure in the example is illustrated using the first mapping network.

[0211] For the i-th layer residual coding structure, when i is 1, i.e., for the first layer residual coding structure, as mentioned above, the first quantization network of the main coding structure can also be used to determine the first residual data of the basic coded data and the codeword corresponding to the basic coded data. The basic coded data is obtained after dimensionality reduction and encoding of the target multimedia data. This first residual data can be output to the second encoder network of the first layer residual coding structure.

[0212] The transmitting device can perform dimensionality reduction and encoding processing on the first residual data based on the second encoder network of the first layer residual coding structure to obtain the first encoded data, and output the first encoded data to the second quantization network of the first layer residual coding structure;

[0213] The transmitting device determines the codeword and its index corresponding to the first encoded data in a preset code table using a second quantization network based on the first-layer residual coding structure, and determines the first sub-coded bitstream based on the index of the codeword corresponding to the first encoded data using the second quantization network based on the first-layer residual coding structure.

[0214] For the i-th layer residual coding structure, when i is an integer greater than 1 and less than or equal to N, the transmitting device can perform dimensionality reduction and coding processing on the i-th residual data based on the (i+1)-th encoder network of the i-th layer residual coding structure to obtain the i-th coded data, and output the i-th coded data to the (i+1)-th quantization network of the i-th layer residual coding structure; wherein, the i-th residual data is obtained by processing the (i-1)-th residual data by the (i-1)-th layer residual coding structure;

[0215] The transmitting device can determine the codeword and its index corresponding to the i-th encoded data in a preset code table based on the i+1 quantization network of the i-th residual coding structure, and determine the i-th sub-coded bitstream based on the index of the codeword corresponding to the i-th encoded data in the i+1 quantization network of the i-th residual coding structure.

[0216] It should be noted that the principles of how the (i+1)th quantization network determines the codeword and its index corresponding to the i-th encoded data, and how it determines the i-th sub-encoded bitstream, are similar to those of the first quantization network, and will not be elaborated here.

[0217] It should be noted that the settings of the above-mentioned i+1 encoder network parameters need to take into account the dimension of the preset code table in the i+1 quantization network. That is, the dimension of the i-th encoded data output by the i+1 encoder network is the same as the dimension of the code table.

[0218] In one possible implementation, the (i+1)th quantization network can also be used to determine the residual data of the i-th encoded data and the codeword corresponding to the i-th encoded data. Specifically, the transmitting device can determine the (i+1)th residual data based on the (i+1)th quantization network of the (i+1)th residual coding structure, according to the i-th encoded data and the codeword corresponding to the i-th encoded data, and output the (i+1)th residual data to the (i+2)th encoder network of the (i+1)th residual coding structure.

[0219] In this embodiment, the i-th layer residual coding structure in the N-layer residual coding structure includes an (i+1)-th encoder network and an (i+1)-th quantization network. The transmitting device can perform dimensionality reduction and coding processing on the residual data from the previous layer coding structure based on the (i+1)-th encoder network of the i-th layer residual coding structure to obtain the coded data of this layer. Then, it can perform quantization processing on the coded data of this layer based on the (i+1)-th quantization network of the i-th layer residual coding structure to obtain the sub-coded bitstream output by this layer residual coding structure. Through the residual coding structure, deep coding processing of the target multimedia data can be performed, obtaining more detailed features and improving the fidelity of the processed multimedia data.

[0220] In some embodiments, reference Figure 9 , Figure 9 This is a schematic diagram of another main coding structure provided in an embodiment of this application. Figure 8 Based on the embodiments, the i-th layer residual coding structure in the N-layer residual coding structure may further include an (i+1)-th mapping network. This (i+1)-th mapping network may be a CNN, used to ensure that the dimension of the i-th encoded data input to the (i+1)-th quantization network is consistent with the dimension of the preset code table in the (i+1)-th quantization network.

[0221] Under this architecture, the transmitting device can perform dimensionality reduction and encoding processing on the i-th residual data based on the (i+1)-th encoder network of the i-th layer residual coding structure to obtain the i-th encoded data, which may include the following steps:

[0222] S4031. An encoder network based on the (i+1)th layer residual coding structure performs dimensionality reduction and coding on the i-th residual data to obtain the i-th intermediate coding data, and outputs the i-th intermediate coding data to the second mapping network of the i-th layer residual coding structure.

[0223] For example, the i-th intermediate encoded data mentioned above refers to the convolution vector after dimensionality reduction and encoding of the i-th residual data.

[0224] S4032. Based on the i+1th mapping network of the i-th layer residual coding structure, the i-th intermediate coding data is dimensionality reduced to obtain the i-th coding data, and the i-th coding data is output to the i+1th quantization network of the i-th layer residual coding structure.

[0225] It should be noted that the residual coding structure of the i-th layer may or may not include the (i+1)-th mapping network. This application embodiment does not limit this, and the specific configuration can be made according to actual needs.

[0226] In this embodiment, the i-th layer residual coding structure includes an (i+1)-th encoder network, an (i+1)-th mapping network, and an (i+1)-th quantization network. The transmitting device can perform dimensionality reduction and encoding processing on the i-th residual data based on the (i+1)-th encoder network of the residual coding structure to obtain the i-th intermediate encoded data. The (i+1)-th mapping network of the residual coding structure can perform dimensionality reduction processing on the i-th intermediate encoded data to ensure that the dimension of the output i-th encoded data is the same as the dimension of the preset code table of the (i+1)-th quantization network. The (i+1)-th quantization network of the residual coding structure performs quantization processing on the i-th encoded data. In this way, the architecture of the (i+1)-th encoder network does not need to consider the dimension of the code table, reducing the complexity of the (i+1)-th encoder network.

[0227] In summary, through the various methods described in the foregoing embodiments, the target multimedia data has been encoded to obtain processed multimedia data including a basic encoded bitstream and multiple sub-encoded bitstreams, which is then sent to the receiving device for playback. The following describes how the receiving device decodes the processed multimedia data.

[0228] Figure 10 This is a flowchart illustrating a multimedia data processing method provided in an embodiment of this application, as shown below. Figure 10 As shown, the method includes:

[0229] S501, The receiving device receives the processed multimedia data sent by the sending device.

[0230] The processed multimedia data described above is obtained by the sending device based on the output bitstream. This output bitstream is obtained by the sending device encoding the target multimedia data based on a preset encoding model. The output bitstream includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1. The processed multimedia data described above also includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N.

[0231] It should be noted that the embodiments of this application do not limit the communication connection method between the receiving device and the sending device.

[0232] S502. The receiving device determines the encoding information based on the processed multimedia data.

[0233] The aforementioned encoding information is used to indicate the bitstream information included in the processed multimedia data. For example, this encoding information may include the number of bitstreams, bitstream identifiers, or encoding model identifiers, which can be set according to actual needs.

[0234] For example, the preset field in the processed multimedia data is encoded information. The receiving device can extract the information from the preset field to obtain the encoded information.

[0235] S503. The receiving device inputs the processed multimedia data into a preset decoding model for decoding processing according to the encoding information to obtain the target multimedia data and play the target multimedia data.

[0236] The aforementioned target multimedia data refers to the data obtained after decoding the processed multimedia data.

[0237] For example, as mentioned above, the processed multimedia data includes a basic encoded bitstream and p sub-encoded bitstreams, which are obtained by layering the target multimedia data using the above encoding model. Therefore, the receiving device needs to perform layered decoding processing on the processed multimedia data using a preset decoding model to obtain the target multimedia data, and then play the target multimedia data.

[0238] The aforementioned preset decoding model is used to perform layered decoding on the processed multimedia data in order to facilitate the playback of the target multimedia data.

[0239] In one possible implementation, the above encoding model includes N+1 encoding structures, which are connected to each other to output the basic encoded bitstream and each sub-encoded bitstream. Correspondingly, the decoding model includes N+1 decoding structures, which are connected to each other to perform decoding processing on the basic encoded bitstream and each sub-encoded bitstream.

[0240] In one example, the encoding model and the decoding model are set accordingly. The above encoding information can indicate the identifier of the encoding model used by the processed multimedia data. Therefore, the processed multimedia data can be input into the decoding model corresponding to the encoding model for decoding processing according to the encoding information to obtain the target multimedia data.

[0241] In another example, the aforementioned encoding information can indicate the number of sub-coded bitstreams. Since the processed multimedia data includes a basic encoded bitstream and multiple sub-coded bitstreams, the decoding processing methods differ depending on the number of sub-coded bitstreams. Therefore, it is necessary to determine the number of sub-coded bitstreams included in the processed multimedia data based on the encoding information. Then, based on the number of sub-coded bitstreams, the corresponding decoding model for the processed multimedia data can be determined. Therefore, based on this encoding information, the processed multimedia data can be input into the corresponding decoding model for decoding processing to obtain the target multimedia data.

[0242] In this embodiment, the receiving device can determine the encoding information based on the processed multimedia data sent by the sending device, and then decode the processed multimedia data based on a preset decoding model to obtain the target multimedia data, which can then be played. By combining the encoding model of the sending device with the encoding processing of the target multimedia data in this way, the storage resource usage of the target multimedia data in the sending device can be reduced. Simultaneously, the sending device further reduces the data volume of the target multimedia data through layered encoding, increases the transmission rate of the target multimedia data, shortens the latency of the receiving device waiting for data transmission, and thus reduces the time the user waits for the multimedia data to play.

[0243] Figure 11 This is a flowchart illustrating a multimedia data processing method provided in an embodiment of this application, as shown below. Figure 11 As shown, the method includes:

[0244] S601, The receiving device receives the processed multimedia data sent by the sending device.

[0245] It should be noted that this step is similar to step S501, and will not be repeated here.

[0246] S602. The receiving device determines the encoding information based on the processed multimedia data.

[0247] It should be noted that this step is similar to step S502, and will not be repeated here.

[0248] S603. The receiving device determines the number P of sub-coded bitstreams included in the processed multimedia data based on the encoding information.

[0249] For example, the preset field in the above encoding information can indicate the number of sub-coded bitstreams. Therefore, the receiving device can extract information from the preset field to obtain the number P of sub-coded bitstreams included in the processed multimedia data.

[0250] S604. The receiving device determines the decoding model corresponding to the number P of sub-coded bitstreams from a preset model library.

[0251] The aforementioned model library includes multiple decoding models, each capable of decoding a different number of sub-coded bitstreams.

[0252] For example, the processed multimedia data includes P sub-coded bitstreams, indicating that decoding of the P sub-coded bitstreams is required. Therefore, it is necessary to determine a decoding model capable of decoding the P sub-coded bitstreams. After the receiving device determines the number P sub-coded bitstreams, it can determine the decoding model corresponding to that number P from the model library.

[0253] S605. The receiving device inputs the processed multimedia data into the decoding model corresponding to the number P of sub-coded bitstreams for decoding processing to obtain the target multimedia data.

[0254] For example, the above decoding model may include a P+1 layer decoding structure, where one layer is used to decode the basic coded bitstream, and P layers are used to decode P sub-coded bitstreams. Therefore, after determining the decoding model corresponding to the number P sub-coded bitstreams, the processed multimedia data can be input into the decoding model corresponding to the number P sub-coded bitstreams to obtain the target multimedia data.

[0255] S606, The receiving device plays the target multimedia data.

[0256] In this embodiment, the receiving device can determine the encoding information based on the processed multimedia data received from the sending device. Then, based on the encoding information, it can determine the number of sub-coded bitstreams included in the processed multimedia data. Based on the number of sub-coded bitstreams, it can determine the decoding model, and then use the decoding model to decode the processed multimedia data to obtain the target multimedia data, which is then played. This method accurately determines the decoding model capable of decoding the processed multimedia data based on the encoding information, improving the accuracy and reproducibility of the decoded target multimedia data. Furthermore, combining the encoding model of the sending device with the encoding processing of the target multimedia data can reduce the storage resource usage of the target multimedia data in the sending device.

[0257] The structure of the decoding model provided in the embodiments of this application will be described in detail below.

[0258] Figure 12 This is a schematic diagram of the structure of a decoding model provided in an embodiment of this application, such as... Figure 12 As shown, the decoding model includes a main decoding structure and a P-layer residual decoding structure.

[0259] The main decoding structure and each residual decoding structure described above can be a CNN, but the parameters of CNNs with different decoding structures are different.

[0260] The receiving device can decode the basic encoded bitstream in the processed multimedia data based on the above main decoding structure; and decode each sub-encoded bitstream in the processed multimedia data based on each residual decoding structure, and then summarize them to obtain the target multimedia data.

[0261] In this architecture, the receiving device inputs the processed multimedia data into the decoding model corresponding to the number P of sub-coded bitstreams for decoding processing to obtain the target multimedia data, which may include the following steps:

[0262] S7011. The receiving device determines the main decoding structure in the decoding model corresponding to the basic encoded bitstream, and the residual decoding structure in the decoding model corresponding to each sub-encoded bitstream, based on the encoding information.

[0263] For example, as mentioned above, the above encoding information can indicate the identifier of each bitstream. Therefore, the basic encoded bitstream and each sub-encoded bitstream can be determined based on the identifier of each bitstream. In turn, the main decoding structure in the decoding model corresponding to the basic encoded bitstream and the residual decoding structure in the decoding model corresponding to each sub-encoded bitstream can be determined.

[0264] S7012. The receiving device inputs the sub-coded bitstream into the residual decoding structure in the decoding model corresponding to the sub-coded bitstream for decoding processing to obtain the first sub-decoded bitstream.

[0265] For example, the first sub-decoded bitstream mentioned above refers to the bitstream after decoding P sub-coded bitstreams based on the P-layer residual decoding structure.

[0266] S7013, The receiving device inputs the basic encoded bitstream and the first sub-decoded bitstream into the main decoding structure in the decoding model corresponding to the basic encoded bitstream for decoding processing to obtain the target multimedia data.

[0267] For example, the receiving device can perform decoding processing on the basic encoded bitstream and the first sub-decoded bitstream based on the main decoding structure to obtain the target multimedia data.

[0268] It should be noted that the embodiments of this application do not limit how each decoding structure performs decoding processing.

[0269] In this embodiment, the receiving device can decode the processed multimedia data based on each layer of the residual decoding structure and the main decoding structure of the decoding model to obtain the target multimedia data.

[0270] (III) Residual Decoding Structure

[0271] In some embodiments, reference Figure 13 , Figure 13This is a schematic diagram of a residual decoding structure provided in an embodiment of this application. The j-th layer residual decoding structure in the P-layer residual decoding structure includes a (j+1)-th dequantization network, a (j+1)-th feature concatenation network, and a (j+1)-th decoder network. Here, j is an integer greater than or equal to 1 and less than or equal to P.

[0272] The aforementioned (j+1)th dequantization network, such as a CNN, is used to determine the codeword corresponding to the index in the sub-coded bitstream based on a preset code table.

[0273] The aforementioned (j+1)th feature concatenation network is used to add features to the decoded data output by the (j+1)th dequantization network and the decoded bitstream from the next layer decoding structure.

[0274] The aforementioned (j+1)th decoder network, for example, could be a CNN, used for both dimensionality upscaling and decoding.

[0275] It should be noted that the model parameters of the decoder network, dequantization network, and feature splicing network in each residual decoding structure are different.

[0276] In this architecture, when j=1, refer to Figure 13 The receiving device inputs the sub-coded bitstream into the residual decoding structure in the decoding model corresponding to the sub-coded bitstream for processing to obtain the first sub-decoded bitstream, which may include the following steps:

[0277] S7021. The receiving device, based on the second inverse quantization network of the first layer residual decoding structure, determines the codeword corresponding to the index in the first sub-encoded code stream in the preset code table, obtains the first decoded data, and outputs the first decoded data to the second feature splicing network of the first layer residual decoding structure.

[0278] For example, as mentioned above, the preset code table includes codewords and their corresponding indices. Therefore, the receiving device can determine the codewords corresponding to the indices in the first sub-coded bitstream based on the code table in the second inverse quantization network, and obtain the first decoded data. This first decoded data can be regarded as a multi-dimensional convolutional vector.

[0279] S7022. The receiving device, based on the second feature splicing network of the first layer residual decoding structure, performs feature splicing on the first decoded data output by the second inverse quantization network of the received first layer residual decoding structure and the second sub-decoded bitstream output by the third decoder network of the received second layer residual decoding structure to obtain the first intermediate decoded data, and outputs the first intermediate decoded data to the second decoder network of the first layer residual decoding structure.

[0280] For example, the aforementioned first intermediate decoded data refers to the data obtained by feature concatenating the first decoded data and the second sub-decoded bitstream. The aforementioned second sub-decoded bitstream is obtained by decoding the second sub-coded bitstream based on the second-layer residual decoding structure.

[0281] S7023, The receiving device uses a second decoder network based on the first layer residual decoding structure to perform dimensionality upscaling and decoding on the first intermediate decoded data to obtain the first sub-decoded bitstream.

[0282] For example, the receiving device can perform dimensionality upscaling and decoding processing on the first intermediate decoded data based on the first decoder network to obtain the first sub-decoded bitstream. It should be noted that this application embodiment does not limit how the dimensionality upscaling processing is performed.

[0283] When j is an integer greater than 1 and less than P, refer to Figure 13 As shown, the receiving device inputs the sub-coded bitstream into the residual decoding structure in the decoding model corresponding to the sub-coded bitstream for processing to obtain the first sub-decoded bitstream, which may include the following steps:

[0284] S7031. The receiving device, based on the (j+1)th inverse quantization network of the j-th layer residual decoding structure, determines the codeword corresponding to the index in the j-th sub-coded bitstream in the preset code table, obtains the j-th decoded data, and outputs the j-th decoded data to the (j+1)th feature splicing network of the j-th layer residual decoding structure.

[0285] S7032. The receiving device, based on the j+1 feature splicing network of the j-th layer residual decoding structure, performs feature splicing on the j-th decoded data output by the j+1 inverse quantization network of the received j-th layer residual decoding structure and the j+1 sub-decoded bitstream output by the j+2 decoder network of the received j+1 layer residual decoding structure to obtain the j-th intermediate decoded data, and outputs the j-th intermediate decoded data to the j+1 decoder network of the j-th layer residual decoding structure.

[0286] S7033: The receiving device, based on the j+1 decoder network of the j-th layer residual decoding structure, performs dimensionality upscaling and decoding processing on the j-th intermediate decoding data to obtain the j-th sub-decoded bitstream, and outputs the j-th sub-decoded bitstream to the j-th feature splicing network of the j-1 layer residual decoding structure.

[0287] When j is P, refer to Figure 13 As shown, the P-th layer residual decoding structure includes a P+1-th inverse quantization network and a P+1-th decoder network. Under this architecture, the receiving device inputs the sub-coded bitstream into the residual decoding structure in the decoding model corresponding to the sub-coded bitstream for processing to obtain the first sub-decoded bitstream. This can include the following steps:

[0288] S7041. The receiving device, based on the P+1 inverse quantization network of the P-th layer residual decoding structure, determines the codeword corresponding to the index in the P-th sub-encoded code stream in the preset code table, obtains the P-th decoded data, and outputs the P-th decoded data to the P+1 decoder network of the P-th layer residual decoding structure.

[0289] S7042. The receiving device performs dimensionality upscaling and decoding on the P-th decoded data based on the P+1 decoder network of the P-th residual decoding structure to obtain the P-th sub-decoded bitstream, and outputs the P-th sub-decoded bitstream to the P-th feature splicing network of the P-1 residual decoding structure.

[0290] In this embodiment, the receiving device can perform decoding processing on each sub-coded bitstream step by step from the P-th layer residual decoding structure to the first layer residual decoding structure, and summarize it into the feature splicing network of the upper layer residual decoding structure to realize the layered decoding of each sub-coded bitstream, and finally summarize it to obtain the first sub-decoded bitstream.

[0291] (iv) Main Decoding Structure

[0292] In some embodiments, reference Figure 14 , Figure 14 This is a schematic diagram of a main decoding structure provided in an embodiment of this application. The main decoding structure includes a first inverse quantization network, a first feature splicing layer, and a first decoder network.

[0293] The aforementioned first inverse quantization network, such as a CNN, is used to determine the codeword corresponding to the index in the basic encoded bitstream based on a preset code table.

[0294] The aforementioned first feature concatenation network is used to perform feature addition on the decoded data output by the first inverse quantization network and the first sub-decoded bitstream from the first layer residual decoding structure.

[0295] The first decoder network mentioned above, such as a CNN, is used for both dimensionality upscaling and decoding.

[0296] Under this architecture, refer to Figure 14 As shown, the receiving device inputs the basic encoded bitstream and the first sub-decoded bitstream into the main decoding structure in the decoding model corresponding to the basic encoded bitstream for processing to obtain the target multimedia data. This may include the following steps:

[0297] S7051. The receiving device, based on the first inverse quantization network of the main decoding structure, determines the codeword corresponding to the index in the basic encoded bitstream in the preset code table, obtains the basic decoding data, and outputs the basic decoding data to the first feature splicing network of the main decoding structure.

[0298] For example, as mentioned above, the preset code table includes codewords and their corresponding indices. Therefore, the receiving device can determine the codewords corresponding to the indices in the basic encoded bitstream based on the code table in the first inverse quantization network, and obtain the basic decoded data. This basic decoded data can be regarded as a multi-dimensional convolutional vector.

[0299] S7052. The receiving device performs feature splicing on the basic decoded data output by the first inverse quantization network of the received main decoding structure and the first sub-decoded bitstream output by the second decoder network of the received first layer residual decoding structure, based on the first feature splicing network of the main decoding structure, to obtain the final intermediate decoded data, and outputs the final intermediate decoded data to the first decoder network of the main decoding structure.

[0300] For example, the final decoded data mentioned above refers to the vector obtained by feature concatenation of the basic decoded data and the first sub-decoded bitstream.

[0301] S7053, the receiving device, based on the first decoder network of the main decoding structure, performs upscaling and decoding processing on the final intermediate decoded data to obtain the target multimedia data.

[0302] As mentioned above, the final intermediate decoded data is a vector that aggregates the features of the basic encoded bitstream and the features of each sub-encoded bitstream. Therefore, the target multimedia data can be obtained by decoding the final intermediate decoded data.

[0303] In this embodiment, the receiving device can perform inverse quantization on the basic encoded bitstream using the main decoding structure, then concatenate it with the first sub-decoded bitstream to obtain the final intermediate decoded data. This data is then subjected to dimensionality upscaling and decoding to obtain the target multimedia data. This method achieves layered decoding of the processed multimedia data to obtain the target multimedia data.

[0304] The decoding model described above can decode the received processed multimedia data to obtain the target multimedia data. The training method for this decoding model is explained below.

[0305] The training methods for the encoding and decoding models of the above architecture will be explained below. Figure 15 This is a flowchart illustrating a method for training an encoding / decoding model provided in an embodiment of this application. Figure 15 As shown, the method includes:

[0306] S801, Obtain training data.

[0307] The training data mentioned above includes sample multimedia data.

[0308] The training data mentioned above can be obtained by the sending device communicating with other electronic devices; it can also be training data imported by the sending device from an external device, such as a USB flash drive; or the sending device can obtain the training data by calling data from a preset database.

[0309] S802. Based on the training data, train the initial main coding structure and the initial main decoding structure to obtain the main coding structure and the main decoding structure.

[0310] The above encoding / decoding model includes one initial master coding structure and N initial residual coding structures, as well as one initial master decoding structure and N initial residual decoding structures. It should be noted that the architecture of each coding or decoding structure included in the encoding / decoding model can be referred to the description in the foregoing embodiments, and will not be repeated here. It should be understood that the training of the coding or decoding structure mentioned below refers to the training of the model parameters of each network in the coding or decoding structure; the training of a single network will not be described here.

[0311] In some embodiments, the training data can be dimensionality reduced and encoded based on the initial master encoding structure to obtain basic encoded training data; and the basic encoded training data can be dimensionality increased and decoded based on the initial master decoding structure to obtain basic decoded training data; and the model parameters of the initial master encoding structure and the initial master decoding structure can be adjusted based on the training data, the basic encoded training data, and the basic decoded training data to obtain the master encoding structure and the master decoding structure.

[0312] For example, the model parameters mentioned above include weight values ​​and / or bias values. The training data can be input into the initial master encoding structure, and dimensionality reduction and encoding processes can be performed using the preset initial model parameters of the initial master encoding structure to obtain basic encoded training data. Then, the basic encoded training data can be input into the initial decoding structure, and dimensionality increase and decoding processes can be performed using the preset initial model parameters of the initial master decoding structure to obtain basic decoded training data. Based on the training data, the basic encoded training data, and the basic decoded training data, the model parameters of the initial master encoding structure and the initial master decoding structure are adjusted to obtain the master decoding structure.

[0313] Specifically, the training data can first be dimensionality-reduced and encoded using the first initial encoder network in the initial master encoding structure to obtain initial intermediate encoded training data. Then, based on the preset code table in the first initial quantization network, the initial intermediate encoded training data is quantized to obtain basic encoded training data. The basic encoded training data is then input into the initial decoding structure, and dimensionality-increasing and decoding are performed using the preset initial parameters of the initial master decoding structure to obtain basic decoded training data. Based on the training data, basic encoded training data, and basic decoded training data, the model parameters of the initial master encoding structure and the initial master decoding structure are adjusted to obtain the master decoding structure.

[0314] During model parameter tuning, gradient descent can be used to adjust the parameters based on a preset objective function. This preset objective function characterizes the mapping relationship between the model parameters and the reconstruction loss value. The reconstruction loss value includes a first loss value and a second loss value. The reconstruction loss value can be the sum of the first and second loss values, a weighted sum of the first and second loss values, or the average of the first and second loss values.

[0315] The first loss value and the second loss value can be determined by the first loss function and the second loss function, respectively. The first loss function and the second loss function can be represented by the following formulas (1) and (2):

[0316]

[0317]

[0318] Among them, e i represents the i-th vector in the data after quantization based on a preset code table, such as the i-th vector in the aforementioned basic coding training data; g represents the data before quantization based on the preset code table, such as the aforementioned initial intermediate coding training data; n represents the number of vectors in the data after quantization based on the preset code table; Loss1 represents the first loss value; x j y represents the element value of the j-th sample point in the training data; j represents the element value of the j-th sample point in the basic decoding training data; m represents the number of sample points; Loss2 represents the second loss value.

[0319] Furthermore, the reconstruction loss value can be expressed as shown in the following formula (3), and the preset objective function can be expressed as shown in the following formula (4).

[0320] Loss=Loss1+Loss2, formula (3)

[0321]

[0322] Where Loss represents the reconstruction loss value; θ1 represents the initial model parameters; θ2 represents the adjusted model parameters; and θ represents the partial derivative sign.

[0323] S803. For the initial residual coding structure and the initial residual decoding structure of the Lth layer, based on the main coding structure, the main decoding structure, and the training data, train the initial residual coding structure and the initial residual decoding structure of the Lth layer to obtain the Lth layer residual coding structure and the Lth layer residual decoding structure.

[0324] Where L is an integer greater than or equal to 1 and less than or equal to N.

[0325] In some embodiments, for the initial residual coding structure and the initial residual decoding structure of the Lth layer, the training data is subjected to dimensionality reduction and encoding based on the main coding structure and the (L-1)th layer residual coding structure to obtain the Lth residual training data; according to the main decoding structure, the training data and the Lth residual training data, the model parameters of the initial residual coding structure and the initial residual decoding structure of the Lth layer are adjusted to obtain the Lth layer residual coding structure and the Lth layer residual decoding structure; wherein, the (L-1)th layer residual coding structure is trained based on the (L-1)th layer initial residual coding structure.

[0326] For example, for the first-layer initial residual coding structure and the first-layer initial residual decoding structure, the trained main coding structure and main decoding structure are obtained through the above step S802. The training data is input into the main coding structure for dimensionality reduction and coding processing to obtain basic coding training data. Furthermore, first residual training data can be obtained, which is used to characterize the difference between the basic coding training data and the codewords corresponding to the basic coding training data determined based on the code table preset by the main coding structure. Simultaneously, the first coding training data is input into the main decoding structure for dimensionality increase and decoding processing to obtain basic decoding training data.

[0327] The first residual training data is input into the first-layer initial residual encoding structure. Dimensionality reduction and encoding are performed using preset initial model parameters to obtain the first encoded training data. The first encoded training data is then input into the first-layer initial residual decoding structure. Dimensionality increase and decoding are performed using preset initial model parameters to obtain the first decoded training data, which is then input into the main decoding structure. The main decoding structure performs dimensionality increase and decoding on the basic decoding training data and the first decoded training data to obtain the basic decoding training data. Based on the basic decoding training data and the training data, the model parameters of the first-layer initial residual encoding structure and the first-layer initial residual decoding structure are adjusted to obtain the first-layer residual encoding structure and the first-layer residual decoding structure.

[0328] It should be noted that adjusting the model parameters of the first-layer initial residual coding structure and the first-layer initial residual decoding structure is similar to the aforementioned step S802, and will not be repeated here.

[0329] Furthermore, based on the trained master coding structure, master decoding structure, first-layer residual coding structure, and first-layer residual decoding structure, the second-layer initial residual coding structure and the second-layer initial residual decoding structure can be trained to obtain the second-layer residual coding structure and the second-layer residual decoding structure.

[0330] Similarly, based on the trained master coding structure, master decoding structure, first-layer residual coding structure, first-layer residual decoding structure, ..., L-1-th layer residual coding structure, L-1-th layer residual decoding structure, the initial L-th layer residual coding structure and the initial L-th layer residual decoding structure can be trained to obtain the L-th layer residual coding structure and the L-th layer residual decoding structure. This process continues until the N-th layer residual coding structure and the N-th layer residual decoding structure are obtained.

[0331] S804. Based on the main coding structure and the N-layer residual coding structure, obtain the preset coding model; and based on the main decoding structure and the N-layer residual decoding structure, obtain the preset decoding model.

[0332] For example, a preset encoding model can be obtained by combining the main encoding structure and the N-layer residual encoding structure to encode the target multimedia data; and a preset decoding model can be obtained by combining the main decoding structure and the N-layer residual decoding structure to decode the processed multimedia data.

[0333] It should be noted that the above method can be used to train preset encoding models and preset decoding models with different values ​​of N, and store them in the model library.

[0334] In this embodiment, the encoding model is obtained by performing layered training on each encoding structure of the initial encoding model, thus ensuring the accuracy of the output data of each encoding structure in the subsequent encoding model.

[0335] It should be noted that the execution subject of the above steps S801-S804 can be a transmitting device or a receiving device, or other electronic devices, such as servers. This application embodiment does not limit this. For example, after training in other electronic devices, the preset encoding model can be ported to the transmitting device for use, and the preset decoding model can be ported to the receiving device for use.

[0336] In this embodiment, by performing layered training on the initial master coding structure and the N-layer initial residual coding structure, as well as the initial master decoding structure and the N-layer initial residual decoding structure in the coding and decoding model, a preset coding model and a preset decoding model are obtained, providing a foundation for the coding and decoding processing of multimedia data.

[0337] The following explanation uses the above encoding model (including a main encoding structure and a 1-layer residual encoding structure) and decoding model (including a main decoding structure and a 1-layer residual decoding structure) as examples to illustrate how to encode and decode multimedia data with a mono length of 9600 sample points. The multimedia data is represented as (1, 9600), where 1 represents mono and 9600 represents the length.

[0338] Encoding process: The transmitting device inputs multimedia data into the encoding model, which is then reduced in dimensionality by the first encoder network of the main encoding structure to initial intermediate encoded data of (64, 3200); then reduced in dimensionality by the first mapping network of the main encoding structure to basic encoded data of (64, 3200); the basic encoded data of (64, 3200) is quantized based on the first quantization network to obtain 3200 10-bit indices, which are the basic encoded bitstream.

[0339] The first residual data (64, 3200) output from the first quantization network is then input into the second encoder network and the second mapping network of the first layer residual coding structure to obtain the first coded data (64, 800). The data is then quantized by the second quantization network of the first layer residual coding structure to obtain 800 10-bit indices, which are sub-coded bitstream 1. The output bitstream is obtained by summing the data, which includes the basic coded bitstream and sub-coded bitstream 1.

[0340] Decoding process: The receiving device performs dequantization processing based on the 800 indices in the sub-coded bitstream 1 and the second dequantization network of the first residual decoding structure to obtain the first decoded data (64, 800). The data is then processed by the second decoder network of the first residual decoding structure to obtain the first sub-decoded bitstream (64, 3200), which is then output to the first feature splicing network of the main decoding structure.

[0341] The receiving device performs dequantization processing based on the 3200 10-bit indices in the basic encoded bitstream and the first dequantization network of the main decoding structure to obtain basic decoded data of (64, 3200). Based on the first feature concatenation network of the main decoding structure, the basic decoded data and the first sub-decoded bitstream are concatenated to obtain the final intermediate decoded data of (64, 3200). Finally, the data is processed based on the first decoder network of the main decoding structure to obtain multimedia data of (1, 9600).

[0342] Using the above method, multimedia data of (1,9600) can be encoded at the sending end device, reducing the storage space at the sending end device, and can be decoded at the receiving end device to restore multimedia resources.

[0343] Figure 16 This is a schematic diagram of the structure of a multimedia data processing device provided in an embodiment of this application, as shown below. Figure 16 As shown, the device 90 includes:

[0344] Acquisition module 901 is used to acquire target multimedia data;

[0345] The encoding module 902 is used to input the target multimedia data into a preset encoding model for encoding processing to obtain an output bitstream; wherein, the output bitstream includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1;

[0346] The output module 903 is used to obtain processed multimedia data based on the output bitstream and send the processed multimedia data to the receiving device; wherein the processed multimedia data includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N.

[0347] In one possible implementation, the encoding model includes a single main encoding structure and N residual encoding structures; the encoding module 902 is specifically used for:

[0348] Based on the main coding structure, the target multimedia data is encoded to obtain the basic encoded bitstream;

[0349] Based on each of the residual coding structures, the target multimedia data is encoded to obtain each of the sub-coded bitstreams.

[0350] In one possible implementation, the main coding structure includes a first encoder network and a first quantization network; based on the main coding structure, the target multimedia data is encoded to obtain the basic coded bitstream, including:

[0351] The first encoder network based on the main coding structure performs dimensionality reduction and coding processing on the target multimedia data to obtain basic coded data, and outputs the basic coded data to the first quantization network of the main coding structure.

[0352] Based on the first quantization network of the main coding structure, the codewords and their indices corresponding to the basic coding data are determined in a preset code table; and based on the first quantization network of the main coding structure, the basic coding bitstream is determined according to the index of the codewords corresponding to the basic coding data; wherein, the code table includes codewords with multiple preset indices.

[0353] In one possible implementation, the main encoding structure further includes a first mapping network; the encoding module 902 is specifically used for:

[0354] The first encoder network based on the main coding structure performs dimensionality reduction and encoding processing on the target multimedia data to obtain initial intermediate encoded data, and outputs the initial intermediate encoded data to the first mapping network of the main coding structure.

[0355] The first mapping network based on the main coding structure performs dimensionality reduction processing on the initial intermediate coding data to obtain basic coding data, and outputs the basic coding data to the first quantization network of the main coding structure.

[0356] In one possible implementation, after determining the codewords and their indices corresponding to the basic encoded data in a preset code table within a first quantization network based on the master coding structure, the method includes:

[0357] The first quantization network based on the master coding structure determines the first residual data according to the basic coding data and the codewords corresponding to the basic coding data, and outputs the first residual data to the first layer residual coding structure.

[0358] In one possible implementation, the i-th layer residual coding structure in the N-layer residual coding structure includes an (i+1)-th encoder network and an (i+1)-th quantization network; where i is an integer greater than or equal to 1 and less than or equal to N; the above coding module 902 is specifically used for:

[0359] Based on the (i+1)th encoder network of the i-th layer residual coding structure, the i-th residual data is subjected to dimensionality reduction and coding processing to obtain the i-th coded data, and the i-th coded data is output to the (i+1)th quantization network of the i-th layer residual coding structure; wherein, when i is 1, the first residual data is obtained by the main coding structure processing the target multimedia data; when i is not 1, the i-th residual data is obtained by the (i-1)th layer residual coding structure processing the (i-1)th residual data;

[0360] Based on the (i+1)th quantization network of the i-th layer residual coding structure, the codeword and its index corresponding to the i-th encoded data are determined in a preset code table, and the i-th sub-coded bitstream is determined based on the (i+1)th quantization network of the i-th layer residual coding structure and the index of the codeword corresponding to the i-th encoded data.

[0361] In one possible implementation, the i-th layer residual coding structure further includes an (i+1)-th mapping network; based on the (i+1)-th encoder network of the i-th layer residual coding structure, the encoding module 902 is specifically used for:

[0362] Based on the (i+1)th encoder network of the i-th layer residual coding structure, the i-th residual data is subjected to dimensionality reduction and coding processing to obtain the i-th intermediate coding data, and the i-th intermediate coding data is output to the (i+1)th mapping network of the i-th layer residual coding structure.

[0363] Based on the (i+1)th mapping network of the i-th layer residual coding structure, the i-th intermediate coding data is dimensionality reduced to obtain the i-th coding data, and the i-th coding data is output to the (i+1)th quantization network of the i-th layer residual coding structure.

[0364] In one possible implementation, after the encoding module 902 determines the codeword and its index corresponding to the i-th encoded data in a preset code table using the (i+1)-th quantization network based on the i-th layer residual coding structure, the encoding module 902 is specifically used for:

[0365] Based on the (i+1)th quantization network of the i-th layer residual coding structure, the (i+1)th residual data is determined according to the i-th encoded data and the codeword corresponding to the i-th encoded data, and the (i+1)th residual data is output to the (i+2)th encoder network of the (i+1)th layer residual coding structure.

[0366] In one possible implementation, the encoding module 902 described above is specifically used for:

[0367] Determine the network bandwidth between the sending and receiving devices;

[0368] The number of levels N of the coding model is determined based on the network bandwidth; wherein the number of levels is used to indicate the number of sub-coded bitstreams included in the output bitstream;

[0369] Based on the number of levels N, the encoding model corresponding to the number of levels N is determined from a preset model library; wherein, the model library includes encoding models with different numbers of levels N;

[0370] The target multimedia data is input into the encoding model corresponding to the number of layers N to obtain the output bitstream.

[0371] In one possible implementation, the output module 903 is specifically used for:

[0372] Determine the network bandwidth between the sending and receiving devices;

[0373] Based on the network bandwidth, determine the number P of sub-coded bitstreams; and based on the number P of sub-coded bitstreams, determine the output bitstream to be sent from the output bitstreams;

[0374] Based on the output bitstream to be sent, the processed multimedia data is obtained.

[0375] The multimedia data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0376] Figure 17 This is a schematic diagram of another multimedia data processing device provided in an embodiment of this application. Figure 17 As shown, the device 91 includes:

[0377] The receiving module 911 is used to receive processed multimedia data sent by the sending device; wherein the processed multimedia data is obtained by the sending device based on the output bitstream, and the output bitstream is obtained by the sending device based on a preset encoding model; the output bitstream includes a basic encoded bitstream and N sub-encoded bitstreams, where N is an integer greater than or equal to 1; the processed multimedia data includes a basic encoded bitstream and P sub-encoded bitstreams, where P is an integer greater than or equal to 1 and less than or equal to N;

[0378] The determining module 912 is used to determine encoding information based on the processed multimedia data; wherein the encoding information is used to indicate the bitstream information included in the processed multimedia data;

[0379] The decoding module 913 is used to input the processed multimedia data into a preset decoding model for decoding processing according to the encoding information, to obtain target multimedia data, and to play the target multimedia data.

[0380] In one possible implementation, the decoding module 913 described above is specifically used for:

[0381] Based on the encoding information, determine the number P of sub-coded bitstreams included in the processed multimedia data;

[0382] Based on the number P of the sub-coded bitstreams, a decoding model corresponding to the number P of the sub-coded bitstreams is determined from a preset model library; wherein, the model library includes multiple decoding models, and each decoding model can decode and process a different number of sub-coded bitstreams;

[0383] The processed multimedia data is input into the decoding model corresponding to the number P of the sub-encoded bitstreams for decoding processing to obtain the target multimedia data.

[0384] In one possible implementation, the decoding model includes a single main decoding structure and a P-layer residual decoding structure; the aforementioned decoding module 913 is specifically used for:

[0385] Based on the encoding information, determine the main decoding structure in the decoding model corresponding to the basic encoded bitstream, and the residual decoding structure in the decoding model corresponding to each sub-encoded bitstream;

[0386] The sub-coded bitstream is input into the residual decoding structure in the decoding model corresponding to the sub-coded bitstream for decoding processing to obtain the first sub-decoded bitstream;

[0387] The basic encoded bitstream and the first sub-decoded bitstream are input into the main decoding structure in the decoding model corresponding to the basic encoded bitstream for decoding processing to obtain the target multimedia data.

[0388] In one possible implementation, the j-th layer residual decoding structure in the P-layer residual decoding structure includes a second inverse quantization network, a second feature concatenation network, and a second decoder network; wherein j is 1, and P is greater than 1; the aforementioned decoding module 913 is specifically used for:

[0389] The second inverse quantization network based on the first layer residual decoding structure determines the codeword corresponding to the index in the first sub-encoded bitstream in the preset code table, obtains the first decoded data, and outputs the first decoded data to the second feature splicing network of the first layer residual decoding structure.

[0390] The second feature concatenation network based on the first layer residual decoding structure performs feature concatenation on the first decoded data output by the second inverse quantization network of the first layer residual decoding structure and the second sub-decoded bitstream output by the third decoder network of the second layer residual decoding structure to obtain the first intermediate decoded data, and outputs the first intermediate decoded data to the second decoder network of the first layer residual decoding structure.

[0391] The second decoder network, based on the first-layer residual decoding structure, performs dimensionality upscaling and decoding on the first intermediate decoded data to obtain the first sub-decoded bitstream.

[0392] In one possible implementation, the j-th layer residual decoding structure in the P-layer residual decoding structure includes a (j+1)-th inverse quantization network, a (j+1)-th feature concatenation network, and a (j+1)-th decoder network; where j is an integer greater than 1 and less than P; the aforementioned decoding module 913 is specifically used for:

[0393] Based on the j+1 inverse quantization network of the j-th layer residual decoding structure, the codeword corresponding to the index in the j-th sub-encoded bitstream is determined in the preset code table to obtain the j-th decoded data, and the j-th decoded data is output to the j+1 feature splicing network of the j-th layer residual decoding structure.

[0394] Based on the j+1 feature concatenation network of the j-th residual decoding structure, the j-th decoded data output by the j+1 inverse quantization network of the j-th residual decoding structure and the j+1 sub-decoded bitstream output by the j+2 decoder network of the j+1 residual decoding structure are concatenated to obtain the j-th intermediate decoded data, and the j-th intermediate decoded data is output to the j+1 decoder network of the j-th residual decoding structure.

[0395] Based on the j+1 decoder network of the j-th layer residual decoding structure, the j-th intermediate decoded data is subjected to dimensionality upscaling and decoding to obtain the j-th sub-decoded bitstream, and the j-th sub-decoded bitstream is output to the j-th feature splicing network of the j-1 layer residual decoding structure.

[0396] In one possible implementation, the j-th layer residual decoding structure in the P-layer residual decoding structure includes a P+1-th inverse quantization network and a P+1-th decoder network; where j is P; the decoding module 913 is specifically used for:

[0397] The P+1 dequantization network based on the P-th layer residual decoding structure determines the codeword corresponding to the index in the P-th sub-encoded bitstream in the preset code table, obtains the P-th decoded data, and outputs the P-th decoded data to the P+1 decoder network of the P-th layer residual decoding structure.

[0398] Based on the P+1 decoder network of the P-th layer residual decoding structure, the P-th decoded data is subjected to dimensionality upscaling and decoding to obtain the P-th sub-decoded bitstream, and the P-th sub-decoded bitstream is output to the P-th feature splicing network of the P-1 layer residual decoding structure.

[0399] In one possible implementation, the main decoding structure includes a first inverse quantization network, a first feature concatenation layer, and a first decoder network; the decoding module 913 is specifically used for:

[0400] The first inverse quantization network based on the main decoding structure determines the codeword corresponding to the index in the basic encoded bitstream in the preset code table, obtains the basic decoding data, and outputs the basic decoding data to the first feature concatenation network of the main decoding structure.

[0401] Based on the first feature concatenation network of the main decoding structure, the basic decoded data output by the first inverse quantization network of the received main decoding structure and the first sub-decoded bitstream output by the second decoder network of the received first layer residual decoding structure are concatenated to obtain the final intermediate decoded data, and the final intermediate decoded data is output to the first decoder network of the main decoding structure.

[0402] The first decoder network based on the main decoding structure performs dimensionality upscaling and decoding on the final intermediate decoded data to obtain the target multimedia data.

[0403] The multimedia data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0404] Figure 18 This is a schematic diagram of the structure of a codec model training device provided in an embodiment of this application. Figure 18 As shown, the device 92 includes:

[0405] The acquisition module 921 is used to acquire training data; wherein, the training data includes sample multimedia data;

[0406] The first training module 922 is used to train the initial master coding structure and the initial master decoding structure based on the training data to obtain the master coding structure and the master decoding structure.

[0407] The second training module 923 is used to train the initial residual coding structure and the initial residual decoding structure of the Lth layer based on the main coding structure, the main decoding structure, and the training data to obtain the initial residual coding structure and the initial residual decoding structure of the Lth layer; wherein, L is an integer greater than or equal to 1 and less than or equal to N;

[0408] The combination module 924 is used to obtain a preset encoding model based on the main encoding structure and the N-layer residual encoding structure; and to obtain a preset decoding model based on the main decoding structure and the N-layer residual decoding structure.

[0409] The preset encoding model is used to encode the target multimedia data described in the foregoing embodiments; the preset decoding model is used to decode the processed multimedia data described in the foregoing embodiments.

[0410] In one possible implementation, the first training module 922 is specifically used to perform dimensionality reduction and encoding processing on the training data based on the initial master encoding structure to obtain basic encoded training data; and to perform dimensionality increase and decoding processing on the basic encoded training data based on the initial master decoding structure to obtain basic decoded training data; and to adjust the model parameters of the initial master encoding structure and the initial master decoding structure based on the training data, the basic encoded training data, and the basic decoded training data to obtain the master encoding structure and the master decoding structure; wherein, the model parameters include weight values ​​and / or bias values.

[0411] In one possible implementation, the second training module 923 is specifically used to perform dimensionality reduction and encoding processing on the training data based on the main encoding structure and the (L-1)th layer residual encoding structure for the L-th layer initial residual encoding structure and the L-th layer initial residual decoding structure, to obtain L-th residual training data; wherein, the (L-1)th layer residual encoding structure is obtained by training based on the (L-1)th layer initial residual encoding structure; and the model parameters of the L-th layer initial residual encoding structure and the L-th layer initial residual decoding structure are adjusted according to the main decoding structure, the training data, and the L-th layer residual training data to obtain the L-th layer residual encoding structure and the L-th layer residual decoding structure.

[0412] The multimedia data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0413] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 19 As shown, the electronic device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus 1004. This electronic device may be, for example, the aforementioned transmitting device, receiving device, or server.

[0414] In a specific implementation, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above-described method.

[0415] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0416] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0417] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0418] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0419] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0420] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0421] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0422] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0423] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0424] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0425] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0426] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0427] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0428] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A multimedia data processing method, characterized by, The method is applied to a sending end device, and the method comprises: obtaining target multimedia data; inputting the target multimedia data into a preset encoding model for encoding processing to obtain an output code stream; wherein the output code stream comprises one basic encoding code stream and N sub-encoding code streams, N being an integer greater than or equal to 1; obtaining processed multimedia data based on the output code stream, and sending the processed multimedia data to a receiving end device; wherein the processed multimedia data is used for decoding processing based on a preset decoding model to obtain the target multimedia data; the processed multimedia data comprises one basic encoding code stream and P sub-encoding code streams, P being an integer greater than or equal to 1 and less than or equal to N.

2. The method of claim 1, wherein, The encoding model comprises one main encoding structure and N residual encoding structures; inputting the target multimedia data into a preset encoding model for encoding processing to obtain an output code stream, comprising: encoding processing of the target multimedia data based on the main encoding structure to obtain the basic encoding code stream; encoding processing of the target multimedia data based on each residual encoding structure to obtain each sub-encoding code stream.

3. The method of claim 2, wherein, The main encoding structure comprises a first encoder network and a first quantization network; encoding processing of the target multimedia data based on the main encoding structure to obtain the basic encoding code stream, comprising: dimension reduction processing and encoding processing of the target multimedia data based on the first encoder network of the main encoding structure to obtain basic encoding data, and outputting the basic encoding data to the first quantization network of the main encoding structure; determining a code word corresponding to the basic encoding data and an index of the code word in a preset code table based on the first quantization network of the main encoding structure, and determining a basic encoding code stream based on the first quantization network of the main encoding structure according to the index of the code word corresponding to the basic encoding data; wherein the code table comprises a plurality of preset index code words.

4. The method of claim 3, wherein, The main encoding structure further comprises a first mapping network; dimension reduction processing and encoding processing of the target multimedia data based on the first encoder network of the main encoding structure to obtain basic encoding data, comprising: dimension reduction processing and encoding processing of the target multimedia data based on the first encoder network of the main encoding structure to obtain initial intermediate encoding data, and outputting the initial intermediate encoding data to the first mapping network of the main encoding structure; dimension reduction processing of the initial intermediate encoding data based on the first mapping network of the main encoding structure to obtain basic encoding data, and outputting the basic encoding data to the first quantization network of the main encoding structure.

5. The method of claim 3, wherein, After determining the code word corresponding to the basic encoding data and the index of the code word in a preset code table based on the first quantization network of the main encoding structure, the method comprises: determining first residual data based on the first quantization network of the main encoding structure according to the basic encoding data and the code word corresponding to the basic encoding data, and outputting the first residual data to the first layer residual encoding structure.

6. The method of claim 2, wherein, The i-th residual coding structure in the N-layer residual coding structure comprises an i+1-th encoder network and an i+1-th quantization network; wherein i is an integer greater than or equal to 1 and less than or equal to N; each of the residual coding structures is used to encode the target multimedia data to obtain each of the sub-encoding code streams, comprising: The i+1-th encoder network of the i-th residual coding structure is used to reduce the dimension of the i-th residual data and encode the i-th residual data to obtain i-th encoding data, and the i+1-th quantization network of the i-th residual coding structure is used to output the i-th encoding data; wherein when i is 1, the first residual data is obtained by processing the target multimedia data by the main coding structure; when i is not 1, the i-th residual data is obtained by processing the i-1-th residual data by the i-1-th residual coding structure; The i+1-th quantization network of the i-th residual coding structure is used to determine the code word corresponding to the i-th encoding data and the index of the code word in a preset code table, and the i+1-th quantization network of the i-th residual coding structure is used to determine the i-th sub-encoding code stream according to the index of the code word corresponding to the i-th encoding data.

7. The method of claim 6, wherein, The i-th residual coding structure further comprises an i+1-th mapping network; the i+1-th encoder network of the i-th residual coding structure is used to reduce the dimension of the i-th residual data and encode the i-th residual data to obtain i-th encoding data, comprising: The i+1-th encoder network of the i-th residual coding structure is used to reduce the dimension of the i-th residual data and encode the i-th residual data to obtain i-th intermediate encoding data, and the i+1-th mapping network of the i-th residual coding structure is used to output the i-th intermediate encoding data; The i+1-th mapping network of the i-th residual coding structure is used to reduce the dimension of the i-th intermediate encoding data to obtain the i-th encoding data, and the i+1-th quantization network of the i-th residual coding structure is used to output the i-th encoding data.

8. The method of claim 6, wherein, After determining the code word corresponding to the i-th encoding data and the index of the code word in a preset code table by the i+1-th quantization network of the i-th residual coding structure, the method comprises: The i+1-th quantization network of the i-th residual coding structure is used to determine the i+1-th residual data according to the i-th encoding data and the code word corresponding to the i-th encoding data, and the i+2-th encoder network of the i+1-th residual coding structure is used to output the i+1-th residual data.

9. The method according to any one of claims 1 to 8, characterized in that, The method of inputting the target multimedia data into a preset encoding model to obtain an output code stream comprises: Determining the network bandwidth between the sending end device and the receiving end device; According to the network bandwidth, determining the number of layers N of the encoding model; wherein the number of layers is used to indicate the number of sub-encoding code streams included in the output code stream; According to the number of layers N, determining the encoding model corresponding to the number of layers N from a preset model library; wherein the model library comprises encoding models of different numbers of layers N; Inputting the target multimedia data into the encoding model corresponding to the number of layers N to obtain an output code stream.

10. The method according to any one of claims 1 to 8, characterized in that, The method comprises the following steps: determining the network bandwidth between the sending terminal device and the receiving terminal device; determining the number P of sub-encoding code streams according to the network bandwidth; and determining the output code stream to be sent from the output code stream according to the number P of sub-encoding code streams; obtaining the processed multimedia data according to the output code stream to be sent.

11. A multimedia data processing method, characterized by, The method is applied to the receiving terminal device, and the method comprises the following steps: receiving the processed multimedia data sent by the sending terminal device; wherein the processed multimedia data is obtained by the sending terminal device based on the output code stream, the output code stream is obtained by the sending terminal device based on a preset encoding model for encoding processing of target multimedia data; the output code stream comprises one basic encoding code stream and N sub-encoding code streams, N is an integer greater than or equal to 1; the processed multimedia data comprises one basic encoding code stream and P sub-encoding code streams, P is an integer greater than or equal to 1 and less than or equal to N; determining the encoding information according to the processed multimedia data; wherein the encoding information is used for indicating the code stream information included in the processed multimedia data; inputting the processed multimedia data into a preset decoding model for decoding processing according to the encoding information, obtaining target multimedia data, and playing the target multimedia data.

12. The method of claim 11, wherein, The method comprises the following steps: determining the number P of sub-encoding code streams included in the processed multimedia data according to the encoding information; determining the decoding model corresponding to the number P of sub-encoding code streams from a preset model library according to the number P of sub-encoding code streams; wherein the model library comprises a plurality of decoding models, and the number of sub-encoding code streams that can be decoded and processed by each decoding model is different; inputting the processed multimedia data into the decoding model corresponding to the number P of sub-encoding code streams for decoding processing to obtain target multimedia data.

13. The method of claim 12, wherein, The decoding model comprises one layer of main decoding structure and P layers of residual decoding structure; inputting the processed multimedia data into the decoding model corresponding to the number P of sub-encoding code streams for decoding processing to obtain target multimedia data comprises the following steps: determining the main decoding structure in the decoding model corresponding to the basic encoding code stream and the residual decoding structure in the decoding model corresponding to each sub-encoding code stream according to the encoding information; inputting the sub-encoding code stream into the residual decoding structure in the decoding model corresponding to the sub-encoding code stream for decoding processing to obtain a first sub-decoding code stream; inputting the basic encoding code stream and the first sub-decoding code stream into the main decoding structure in the decoding model corresponding to the basic encoding code stream for decoding processing to obtain the target multimedia data.

14. The method of claim 13, wherein, The jth layer of residual decoding structure in the P layers of residual decoding structure comprises a second inverse quantization network, a second feature splicing network and a second decoder network; wherein j is 1, and P is greater than 1. The sub-encoding code stream is input into a residual decoding structure in the decoding model corresponding to the sub-encoding code stream for processing to obtain a first sub-decoding code stream, comprising: a second dequantization network based on the first-layer residual decoding structure, determining a code word corresponding to an index in the first sub-encoding code stream in a preset code table to obtain first decoding data, and outputting the first decoding data to a second feature splicing network of the first-layer residual decoding structure; a second feature splicing network based on the first-layer residual decoding structure, performing feature splicing on the first decoding data output by the second dequantization network of the first-layer residual decoding structure and a second sub-decoding code stream output by a third decoder network of a second-layer residual decoding structure to obtain first intermediate decoding data, and outputting the first intermediate decoding data to a second decoder network of the first-layer residual decoding structure; a second decoder network based on the first-layer residual decoding structure, performing dimensionality increasing processing and decoding processing on the first intermediate decoding data to obtain the first sub-decoding code stream.

15. The method of claim 14, wherein, The jth residual decoding structure in the P-layer residual decoding structure comprises a (j+1)th dequantization network, a (j+1)th feature splicing network, and a (j+1)th decoder network; wherein j is an integer greater than 1 and less than P; the method further comprises: a (j+1)th dequantization network based on the jth residual decoding structure, determining a code word corresponding to an index in the jth sub-encoding code stream in a preset code table to obtain jth decoding data, and outputting the jth decoding data to a (j+1)th feature splicing network of the jth residual decoding structure; a (j+1)th feature splicing network based on the jth residual decoding structure, performing feature splicing on the jth decoding data output by the (j+1)th dequantization network of the jth residual decoding structure and a (j+1)th sub-decoding code stream output by a (j+2)th decoder network of a (j+1)th residual decoding structure to obtain jth intermediate decoding data, and outputting the jth intermediate decoding data to a (j+1)th decoder network of the jth residual decoding structure; a (j+1)th decoder network based on the jth residual decoding structure, performing dimensionality increasing processing and decoding processing on the jth intermediate decoding data to obtain the jth sub-decoding code stream, and outputting the jth sub-decoding code stream to a jth feature splicing network of a (j-1)th residual decoding structure.

16. The method of claim 15, wherein, The jth residual decoding structure in the P-layer residual decoding structure comprises a (P+1)th dequantization network and a (P+1)th decoder network; wherein j is P; the method further comprises: a (P+1)th dequantization network based on the P-layer residual decoding structure, determining a code word corresponding to an index in the Pth sub-encoding code stream in a preset code table to obtain Pth decoding data, and outputting the Pth decoding data to a (P+1)th decoder network of the P-layer residual decoding structure; a (P+1)th decoder network based on the P-layer residual decoding structure, performing dimensionality increasing processing and decoding processing on the Pth decoding data to obtain the Pth sub-decoding code stream, and outputting the Pth sub-decoding code stream to a Pth feature splicing network of a (P-1)th residual decoding structure.

17. The method of claim 13, wherein, The main decoding structure includes a first dequantization network, a first feature splicing layer, and a first decoder network; the basic encoding code stream and the first sub-decoding code stream are input into the main decoding structure in the decoding model corresponding to the basic encoding code stream for processing to obtain the target multimedia data, including: Based on the first dequantization network of the main decoding structure, the index corresponding to the code word in the basic encoding code stream is determined in the preset code table to obtain basic decoding data, and the basic decoding data is output to the first feature splicing network of the main decoding structure; Based on the first feature splicing network of the main decoding structure, the basic decoding data output by the first dequantization network of the main decoding structure received and the first sub-decoding code stream output by the second decoder network of the first layer residual decoding structure received are spliced to obtain final intermediate decoding data, and the final intermediate decoding data is output to the first decoder network of the main decoding structure; Based on the first decoder network of the main decoding structure, the final intermediate decoding data is processed by dimensionality reduction and decoding to obtain the target multimedia data.

18. A method for training a coding model, comprising: The coding and decoding model includes one layer of initial main encoding structure and N layers of initial residual encoding structure, and one layer of initial main decoding structure and N layers of initial residual decoding structure; the method includes: Obtaining training data; wherein the training data includes sample multimedia data; Based on the training data, the initial main encoding structure and the initial main decoding structure are trained to obtain a main encoding structure and a main decoding structure; For the Lth layer initial residual encoding structure and the Lth layer initial residual decoding structure, according to the main encoding structure, the main decoding structure, and the training data, the Lth layer initial residual encoding structure and the Lth layer initial residual decoding structure are trained to obtain the Lth layer residual encoding structure and the Lth layer residual decoding structure; wherein L is an integer greater than or equal to 1 and less than or equal to N; According to the main encoding structure and the N-layer residual encoding structure, a preset encoding model is obtained; and according to the main decoding structure and the N-layer residual decoding structure, a preset decoding model is obtained; The preset encoding model is used for encoding processing on the target multimedia data in any one of claims 1-10; and the preset decoding model is used for decoding processing on the processed multimedia data in any one of claims 11-17.

19. The method of claim 18, wherein, Based on the training data, the initial main encoding structure and the initial main decoding structure are trained to obtain a main encoding structure and a main decoding structure, including: Based on the initial main encoding structure, the training data is processed by dimensionality reduction and encoding to obtain basic encoding training data; and based on the initial main decoding structure, the basic encoding training data is processed by dimensionality reduction and decoding to obtain basic decoding training data; Based on the training data, the basic encoding training data and the basic decoding training data, model parameters of the initial main encoding structure and the initial main decoding structure are adjusted to obtain a main encoding structure and a main decoding structure; wherein the model parameters include weight values and / or bias values.

20. The method of any one of claims 18 or 19, wherein, According to the main encoding structure, the main decoding structure, and the training data, the Lth layer initial residual encoding structure and the Lth layer initial residual decoding structure are trained to obtain an Lth layer residual encoding structure and an Lth layer residual decoding structure, including: For the Lth layer initial residual encoding structure and the Lth layer initial residual decoding structure, based on the main encoding structure and the (L-1)th layer residual encoding structure, the training data is processed by dimension reduction and encoding to obtain Lth residual training data; wherein the (L-1)th layer residual encoding structure is obtained based on the (L-1)th layer initial residual encoding structure; According to the main decoding structure, the training data and the Lth residual training data, the model parameters of the Lth layer initial residual encoding structure and the Lth layer initial residual decoding structure are adjusted to obtain the Lth layer residual encoding structure and the Lth layer residual decoding structure.

21. A multimedia data processing apparatus, characterized by comprising: The device is applied to a sending end device, and the device comprises: An acquisition module is configured to acquire target multimedia data; An encoding module is configured to input the target multimedia data into a preset encoding model for encoding processing to obtain an output code stream; wherein the output code stream comprises one basic encoding code stream and N sub-encoding code streams, and N is an integer greater than or equal to 1; An output module is configured to obtain processed multimedia data according to the output code stream, and send the processed multimedia data to a receiving end device; wherein the processed multimedia data comprises one basic encoding code stream and P sub-encoding code streams, and P is an integer greater than or equal to 1 and less than or equal to N.

22. A multimedia data processing apparatus, characterized by comprising: The device is applied to a receiving end device, and the device comprises: A receiving module is configured to receive processed multimedia data sent by a sending end device; wherein the processed multimedia data is obtained by the sending end device based on an output code stream, and the output code stream is obtained by the sending end device based on a preset encoding model; the output code stream comprises one basic encoding code stream and N sub-encoding code streams, and N is an integer greater than or equal to 1; the processed multimedia data comprises one basic encoding code stream and P sub-encoding code streams, and P is an integer greater than or equal to 1 and less than or equal to N; A determination module is configured to determine encoding information according to the processed multimedia data; wherein the encoding information is used to indicate code stream information included in the processed multimedia data; A decoding module is configured to input the processed multimedia data into a preset decoding model for decoding processing to obtain target multimedia data, and play the target multimedia data.

23. An electronic device, comprising: Comprise: Memory, processor; The memory stores computer execution instructions; The processor executes the computer-executed instructions stored in the memory, so that the processor executes the multimedia data processing method in any one of claims 1-10, the multimedia data processing method in any one of claims 11-17, or the codec model training method in any one of claims 18-20.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executed instructions, and the computer-executed instructions are executed by the processor to implement the multimedia data processing method in any one of claims 1-10, the multimedia data processing method in any one of claims 11-17, or the codec model training method in any one of claims 18-20.

Images