Decoding method and electronic device
By including model identifiers in the encoding stream, synchronous update or switching between the encoding model and the decoding model is solved, and the encoding performance bottleneck caused by insufficient update iteration of AI models in the prior art is improved, and the long-term evolution and efficiency of audio and video encoding are improved.
Patent Information
- Application Number
- PCT/CN2024/091179
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-05-06
- Publication Date
- 2024-12-05
AI Technical Summary
There are bottlenecks in the long-term evolution and performance optimization of existing audio and video encoding and codec technologies, especially due to insufficient update and iteration of AI models, the performance of encoding standards cannot be effectively improved.
A decoding method is proposed, by including a model identifier in the encoding stream, instructing the second electronic device to obtain the model information of the decoding model, synchronous update or switching between the encoding model and the decoding model, and instructing the second electronic device to obtain using a preset identification value. Or switch the decoding model to reduce bandwidth requirements and bitrate overhead.
It has achieved long-term evolution and performance optimization of coding standards, improved the coding efficiency of audio and video data, and adapted to the needs of future technological development.
Smart Images

Figure CN2024091179_05122024_PF_FP_ABST
Abstract
Description
Decoding method and electronic device
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 30, 2023, with application number 202310628717.X and application name “Decoding Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The embodiments of the present application relate to the field of coding and decoding, and in particular to a decoding method and electronic device. Background Art
[0003] With the development of high-definition audio and video and three-dimensional (3D) audio and video, people have an increasingly higher demand for audio and video quality, which has led to a rapid increase in the amount of audio and video data in media applications. This has also made audio and video compression an indispensable part of media applications such as media communications and media broadcasting.
[0004] Traditional audio and video compression technologies are based on the fundamental principles of signal processing. They exploit temporal and spatial correlations in signals to compress raw audio and video data, reducing the data volume and facilitating transmission or storage. With the continuous development and maturity of artificial intelligence (AI), AI has also been introduced into the audio and video codec field, improving encoding performance.
[0005] Generally, after the coding standard is standardized, the AI models involved will generally not be updated or iterated, which will affect the long-term evolution and performance optimization of the coding standard.
[0006] Summary of the Invention
[0007] In view of this, the present application provides a decoding method and an electronic device.
[0008] In a first aspect, an embodiment of the present application provides an encoding method applied to a first electronic device, the method comprising: first, obtaining a signal to be encoded; then, generating a first code stream, the first code stream including a second code stream and a model identifier, the second code stream being obtained by encoding the signal to be encoded based on the encoding model, the value of the model identifier being a preset identifier value, the preset identifier value being used to instruct the second electronic device to obtain model information of a decoding model, the decoding model corresponding to the encoding model.
[0009] In this way, the first electronic device sets the value of the model identifier in the first code stream to a preset identifier value to instruct the second electronic device to obtain model information of the decoding model. After the first code stream is transmitted to the second electronic device, the second electronic device can obtain the model information of the decoding model based on the preset identifier value, thereby achieving synchronous updating of the decoding model of the second electronic device and the encoding model of the first electronic device, or achieving synchronous switching between the decoding model used by the second electronic device for decoding and the encoding model used by the first electronic device for encoding. This is conducive to the long-term evolution of coding standards and optimization of coding performance.
[0010] Exemplarily, both the encoding model and the decoding model can be implemented through neural networks; therefore, the encoding model can also be called an AI encoding model, and the decoding model can also be called an AI decoding model.
[0011] In one possible approach, the encoding model and the decoding model are two independent models; in this case, the encoding model and the decoding model can be trained jointly. In another possible approach, the encoding model and the decoding model are two parts of a single model.
[0012] It should be understood that when the encoding model and the decoding model are two parts of a single model, the preset identification value can be used to instruct the second electronic device to obtain model information of the encoding model and model information of the decoding model; thus, the second electronic device can reconstruct the encoding model based on the model information of the encoding model and reconstruct the decoding model based on the model information of the decoding model. This application is described using the example of the encoding model and the decoding model being two independent models.
[0013] In one possible approach, a new identifier (i.e., a flag) or a new field can be added to the first bitstream as a model identifier. In another possible approach, an existing field in the first bitstream can be reused and used as a model identifier, which is not limited in this application. When reusing an existing field in the first bitstream and using the existing field as a model identifier, this application does not need to change the decoding syntax, that is, it can implement model updates or switching while achieving backward compatibility.
[0014] Exemplarily, the signal to be encoded may include but is not limited to: an audio signal, an image, and a video signal.
[0015] According to a first aspect, the first codestream also includes a third codestream, which is generated based on model information of a decoding model. A preset identification value can be used to indicate that the first codestream includes the third codestream. In this case, the preset identification value can be used to instruct the second electronic device to obtain the model information of the decoding model from the first codestream. In this way, the first electronic device can transmit the model information of the decoding model to the second electronic device, so that the second electronic device can reconstruct the decoding model for decoding based on the model information of the decoding model.
[0016] According to the first aspect, or any implementation of the first aspect above, the method further includes: generating a third code stream based on model information of the decoding model; generating the first code stream includes: encapsulating the second code stream and the third code stream to obtain the first code stream.
[0017] According to the first aspect, or any implementation of the first aspect above, the first codestream further includes index information corresponding to model information of the decoding model. The preset identification value is used to indicate that the first codestream includes index information corresponding to the model information of the decoding model. In this case, the preset identification value can be used to instruct the second electronic device to obtain the model information of the decoding model from the third electronic device based on the index information parsed from the first codestream.
[0018] In this way, the first electronic device only needs to transmit the index information corresponding to the model information of the decoding model. Compared with transmitting the model information of the decoding model, the code rate overhead is smaller and the bandwidth requirement can be reduced.
[0019] Illustratively, the index information may be a Uniform Resource Locator (URL), which is not limited in this application.
[0020] According to the first aspect, or any implementation of the first aspect above, generating the first bitstream includes: obtaining the first bitstream based on the second bitstream and index information corresponding to model information of the decoding model.
[0021] According to the first aspect, or any implementation of the first aspect above, the preset identification value is further used to indicate that the model information of the decoding model is stored in a preset storage location of the third device. In this case, the preset identification value can be used to instruct the second electronic device to obtain the model information of the decoding model from the preset storage location of the third electronic device; the preset storage location can be a location pre-agreed upon by the first electronic device and the second electronic device.
[0022] In this way, the first electronic device does not need to transmit index information corresponding to the model information of the decoding model, nor does it need to transmit the model information of the decoding model; thus, the bit rate overhead is smaller and the bandwidth requirement is lower.
[0023] It should be noted that, assuming that the method by which the second electronic device obtains model information from the first code stream is referred to as the first method; the method by which the second electronic device obtains model information based on the index information corresponding to the model information obtained from the first code stream is referred to as the second method; and the method by which the second electronic device obtains model information from a preset storage location of a third electronic device is referred to as the third method. In one possible method, the first electronic device and the second electronic device may pre-agree on the method by which the second electronic device obtains model information of the decoding model. During the encoding and decoding process, when the first electronic device and the second electronic device agree on the first method, the first electronic device may generate a third code stream based on the model information of the decoding model; and encapsulate the second code stream and the third code stream to obtain the first code stream. During the encoding and decoding process, when the first electronic device and the second electronic device agree on the second method, the first electronic device may generate the first code stream based on the second code stream and the index information corresponding to the model information of the decoding model. During the encoding and decoding process, when the first electronic device and the second electronic device agree on the third method, the first electronic device may use the second code stream to generate the first code stream.
[0024] In one possible approach, during the encoding and decoding process, the first electronic device may select one of the first, second, and third approaches as a target approach. When the target approach is the first approach, the first electronic device may generate a third code stream based on model information of the decoding model; and encapsulate the second and third code streams to obtain the first code stream. When the target approach is the second approach, the first electronic device may generate the first code stream based on the second code stream and index information corresponding to the model information of the decoding model. When the target approach is the third approach, the first electronic device may use the second code stream to generate the first code stream.
[0025] According to the first aspect, or any implementation of the first aspect above, the model identifier is a field of an elementary stream (ES).
[0026] For example, an ES encapsulation format may include multiple fields, and an ES field may refer to a field included in the ES encapsulation format. In other words, after data is encapsulated according to the ES encapsulation format, an ES packet may be obtained. An ES packet may include multiple fields, and an ES field may refer to a field in the ES packet.
[0027] According to the first aspect, or any implementation of the first aspect above, when the target codestream type of the first codestream is a codestream type encapsulated according to the target container format or a codestream type encapsulated according to the target transport protocol, the model identifier is a field of the target container format. In this way, the second electronic device can obtain the value of the model identifier without parsing the ES packet.
[0028] Exemplarily, the target container format includes but is not limited to the transport stream (TS) format, MP4 format, etc., and this application does not impose any limitation on this.
[0029] Exemplarily, the target container format may include multiple fields, and the fields of the target container format may refer to the fields included in the target container format. Alternatively, after data is encapsulated according to the target container format, an encapsulation packet may be obtained, and the encapsulation packet may include multiple fields. The fields of the target container format may refer to the fields in the encapsulation packet obtained by encapsulating according to the target container format. For example, when the target container format is TS format / MP4 format, the fields of the target container format may refer to the fields in the TS packet.
[0030] Exemplarily, the target transmission protocol may include multiple ones, such as a streaming media-based transmission protocol (HTTP Live Streaming, HLS), a real-time transport protocol (Real-time Transport Protocol, RTP), etc., which is not limited in this application.
[0031] According to the first aspect, or any implementation of the first aspect above, when the target codestream type of the first codestream is a codestream type encapsulated according to the target transport protocol, the model identifier is a field of the target transport protocol. In this way, the second electronic device can obtain the value of the model identifier without decapsulating the first codestream according to the target container format or parsing the ES packet.
[0032] Illustratively, the target transmission protocol's encapsulation format may include multiple fields, and the target transmission protocol's fields may refer to the fields included in the target transmission protocol's encapsulation format. Alternatively, data may be encapsulated according to the target transmission protocol's encapsulation format to obtain an encapsulation packet, which may include multiple fields, and the target transmission protocol's fields may refer to the fields in the encapsulation packet obtained by encapsulation according to the target transmission protocol.
[0033] According to the first aspect, or any implementation of the first aspect above, the method further includes: determining whether the decoding model is an updated decoding model; and when the decoding model is an updated decoding model, setting the value of the model identifier to a preset identifier value. In this case, the preset identifier value is used to instruct the second electronic device to obtain model information of the updated decoding model. In this case, the encoding model may not be updated, and only the decoding model may be updated (e.g., an optimized decoding model); in this case, the updated decoding model corresponds to the encoding model.
[0034] In one possible scenario, both the coding model and the decoding model are updated; the method further includes: determining whether both the coding model and the decoding model are updated; encoding the signal to be coded based on the coding model to obtain a second code stream, including: when it is determined that both the coding model and the decoding model are updated, encoding the signal to be coded based on the updated coding model to obtain a second code stream; at this time, the preset identification value is used to instruct the second electronic device to obtain model information of the updated decoding model, and the updated decoding model corresponds to the updated coding model.
[0035] In one possible scenario, the first electronic device includes multiple encoding models and multiple decoding models (the encoding models correspond one-to-one to the decoding models), and the method further includes: determining whether to switch the encoding model used for encoding; when it is determined that the encoding model used for encoding has been switched, encoding the to-be-encoded signal based on the switched encoding model to obtain a second code stream; at this time, the preset identification value is used to instruct the second electronic device to obtain model information of the switched decoding model, and the switched decoding model corresponds to the switched encoding model.
[0036] It should be understood that regardless of whether the encoding model and decoding model in the first electronic device are updated (or regardless of whether the decoding model is updated), and regardless of whether the encoder in the first electronic device switches the encoding model, the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value. In this case, the preset identifier value can be used to represent the type of the decoding model (or encoding model); correspondingly, after obtaining the preset identifier value, the second electronic device determines whether the second electronic device has a decoding model of the type corresponding to the preset identifier value; if the second electronic device does not have a decoding model of the type corresponding to the preset identifier value, it obtains model information of the decoding model.
[0037] According to the first aspect, or any implementation of the first aspect above, encapsulating the second code stream and the third code stream to obtain the first code stream includes: encapsulating the second code stream and the third code stream according to the target code stream type to obtain the first code stream.
[0038] It should be noted that, in one possible approach, the first electronic device and the second electronic device may pre-agree on a target code stream type. In this way, the first electronic device encapsulates the second code stream and the third code stream according to the target code stream type, and the second electronic device decapsulates the first code stream according to the target code stream type.
[0039] In one possible approach, during the encoding process, the first electronic device may select a code stream type from multiple code stream types as a target code stream type as required; correspondingly, the second electronic device identifies the code stream type of the first code stream (i.e., the target code stream type) and then decapsulates the first code stream according to the target code stream type.
[0040] According to the first aspect, or any implementation of the first aspect above, the target bitstream type is an ES type, and encapsulating the second and third bitstreams according to the target bitstream type to obtain the first bitstream includes: encapsulating the third bitstream and the second bitstream into a first ES packet, and using the first ES packet as the first bitstream. That is, the first bitstream is an ES stream, and the second and third bitstreams are encapsulated and transmitted in the same ES packet. In this way, after obtaining the ES packet, the ES packet can be directly transmitted without further encapsulation according to the target container format or target transmission protocol, thereby reducing bitrate overhead and bandwidth requirements. Furthermore, encapsulating the second and third bitstreams in the same ES packet allows the second and third bitstreams to be sent on the same channel, reducing transmission complexity.
[0041] According to the first aspect, or any implementation of the first aspect above, the target bitstream type is a bitstream type encapsulated in a target container format, and encapsulating the second bitstream and the third bitstream in accordance with the target bitstream type to obtain the first bitstream includes: encapsulating the third bitstream and the second bitstream into a second ES packet; and encapsulating the second ES packet in accordance with the target container format to obtain the first bitstream. The ES packet may be divided into multiple small packets and then encapsulated in accordance with the target container format. This facilitates bitstream distribution and audio and video synchronization control by the bitstream distribution device.
[0042] For example, when the target container format is TS format and the first code stream is a TS stream, the second and third code streams can be encapsulated in the same ES packet, and the ES packet can be encapsulated into a TS stream for transmission. When the target container format is MP4 format and the first code stream is an MP4 stream, the second and third code streams can be encapsulated in the same ES packet, and the ES packet can be encapsulated into an MP4 TS stream.
[0043] According to the first aspect, or any implementation of the first aspect above, the target stream type is a stream type encapsulated in a target container format, and encapsulating the second and third streams in accordance with the target stream type to obtain the first stream includes: encapsulating the third stream into a third ES packet and the second stream as a fourth ES packet; and encapsulating the third and fourth ES packets in accordance with the target container format, setting the type of the third ES packet to a preset type and the type of the fourth ES packet to a media type to obtain the first stream. In other words, the second and third streams are encapsulated and transmitted in different ES packets. In this manner, compared to encapsulating the second and third streams in the same ES packet, the second electronic device does not need to split the ES packet to parse the second and third streams, thereby reducing the complexity of the parsing process.
[0044] Exemplarily, the media type may include but is not limited to: audio type, video type, and image type.
[0045] According to the first aspect, or any implementation of the first aspect above, the preset type is a metadata type, that is, the third code stream is encapsulated as metadata.
[0046] According to the first aspect, or any implementation of the first aspect above, the target stream type is a stream type encapsulated in a target container format, and encapsulating the second and third streams in accordance with the target stream type to obtain the first stream includes: encapsulating the third stream into a fifth ES packet and the second stream as a sixth ES packet; encapsulating the fifth ES packet in accordance with the target container format and setting the type of the fifth ES packet to a preset type to obtain a fourth stream; encapsulating the sixth ES packet in accordance with the target container format and setting the type of the sixth ES packet to a media type to obtain a fifth stream; and combining the fourth and fifth streams to obtain the first stream. In other words, the second and third streams are separately encapsulated in accordance with the target container format, for example, encapsulating the second and third streams into two TS streams. In this way, compared to encapsulating the second and third streams in a single TS stream, the second electronic device does not need to split the TS stream to parse the second and third streams, thereby reducing the complexity of the parsing process.
[0047] It should be noted that combining the fourth code stream and the fifth code stream may be understood as combining only the fourth code stream and the fifth code stream.
[0048] According to the first aspect, or any implementation of the first aspect above, the target bitstream type is a bitstream type encapsulated according to the target transmission protocol, and encapsulating the second and third bitstreams according to the target bitstream type to obtain the first bitstream includes: encapsulating the third bitstream into a seventh ES packet and the second bitstream into an eighth ES packet; encapsulating the seventh ES packet according to the target container format and setting the type of the seventh ES packet to a preset type to obtain the sixth bitstream; encapsulating the eighth ES packet according to the target container format and setting the type of the eighth ES packet to a media type to obtain the seventh bitstream; and encapsulating the sixth and seventh bitstreams according to the target transmission protocol to obtain the first bitstream. That is, after encapsulating the second and third bitstreams according to the target container format, they are then encapsulated according to the target transmission protocol. Second bitstreams with different reconstruction accuracies (which can be understood as the accuracy of the reconstructed signal obtained by decoding the second bitstream) can be generated. This facilitates the bitstream distribution device to distribute the first bitstream containing the second bitstreams with different reconstruction accuracies according to bandwidth, ensuring smooth playback of the reconstructed signal.
[0049] It should be noted that the first code stream obtained by combining the fourth code stream and the fifth code stream only includes the fourth code stream and the fifth code stream; and the first code stream obtained by encapsulating the fourth code stream and the fifth code stream according to the target transmission protocol, in addition to including the sixth code stream and the seventh code stream, also includes the fields included in the encapsulation format of the target transmission protocol.
[0050] According to the first aspect, or any implementation of the first aspect above, the first bitstream further includes an index file, the index file including index information for the sixth bitstream and index information for the seventh bitstream, and the index information for the sixth bitstream in the index file precedes the index information for the seventh bitstream. In this way, the second electronic device can first obtain the model information used to reconstruct the decoding model, thereby reducing the delay in decoding the second bitstream corresponding to each frame signal.
[0051] Exemplarily, the target code stream type is a code stream type encapsulated according to HLS, and the first code stream may be HLS.
[0052] According to the first aspect, or any implementation of the first aspect above, the first bitstream further includes an index file, and the attribute information of the initialization tag in the index file includes index information of the sixth bitstream. This ensures that the decoder can decode the second bitstream after confirming that the decoding model reconstruction is complete and reading the decoding model (i.e., initializing the decoder).
[0053] Exemplarily, the target code stream type is a code stream type encapsulated according to HLS, and the first code stream may be HLS.
[0054] According to the first aspect, or any implementation of the first aspect above, the signal to be encoded is M frames, the second codestream is M, the model information includes N information groups, the third codestream includes N substreams, the N information groups correspond one-to-one to the N substreams, the first ES packet includes N ninth ES packets and M−N tenth ES packets, where M and N are positive integers and M is greater than or equal to N; encapsulating the third codestream and the second codestream into the first ES packet includes: encapsulating the N substreams and the N second codestreams into N ninth ES packets, and encapsulating the remaining M−N second codestreams into M−N tenth ES packets; wherein the ninth ES packet includes one substream and one second codestream, and the tenth ES packet includes one second codestream.
[0055] A second code stream is generated by encoding a frame of the signal to be encoded based on the coding model. N can be determined based on the data volume of the model information (or the encoded data of the model information), the network bandwidth, and the encoded data of the audio signal; this ensures the smoothness of the reconstructed signal played by the second electronic device.
[0056] According to the first aspect, or any implementation of the first aspect above, the first code stream further includes an index file, and the index file includes index information corresponding to the model information of the decoding model.
[0057] Exemplarily, the target code stream type is a code stream type encapsulated according to HLS, and the first code stream may be HLS.
[0058] According to the first aspect, or any implementation of the first aspect above, obtaining the first bitstream based on the second bitstream and index information corresponding to model information of the decoding model includes: generating an eighth bitstream based on the index information corresponding to the model information of the decoding model; and encapsulating the second bitstream and the eighth bitstream to obtain the first bitstream. This is similar to the method of encapsulating the second bitstream and the third bitstream to obtain the first bitstream, and will not be further described here.
[0059] According to the first aspect, or any implementation of the first aspect above, the third code stream includes description information of the model information and the model information, or the third code stream includes description information of the model information and encoded data of the model information.
[0060] According to the first aspect, or any implementation of the first aspect above, the model information includes model structure information or model parameters.
[0061] The model structure information may refer to the network structure information of the neural network, and the network structure information may include the connection relationship between network layers. The decoding model may include multiple types of network layers (e.g., convolutional layers, fully connected layers, RNN (Recurrent Neural Network) layers, activation layers, etc.), and each type of network layer may be one or more.
[0062] Exemplarily, model parameters may refer to network parameters of a neural network, and network parameters may refer to weight matrices and bias matrices corresponding to network layers; wherein the weight matrix is composed of weight vectors of multiple network layers, and the bias matrix is composed of bias vectors of multiple network layers. Each network layer may be composed of one or more neurons, and a value in the weight vector of the network layer represents the weight value of a neuron in the network layer, and a value in the bias vector of the network layer represents the bias value of a neuron in the network layer.
[0063] In one possible approach, the model information of the decoding model may be a model file of the decoding model, that is, including connections between all network layers of the decoding model and network parameters of all network layers.
[0064] In one possible manner, the model information of the decoding model may be connections between updated network layers of the decoding model and / or network parameters of updated network layers.
[0065] According to the first aspect, or any implementation of the first aspect above, the preset identification value is further used to characterize the type of the decoding model, so that the second electronic device can easily know whether it needs to obtain model information of the decoding model and perform model reconstruction.
[0066] In a second aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a first code stream, the first code stream including a model identifier and a second code stream, the second code stream being generated based on an encoded signal; when the value of the model identifier is a preset identifier value, obtaining model information; then, reconstructing the model based on the model information to obtain a decoding model; thereafter, decoding the second code stream based on the decoding model to obtain a reconstructed signal.
[0067] The decoding method can be applied to a second electronic device. When the second electronic device determines that the model identifier is a preset identifier value, it obtains model information and reconstructs the model, thereby synchronously updating the decoding model with the encoding model of the first electronic device, or synchronously switching the decoding model used for decoding with the encoding model used by the first electronic device for encoding. This facilitates the long-term evolution of coding standards and optimizes coding performance.
[0068] Exemplarily, the reconstructed signal may include, but is not limited to, a reconstructed audio signal, a reconstructed image, or a reconstructed video signal.
[0069] According to the second aspect, when the value of the model identifier is a preset identifier value, the first code stream also includes a third code stream, and the third code stream is generated based on the model information of the decoding model; obtaining the model information includes: parsing the third code stream from the first code stream; and obtaining the model information of the decoding model based on the third code stream.
[0070] According to the second aspect, or any implementation of the second aspect above, when the value of the model identifier is a preset identifier value, the first codestream also includes index information corresponding to the model information of the decoding model; obtaining the model information includes: parsing the index information corresponding to the model information of the decoding model from the first codestream; and obtaining the model information of the decoding model based on the index information corresponding to the model information of the decoding model.
[0071] According to the second aspect, or any implementation method of the above second aspect, the preset identification value is used to represent the model information of the decoding model stored in the preset storage location of the remote electronic device; obtaining the model information includes: obtaining the model information of the decoding model from the preset storage location in the remote electronic device.
[0072] The remote electronic device may also refer to the third electronic device mentioned above.
[0073] According to the second aspect, or any implementation of the second aspect above, the model identifier is a field of ES; the method further includes: parsing the field corresponding to the model identifier in the first code stream to obtain a value of the model identifier.
[0074] According to the second aspect, or any implementation of the second aspect above, the target codestream type of the first codestream is a codestream type encapsulated according to a target container format or a codestream type encapsulated according to a target transmission protocol, and the model identifier is a field of the target container format; the method further includes: decapsulating the first codestream according to the target container format to obtain a field corresponding to the model identifier; and parsing the field corresponding to the model identifier to obtain a value of the model identifier.
[0075] According to the second aspect, or any implementation of the second aspect above, the target codestream type of the first codestream is a codestream type encapsulated according to the target transmission protocol, and the model identifier is a field of the target transmission protocol; the method further includes: decapsulating the first codestream according to the target transmission protocol to obtain a field corresponding to the model identifier; and parsing the field corresponding to the model identifier to obtain a value of the model identifier.
[0076] According to the second aspect, or any implementation of the second aspect above, the model identifier is used to characterize the type of the decoding model, and the method also includes: when the value of the model identifier is a preset identifier value, determining whether there is a decoding model of the type corresponding to the preset identifier value; reconstructing the model based on the model information to obtain the decoding model, including: when there is no decoding model of the type corresponding to the preset identifier value, reconstructing the model based on the model information to obtain a decoding model of the type corresponding to the preset identifier value.
[0077] According to the second aspect, or any implementation of the second aspect above, parsing the third bitstream from the second bitstream includes: parsing the first bitstream according to the target bitstream type of the first bitstream to obtain the third bitstream.
[0078] According to the second aspect, or any implementation of the second aspect, the target bitstream type is an ES type, the first bitstream further includes a first ES packet, and the first ES packet includes a second bitstream, a third bitstream, and a model identifier; parsing the first bitstream according to the target bitstream type of the first bitstream to obtain the third bitstream includes: reading the third bitstream from the first ES packet; and the method further includes: reading the second bitstream from the first ES packet.
[0079] For example, the present application does not limit the execution order of parsing the second code stream from the first code stream and parsing the value of the model identifier from the first code stream; nor does it limit the execution order of parsing the second code stream from the first code stream and parsing the third code stream from the first code stream.
[0080] According to the second aspect, or any implementation of the second aspect, the target codestream type is a codestream type encapsulated according to a target container format; parsing the first codestream according to the target codestream type of the first codestream to obtain a third codestream includes: decapsulating the first codestream according to the target container format to obtain a second ES packet; reading the third codestream from the second ES packet; and the method further includes: reading the second codestream from the second ES packet.
[0081] According to the second aspect, or any implementation of the second aspect, the target bitstream type is a bitstream type encapsulated according to a target container format; parsing the first bitstream according to the target bitstream type of the first bitstream to obtain a third bitstream includes: decapsulating the first bitstream according to the target container format to obtain a third ES packet, a fourth ES packet, a type of the third ES packet, and a type of the fourth ES packet; reading the third bitstream from the third ES packet, wherein the type of the third ES packet is a preset type; and the method further includes: reading the second bitstream from the fourth ES packet, wherein the type of the fourth ES packet is a media type.
[0082] According to the second aspect, or any implementation of the second aspect above, the preset type is a metadata type.
[0083] According to the second aspect, or any implementation of the second aspect, the target codestream type is a codestream type encapsulated according to a target container format, the first codestream further includes a fourth codestream and a fifth codestream, the fifth codestream includes the second codestream, the fourth codestream includes the third codestream, and the model identifier is located in the fourth codestream or the fifth codestream; parsing the first codestream according to the target codestream type of the first codestream to obtain the third codestream includes: decapsulating the fourth codestream according to the target container format to obtain a fifth ES packet and a type of the fifth ES packet; reading the third codestream from the fifth ES packet, wherein the type of the fifth ES packet is a preset type; and the method further includes: decapsulating the fifth codestream according to the target container format to obtain a sixth ES packet and a type of the sixth ES packet; and reading the second codestream from the sixth ES packet, wherein the type of the sixth ES packet is a media type.
[0084] According to the second aspect, or any implementation of the second aspect, the target codestream type is a codestream type encapsulated according to a target transmission protocol, the first codestream further includes a sixth codestream and a seventh codestream, the seventh codestream includes the second codestream, and the sixth codestream includes the third codestream, and the model identifier is located in the sixth codestream or the seventh codestream; parsing the first codestream according to the target codestream type of the first codestream to obtain the third codestream includes: decapsulating the first codestream according to the target transmission protocol to obtain the sixth codestream; decapsulating the sixth codestream according to the target container format to obtain a seventh ES packet and a type of the seventh ES packet; reading the third codestream from the seventh ES packet, wherein the type of the seventh ES packet is a preset type; and the method further includes: decapsulating the seventh codestream decapsulated from the first codestream according to the target container format to obtain an eighth ES packet and a type of the eighth ES packet; and reading the second codestream from the eighth ES packet, wherein the type of the eighth ES packet is a media type.
[0085] According to the second aspect, or any implementation of the second aspect, the first codestream further includes an index file, the index file includes index information of the sixth codestream and index information of the seventh codestream, and the index information of the sixth codestream is located before the index information of the seventh codestream. Decapsulating the first codestream according to the target transmission protocol to obtain the sixth codestream includes: decapsulating the first codestream according to the target transmission protocol to obtain the index file and the multiple codestreams; reading the index information of the sixth codestream from the index file, and determining the sixth codestream from the multiple codestreams based on the index information of the sixth codestream.
[0086] According to the second aspect, or any implementation of the second aspect, the first codestream further includes an index file, and the attribute information of the initialization tag in the index file includes index information of the sixth codestream. Decapsulating the first codestream according to the target transmission protocol to obtain the sixth codestream includes: decapsulating the first codestream according to the target transmission protocol to obtain the index file and the multiple codestreams; reading the index information of the sixth codestream from the attribute information of the initialization tag in the index file, and determining the sixth codestream from the multiple codestreams based on the index information of the sixth codestream.
[0087] According to the second aspect, or any implementation of the second aspect above, the reconstructed signal is M frames, the number of first ES packets is M, the M first ES packets include N ninth ES packets and M−N tenth ES packets, the ninth ES packet includes a substream and a second stream, the tenth ES packet includes a second stream, M and N are positive integers, and M is greater than or equal to N; reading the third stream from the first ES packet includes: reading N substreams from the N ninth ES packets; splicing the N substreams to obtain the third stream; and reading the second stream from the first ES packet includes: reading N second streams from the N ninth ES packets, and using the M−N tenth ES packets as the M−N second streams.
[0088] According to the second aspect, or any implementation of the second aspect above, the first codestream further includes an index file, the index file including index information corresponding to the model information of the decoding model; parsing the index information corresponding to the model information of the decoding model from the first codestream includes: decapsulating the first codestream according to the target transmission format to obtain the index file; and reading the index information corresponding to the model information of the decoding model from the index file.
[0089] According to the second aspect, or any implementation of the second aspect above, determining the model information based on the third bitstream includes: parsing the model information from the third bitstream; or parsing encoded data of the model information from the third bitstream, and decoding the encoded data of the model information to obtain the model information.
[0090] According to the second aspect, or any implementation of the second aspect, the model information includes model structure information or model parameters.
[0091] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0092] In a third aspect, the present application provides an encoding device applied to a first electronic device, the device comprising:
[0093] A signal acquisition module, used to acquire the signal to be encoded;
[0094] The encoding module is configured to generate a first code stream, wherein the first code stream includes a second code stream and a model identifier, the second code stream is obtained by encoding information to be encoded based on the encoding model, the value of the model identifier is a preset identifier value, and the preset identifier value is used to instruct the second electronic device to obtain model information of a decoding model, where the decoding model corresponds to the encoding model.
[0095] It should be understood that the encoding device can execute the steps of the above-mentioned first aspect and any one of the implementation methods of the first aspect, which will not be repeated here.
[0096] In a fourth aspect, the present application provides a decoding device, comprising:
[0097] A code stream receiving module, configured to receive a first code stream, the first code stream including a model identifier and a second code stream, the second code stream being generated based on the encoded signal;
[0098] The decapsulation module is used to obtain model information when the value of the model identifier is a preset identifier value;
[0099] The decapsulation module is also used to reconstruct the model based on the model information to obtain the decoding model;
[0100] The decoder is configured to decode the second code stream based on the decoding model to obtain a reconstructed signal.
[0101] Exemplarily, the decoding device may be applied to a second electronic device.
[0102] It should be understood that the decoding device can execute the steps of the above-mentioned second aspect and any one of the implementation methods of the second aspect, which will not be repeated here.
[0103] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the encoding method executed by the first electronic device in the first aspect or any possible implementation of the first aspect.
[0104] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0105] In the sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the decoding method executed by the second electronic device in the second aspect or any possible implementation of the second aspect.
[0106] The sixth aspect and any implementation of the sixth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.
[0107] In the seventh aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the steps of the encoding method in the first aspect or any possible implementation of the first aspect are executed.
[0108] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0109] In an eighth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the decoding method in the second aspect or any possible implementation of the second aspect are executed.
[0110] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.
[0111] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.
[0112] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0113] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.
[0114] The tenth aspect and any implementation of the tenth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.
[0115] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.
[0116] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.
[0117] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.
[0118] The twelfth aspect and any implementation of the twelfth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.
[0119] In a thirteenth aspect, an embodiment of the present application provides a code stream generation method, which can be generated according to the encoding method of the first aspect and any implementation method of the first aspect.
[0120] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.
[0121] In a fourteenth aspect, an embodiment of the present application provides a device for storing a code stream, the device comprising: a receiver and at least one storage medium, the receiver being configured to receive the code stream; the at least one storage medium being configured to store the code stream; the code stream being a first code stream generated according to the first aspect and any one of the implementations of the first aspect.
[0122] The fourteenth aspect and any implementation of the fourteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.
[0123] In a fifteenth aspect, an embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the code stream, the code stream being a first code stream generated according to the first aspect and any one of the implementation methods of the first aspect; the transmitter being used to obtain the code stream from the storage medium and send the code stream to an end-side device via a transmission medium.
[0124] The fifteenth aspect and any implementation of the fifteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.
[0125] In a sixteenth aspect, an embodiment of the present application provides a system for distributing a code stream, the system comprising: at least one storage medium for storing at least one code stream, the at least one code stream being a first code stream generated according to the first aspect and any one of the implementation methods of the first aspect, a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.
[0126] The sixteenth aspect and any implementation of the sixteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixteenth aspect and any implementation of the sixteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.
[0127] In a seventeenth aspect, an embodiment of the present application provides a compression system, which includes an encoding end and a decoding end, wherein:
[0128] An encoding end, configured to obtain a signal to be encoded; generate a first code stream, the first code stream including a second code stream and a model identifier, the second code stream being obtained by encoding the signal to be encoded based on the encoding model, the model identifier being a preset identifier value, the preset identifier value being used to instruct the second electronic device to obtain model information of a decoding model, the decoding model corresponding to the encoding model;
[0129] The decoding end is used to receive the first code stream; when the value of the model identifier is a preset identifier value, obtain model information; reconstruct the model based on the model information to obtain a decoding model; and decode the second code stream based on the decoding model to obtain a reconstructed signal.
[0130] The seventeenth aspect and any implementation of the seventeenth aspect correspond to the first aspect and any implementation of the first aspect, respectively, and correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the seventeenth aspect and any implementation of the seventeenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0131] FIG1A is a schematic diagram showing an exemplary system framework;
[0132] FIG1B is a schematic diagram of an exemplary system framework;
[0133] FIG1C is a schematic diagram showing the structure of an exemplary encoding and decoding module;
[0134] FIG2 is a schematic diagram illustrating an exemplary encoding process;
[0135] FIG3 is a schematic diagram illustrating an exemplary decoding process;
[0136] FIG4 is a schematic diagram illustrating an exemplary audio encoding process;
[0137] FIG5 is a schematic diagram illustrating an exemplary audio decoding process;
[0138] FIG6A is a schematic diagram illustrating an exemplary audio encoding process;
[0139] FIG6B is a schematic diagram illustrating an exemplary model information code stream;
[0140] FIG6C is a schematic diagram showing the structure of a first code stream;
[0141] FIG7 is a schematic diagram illustrating an exemplary audio decoding process;
[0142] FIG8 is a schematic diagram illustrating an exemplary audio encoding process;
[0143] FIG9 is a schematic diagram illustrating an exemplary audio decoding process;
[0144] FIG10A is a schematic diagram illustrating an exemplary audio encoding process;
[0145] FIG10B is a schematic diagram showing the structure of a first code stream;
[0146] FIG11 is a schematic diagram illustrating an exemplary audio decoding process;
[0147] FIG12A is a schematic diagram illustrating an exemplary audio encoding process;
[0148] FIG12B is a schematic diagram illustrating an exemplary metadata encapsulation process;
[0149] FIG13 is a schematic diagram illustrating an exemplary audio decoding process;
[0150] FIG14A is a schematic diagram illustrating an exemplary audio encoding process;
[0151] FIG14B is a schematic diagram showing the structure of a first code stream;
[0152] FIG15 is a schematic diagram illustrating an exemplary audio decoding process;
[0153] FIG16A is a schematic diagram illustrating an exemplary audio encoding process;
[0154] FIG16B is a schematic diagram showing the structure of a first code stream;
[0155] FIG17 is a schematic diagram illustrating an exemplary audio decoding process;
[0156] FIG18 is a schematic diagram illustrating an exemplary encoding device;
[0157] FIG19 is a schematic diagram illustrating an exemplary decoding device;
[0158] FIG20 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION
[0159] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0160] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0161] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.
[0162] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.
[0163] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.
[0164] Figure 1A is a schematic diagram of an exemplary system framework. Figure 1A shows a coding and decoding framework for media signals, wherein the media signals may include but are not limited to: audio signals, video signals, and images.
[0165] 1A , illustratively, the first electronic device may include a first acquisition module, a first encoding module, a first channel encoding module, a first channel decoding module, a first decoding module, and a first playback module. It should be understood that the first electronic device may include more or fewer modules than those shown in FIG1A , and this application does not limit this.
[0166] 1A , illustratively, the second electronic device may include a second acquisition module, a second encoding module, a second channel encoding module, a second channel decoding module, a second decoding module, and a second playback module. It should be understood that the second electronic device may include more or fewer modules than those shown in FIG1A , and this application does not limit this.
[0167] Exemplarily, the process of a first electronic device encoding and transmitting a media signal to a second electronic device, and the decoding and playback by the second electronic device can be as follows: the first acquisition module can perform acquisition and output the media signal to the first encoding module. Then, the first encoding module can encode the media signal and output the code stream to the first channel encoding module. Afterwards, the first channel encoding module can channel encode the code stream and transmit the channel-encoded code stream to the second electronic device through a wireless or wired network communication device. Then, the second channel decoding module of the second electronic device can channel decode the received data to obtain a code stream and output the code stream to the second decoding module. Then, the second decoding module can decode the code stream to obtain a reconstructed signal; then the reconstructed signal is output to the second playback module, and the second playback module plays it.
[0168] It should be understood that the process of the second electronic device encoding and transmitting the media signal to the first electronic device, which is decoded and played by the first electronic device, is similar to the process of the first electronic device transmitting the media signal to the second electronic device, which is played by the second electronic device, and will not be repeated here.
[0169] It should be understood that the first electronic device and the second electronic device can be directly connected without communicating through a wireless or wired network communication device, and this application does not impose any restrictions on this.
[0170] Exemplarily, the first electronic device and the second electronic device may include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.
[0171] For example, the codec framework of Figure 1A can be applied to any scenario requiring codecs, such as VR (Virtual Reality) / AR (Augmented Reality) scenarios. In VR and AR scenarios: In one possible approach, the first electronic device is a server and the second electronic device is a VR / AR device; in another possible approach, the second electronic device is a server and the first electronic device is a VR / AR device.
[0172] For example, the first electronic device may be referred to as an encoding end, and the second electronic device may be referred to as a decoding end; or the second electronic device may be referred to as an encoding end, and the first electronic device may be referred to as a decoding end.
[0173] Figure 1B is a schematic diagram of an exemplary system framework, which shows a transcoding framework of a media signal.
[0174] 1B (1), illustratively, a wireless or core network device may include: a channel decoding module, other decoding modules, an encoding module (ie, the encoding module involved in this application), and a channel encoding module. The wireless or core network device may be used for transcoding.
[0175] Exemplarily, the specific application scenario of Figure 1B(1) may be: when the first electronic device is not provided with an encoding module but only with other encoding modules; and the second electronic device is only provided with a decoding module but not with other decoding modules, in order to enable the second electronic device to decode and play the media signal encoded by the first electronic device using other encoding modules, wireless or core network equipment may be used for transcoding.
[0176] Specifically, the first electronic device uses another encoding module to encode the media signal to obtain code stream 1; it then performs channel encoding on code stream 1 and sends it to a wireless or core network device. The channel decoding module of the wireless or core network device can then perform channel decoding and output the decoded code stream 1 to another decoding module. The other decoding module then decodes code stream 1 to obtain a media signal and outputs the media signal to the encoding module. The encoding module can then encode the media signal to obtain code stream 2 and output it to the channel encoding module. The channel encoding module then performs channel encoding on code stream 2 and sends it to the second electronic device. The second electronic device can then call the decoding module to decode code stream 2 obtained by channel decoding to obtain a reconstructed signal; the reconstructed signal can then be played.
[0177] 1B (2), illustratively, a wireless or core network device may include: a channel decoding module, a decoding module (ie, an encoding module involved in this application), other encoding modules, and a channel encoding module. The wireless or core network device may be used for transcoding.
[0178] Exemplarily, the specific application scenario of Figure 1B(2) may be: when the first electronic device is only provided with an encoding module and no other encoding modules; and the second electronic device is not provided with a decoding module and only provided with other decoding modules, in order to enable the second electronic device to decode and play the media signal encoded by the encoding module of the first electronic device, wireless or core network equipment may be used for transcoding.
[0179] Specifically, the first electronic device uses an encoding module to encode the media signal to obtain code stream 1; it then performs channel encoding on code stream 1 and sends it to a wireless or core network device. The channel decoding module of the wireless or core network device then performs channel decoding and outputs the decoded code stream 1 to the decoding module. The decoding module then decodes code stream 1 to obtain a media signal and outputs the media signal to another encoding module. The other encoding module then encodes the media signal to obtain code stream 2 and outputs it to the channel encoding module. The channel encoding module then performs channel encoding on code stream 2 and sends it to a second electronic device. The second electronic device then calls another decoding module to decode code stream 2 obtained through channel decoding to obtain a reconstructed signal; the reconstructed signal can then be played.
[0180] Figure 1C is a schematic diagram illustrating the structure of an exemplary encoding and decoding module. In Figure 1C, the encoding module may refer to the first encoding module and the second encoding module in Figure 1A, as well as the encoding module in Figure 1B. In Figure 1C, the decoding module may refer to the first decoding module and the second decoding module in Figure 1A, as well as the decoding module in Figure 1B.
[0181] Referring to FIG1C(1), the encoding module may include an encoder and an encapsulation module, wherein the encoder is used to encode the media signal based on the encoding model and output a code stream, and the encapsulation model is used to encapsulate the code stream. It should be understood that the encoding module in FIG1C(1) is only an example of the present application, and the encoding module of the present application may include more modules than shown in FIG1C(1), and the present application does not limit this.
[0182] Referring to FIG1C(2), the decoding module may include, for example, a decoder and a decapsulation module; wherein the decapsulation module is used to decapsulate and parse the received code stream; and the decoder may be used to decode the code stream based on the decoding model to obtain a reconstructed signal. It should be understood that the decoding module in FIG1C(2) is only an example of the present application, and the decoding module of the present application may include more modules than those shown in FIG1C(2), and the present application does not limit this.
[0183] Exemplarily, both the encoding model and the decoding model can be implemented by a neural network; therefore, the encoding model can also be called an AI (Artificial Intelligence) encoding model, and the decoding model can also be called an AI decoding model.
[0184] In one possible approach, the encoding model and the decoding model are two independent models; in this case, the encoding model and the decoding model can be trained jointly. In another possible approach, the encoding model and the decoding model are two parts of a single model.
[0185] The following describes the encoding and decoding process of the media signal based on FIG. 1C .
[0186] Figure 2 is a schematic diagram illustrating an exemplary encoding process, wherein the encoding step in the embodiment of Figure 2 can be performed by the first electronic device.
[0187] S201: Acquire a signal to be encoded.
[0188] Exemplarily, the signal to be encoded may be a media signal, wherein the media signal includes but is not limited to an audio signal, an image, and a video signal. The signal to be encoded may be M frames, where M may be a positive integer; when the signal to be encoded is an image, M is equal to 1.
[0189] S202: Generate a first code stream, where the first code stream includes a second code stream and a model identifier. The second code stream is obtained by encoding the signal to be encoded based on the encoding model. The value of the model identifier is a preset identifier value, which is used to instruct the second electronic device to obtain model information of a decoding model, where the decoding model corresponds to the encoding model.
[0190] For example, an encoder in the encoding module may encode the signal to be encoded based on the encoding model to obtain a second bitstream. It should be noted that the second bitstream refers to the bitstream output by the encoder (which may also be referred to as a raw stream in some cases); the second bitstream may include encoded data of the signal to be encoded (or encoded data of the media signal).
[0191] Exemplarily, the encoder may perform preprocessing, such as transformation, on the signal to be encoded to obtain first data; thereafter, the encoder may perform quantization and entropy encoding on the first data to output a second code stream.
[0192] Then, based on the second code stream, the first code stream is generated. Exemplarily, the first code stream can be generated based on the second code stream by the encapsulation module in the encoding module.
[0193] In one possible scenario, if the first electronic device only stores one coding model and one decoding model; when the coding model and the decoding model in the first electronic device are updated, the coding module in the first electronic device can set the value of the model identifier in the first code stream to a preset identification value, and the preset identification value can be used to instruct (or guide) the second electronic device to obtain the model information of the decoding model. In this way, it is possible to ensure that the decoding model of the second electronic device is updated synchronously. (It should be understood that in one possible scenario, the coding model may not be updated, and only the decoding model may be updated (such as optimizing the decoding model); in this case, the updated decoding model corresponds to the coding model. In this case, when the decoding model in the first electronic device is updated, the coding module in the first electronic device can set the value of the model identifier in the first code stream to the preset identification value. This application is explained by taking the updating of both the coding module and the decoding model as an example.)
[0194] In one possible scenario, if a first electronic device stores multiple coding models and multiple decoding models (coding models correspond one to one to decoding models), in the process of encoding the signal to be encoded, the encoder of the first electronic device can switch the coding model used for encoding. After the encoder of the first electronic device switches the coding model used for encoding (for example, switching from coding model 1 to coding model 2), the encoding module in the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value, and the preset identifier value can be used to instruct the second electronic device to obtain the model information of the decoding model. In this way, it can be ensured that the decoder of the second electronic device can switch to the decoding model corresponding to the switched coding model (for example, switching from decoding model 1 to decoding model 2) for decoding.
[0195] In one possible scenario, regardless of whether the encoding model and decoding model in the first electronic device are updated, and regardless of whether the encoder in the first electronic device switches the encoding model, the encoding module in the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value. In this case, the preset identifier value can be used to represent the type of the decoding model (or encoding model); correspondingly, after obtaining the preset identifier value, the second electronic device determines whether the second electronic device has a decoding model of the type corresponding to the preset identifier value; if no decoding model of the type corresponding to the preset identifier value exists, the model information of the decoding model is obtained.
[0196] It should be understood that when the encoding model and the decoding model are two parts of a single model, the preset identification value can be used to instruct the second electronic device to obtain model information of the encoding model and model information of the decoding model; thus, the second electronic device can reconstruct the encoding model based on the model information of the encoding model and reconstruct the decoding model based on the model information of the decoding model. This application is described using the example of the encoding model and the decoding model being two independent models.
[0197] In one possible approach, a new identifier (i.e., a flag) or a new field can be added to the first bitstream as a model identifier. In another possible approach, an existing field in the first bitstream can be reused and used as a model identifier, which is not limited in this application. When reusing an existing field in the first bitstream and using the existing field as a model identifier, this application does not need to change the decoding syntax, that is, it can implement model updates or switching while achieving backward compatibility.
[0198] Exemplarily, the first electronic device may include a device for transmitting a code stream; the device for transmitting a code stream may include: a transmitter and at least one storage medium, the at least one storage medium being used to store the first code stream generated in the embodiment of FIG. 2 above; the transmitter being used to obtain the first code stream from the storage medium and send the first code stream to the end-side device via the transmission medium.
[0199] Exemplarily, after generating a first bitstream, the first electronic device may send the first bitstream to a system for distributing bitstreams. The system for distributing bitstreams may include: at least one storage medium and a streaming device; the storage medium is used to store the at least one first bitstream generated in the embodiment of FIG. 2; and the streaming device is used to obtain a target bitstream from the at least one storage medium and send the target bitstream to the client-side device, wherein the streaming device includes a content server or a content distribution server.
[0200] Figure 3 is a schematic diagram illustrating an exemplary decoding process. Figure 3 illustrates the decoding process corresponding to the encoding process in Figure 2 . The decoding steps in the embodiment of Figure 3 can be performed by a second electronic device. Exemplarily, the second electronic device can include a device for storing a code stream. The device for storing a code stream can include: a receiver for receiving a first code stream; and at least one storage medium for storing the first code stream. The first code stream can then be decoded as described in S302 to S304.
[0201] S301: Receive a first code stream.
[0202] Exemplarily, after the first electronic device sends the first code stream, the second electronic device may receive the first code stream.
[0203] Exemplarily, the decapsulation module in the decoding module of the second electronic device can parse the value of the model identifier from the first code stream; then, based on the parsed value of the model identifier, it can be determined whether the first electronic device updates the encoding module and decoding model, or whether the encoder in the first electronic device switches the encoding model used for encoding.
[0204] S302: When the value of the model identifier is a preset identifier value, obtain model information of the decoding model.
[0205] Exemplarily, when the value of the model identifier is a preset identifier value, it indicates that the first electronic device has updated the encoding module and encoding model, or that the encoder of the first electronic device has switched the encoding model used for encoding. At this time, the second electronic device can obtain model information of the decoding model.
[0206] For example, when the model identifier value is another identifier value, it indicates that the first electronic device has not updated the encoding module and encoding model, or that the encoder of the first electronic device has not switched the encoding model used for encoding; in this case, the second electronic device does not need to obtain the model information of the decoding model. In this case, the decapsulation module in the decoding model of the second electronic device can parse the second bitstream from the first bitstream, and then the decoder can decode the second bitstream based on the decoding model type corresponding to the other identifier value (or the default decoding model) to obtain a reconstructed signal.
[0207] Exemplarily, if the first electronic device sets the value of the model identifier to a preset identifier value regardless of whether the encoding model and decoding model of the first electronic device are updated, and regardless of whether the encoder of the first electronic device switches the encoding model used for encoding; then when the second electronic device parses the value of the model identifier as the preset identifier value, it can be determined whether the second electronic device has a decoding model of the type corresponding to the preset identifier value. When the second electronic device does not have a decoding model of the type corresponding to the preset identifier value, S303 to S304 are executed. When the second electronic device has a decoding model of the type corresponding to the preset identifier value, there is no need to execute S303. After executing S304, the decoding model of the type corresponding to the preset identifier value can be directly used to decode the second code stream.
[0208] S303: Reconstruct the model based on the model information to obtain a decoding model.
[0209] Exemplarily, the decapsulation module may reconstruct the model based on the model information to obtain a decoding model.
[0210] S304: Decode the second code stream based on the decoding model to obtain a reconstructed signal.
[0211] Exemplarily, the decapsulation module may further parse the first bitstream to derive the second bitstream and send the second bitstream to the decoder. It should be understood that this application does not limit the order of parsing the second bitstream from the first bitstream and obtaining the model information or parsing the model identifier value from the first bitstream.
[0212] Next, the decoder can read the reconstructed decoding model (also called decoder initialization), and then decode the second bitstream based on the reconstructed decoding model to obtain a reconstructed signal. Afterwards, the decoder can play the reconstructed signal and perform other processing.
[0213] For example, the decoder may perform entropy decoding and inverse quantization on the second bitstream based on the reconstructed decoding model to obtain second data. Thereafter, the decoder may perform post-processing, such as inverse transformation, on the second data to obtain a reconstructed signal. The reconstructed signal may include, but is not limited to, a reconstructed audio signal, a reconstructed image, or a reconstructed video signal.
[0214] In one possible approach, when the first electronic device has not updated the encoding model and the decoding model, or has not switched the encoding model, the first code stream may not include a model identifier; further, when the second electronic device has not parsed the model identifier from the first code stream, it may use a default decoding model to decode the second code stream parsed from the first code stream to obtain a reconstructed signal.
[0215] In this way, the first electronic device instructs the second electronic device to obtain the model information of the decoding model by setting the value of the model identifier in the first code stream to a preset identifier value; after the first code stream is transmitted to the second electronic device, the second electronic device can obtain the model information of the decoding model according to the preset identifier value, thereby realizing the synchronous update of the decoding model or the synchronous switching of the decoding model used for decoding and the encoding model used for encoding.
[0216] In one possible approach, a first electronic device can transmit model information of a decoding model to a second electronic device. Specifically, a third code stream can be generated based on the model information of the decoding model. The second and third code streams can then be encapsulated to obtain a first code stream. In this case, a preset identification value can be used to indicate that the first code stream includes the third code stream. Thus, the preset identification value can be used to instruct the second electronic device to obtain the model information of the decoding model from the first code stream.
[0217] The following description of the present application is made by taking the case where the signal to be encoded is an audio signal as an example.
[0218] In the embodiments of FIG. 4 to FIG. 17 , the signal to be encoded is an audio signal, the second code stream is an audio code stream, and the third code stream is a model information code stream.
[0219] FIG4 is a schematic diagram illustrating an exemplary audio encoding process.
[0220] S401: Acquire an audio signal.
[0221] Exemplarily, an audio signal to be encoded may be obtained; wherein the audio signal may be M frames, and M may be a positive integer.
[0222] S402: Encode the audio signal based on the coding model to obtain an audio code stream.
[0223] Exemplarily, an audio encoder may encode the audio signal based on the coding model to obtain an audio bitstream. It should be noted that the audio bitstream refers to the bitstream output by the audio encoder; the audio bitstream may include coded data of the audio signal.
[0224] Exemplarily, an audio encoder may perform audio preprocessing, such as transformation, on the audio signal to obtain first data; thereafter, the first data may be processed such as quantization and entropy coding to output an audio code stream.
[0225] S403: Generate a model information code stream based on the model information of the decoding model; wherein the decoding model corresponds to the encoding model.
[0226] S404, encapsulating the audio code stream and the model information code stream to obtain a first code stream; wherein the first code stream includes a model identifier, the value of the model identifier is a preset identifier value, and the preset identifier value is used to indicate that the first code stream includes the model information code stream.
[0227] In one possible scenario, a first electronic device stores only one coding model and one decoding model. After both the coding model and the decoding model in the first electronic device are updated, to ensure that the decoding model in the second electronic device is synchronously updated, after the audio encoder encodes the audio signal for the first time based on the updated coding model to obtain an audio code stream, the first electronic device can generate a model information code stream based on the model information of the updated decoding model. The audio code stream and the model information code stream are then encapsulated to obtain a first code stream. The value of the model identifier in the first code stream is set to a preset identifier value (the preset identifier value is used to indicate that the first code stream contains the model information code stream). The first code stream can then be sent to the second electronic device. In this way, the model information can be transmitted to the second electronic device, and the second electronic device can reconstruct the decoding model based on the model information, thereby updating the decoding model in the second electronic device.
[0228] In one possible scenario, a first electronic device stores multiple coding models and multiple decoding models (with a one-to-one correspondence between coding models and decoding models). During the process of encoding an audio signal, the first electronic device can switch the coding model used for encoding. After the first electronic device switches the coding model used for encoding (for example, from coding model 1 to coding model 2), to ensure that the second electronic device can switch to the decoding model corresponding to the switched coding model (for example, from decoding model 1 to decoding model 2) for decoding, after the audio encoder encodes the audio signal based on the switched coding model (such as coding model 2), the first electronic device can generate a model information code stream based on the model information of the decoding model corresponding to the switched coding model (such as decoding model 2). The audio code stream and the model information code stream are then encapsulated to obtain a first code stream; the value of the model identifier in the first code stream is set to a preset identifier value; and the first code stream can be sent to the second electronic device. In this way, the model information can be transmitted to the second electronic device, which can then reconstruct the decoding model corresponding to the switched coding model (such as decoding model 2) based on the model information.
[0229] In one possible scenario, regardless of whether the encoding model and decoding model of the first electronic device are updated, and regardless of whether the first electronic device switches the encoding model used for encoding, the first electronic device can generate a model information code stream based on the model information of the decoding model corresponding to the encoding model used by the audio encoder to encode the audio signal; then, encapsulate the audio code stream and the model information code stream to obtain a first code stream; and set the value of the model identifier in the first code stream to a preset identifier value; thereafter, the first code stream can be sent to the second electronic device. In this case, the preset identifier value can also be used to characterize the type of the encoding model (or decoding model); further, when the second electronic device determines that there is no decoding model of the type corresponding to the preset identifier value locally, it reconstructs the decoding model based on the model information.
[0230] It should be understood that the first electronic device can also generate a model information code stream of the coding model based on the model information of the updated coding model (or the switched coding model), encapsulate it into the first code stream, and send it to the second electronic device. In this way, the second electronic device can reconstruct the coding model based on the model information code stream of the coding model; when the second electronic device subsequently encodes the audio signal, it can encode the audio signal based on the updated coding model (or the switched coding model).
[0231] It should be understood that when the encoding model and the decoding model are two parts of a model, the first electronic device can encapsulate the model information code stream of the encoding model, the model information code stream of the decoding model, and the audio code stream into a first code stream and send it to the second electronic device. The second electronic device can reconstruct the encoding model based on the model information code stream of the encoding model, and reconstruct the decoding model based on the model information code stream of the decoding model. This application is described as an example in which the encoding model and the decoding model are two independent models.
[0232] In one possible approach, an existing field in the first codestream can be used as the model identifier. When the first electronic device has not updated the encoding model and decoding model, or has not switched the encoding model used for encoding, that is, when the first codestream does not contain a model information codestream, the model identifier can be set to another identifier value; in this case, the other identifier value is used to indicate that the first codestream does not contain a model information codestream.
[0233] In addition, when the preset identification value is also used to characterize the type of decoding model (or the type of encoding model), other identification values can also be used to characterize the type of decoding model (or the type of encoding model); wherein the type of decoding model (or encoding model) characterized by the other identification values is different from the type of decoding model (or encoding model) characterized by the preset identification value.
[0234] In one possible approach, an additional field may be added to the first bitstream as a model identifier. When the first electronic device has not updated the encoding model and decoding model, or has not switched the encoding model used for encoding, that is, when the first bitstream does not include a model information bitstream, there is no need to add an additional field.
[0235] Exemplarily, the model information of the decoding model may include model structure information and / or model parameters.
[0236] The model structure information may refer to the network structure information of the neural network, and the network structure information may include the connection relationship between network layers. The decoding model may include multiple types of network layers (e.g., convolutional layers, fully connected layers, RNN (Recurrent Neural Network) layers, activation layers, etc.), and each type of network layer may be one or more.
[0237] Exemplarily, model parameters may refer to network parameters of a neural network, and network parameters may refer to weight matrices and bias matrices corresponding to network layers; wherein the weight matrix is composed of weight vectors of multiple network layers, and the bias matrix is composed of bias vectors of multiple network layers. Each network layer may be composed of one or more neurons, and a value in the weight vector of the network layer represents the weight value of a neuron in the network layer, and a value in the bias vector of the network layer represents the bias value of a neuron in the network layer.
[0238] In one possible approach, the model information of the decoding model may be a model file of the decoding model, that is, including connections between all network layers of the decoding model and network parameters of all network layers.
[0239] In one possible manner, the model information of the decoding model may be connections between updated network layers of the decoding model and / or network parameters of updated network layers.
[0240] It should be noted that S404 can be executed by the encapsulation module, and S403 can be executed by other modules in the encoding module except the audio encoder and the encapsulation module.
[0241] FIG5 is a schematic diagram illustrating an exemplary audio decoding process. FIG5 is a decoding process corresponding to the audio encoding process of FIG4.
[0242] S501: Receive a first code stream.
[0243] Exemplarily, after the first electronic device sends the first code stream, the second electronic device may receive the first code stream.
[0244] S502: Parse the first code stream to obtain a value of the model identifier.
[0245] Exemplarily, the decapsulation module of the second electronic device can parse the value of the model identifier from the first code stream; then, based on the parsed value of the model identifier, it can be determined whether the first electronic device updates the encoding model and decoding model, or whether the first electronic device switches the encoding model used for encoding.
[0246] S503: When the value of the model identifier is a preset identifier value, parse the first code stream to obtain a model information code stream.
[0247] Exemplarily, when the model identifier value is a preset identifier value, it indicates that the first electronic device has updated the encoding model and decoding model, or that the first electronic device has switched the encoding model, which means that the first code stream also includes a model information code stream. Furthermore, the decapsulation module of the second electronic device can parse the model information code stream from the first code stream.
[0248] For example, when the model identifier value is another identifier value, it indicates that the first electronic device has not updated the encoding model and decoding model, or that the first electronic device has not switched the encoding model, that is, the first bitstream does not contain a model information bitstream. In this case, the second electronic device can parse the audio bitstream from the first bitstream and then decode the audio bitstream based on the decoding model type corresponding to the other identifier value (or the default decoding model) to obtain a reconstructed audio signal.
[0249] Exemplarily, when the first electronic device sends the model information code stream of the decoding model corresponding to the coding model used by the audio encoder to encode the audio signal, regardless of whether the coding model and decoding model of the first electronic device are updated, and regardless of whether the first electronic device switches the coding model used for encoding, to the second electronic device, if the value of the model identifier is a preset identifier value, it can be determined whether there is a decoding model of the type corresponding to the preset identifier value in the second electronic device. When there is no decoding model of the type corresponding to the preset identifier value in the second electronic device, S504 to S506 are executed. When the second electronic device has a decoding model of the type corresponding to the preset identifier value, there is no need to execute S504. After executing S505, the decoding model of the type corresponding to the preset identifier value can be directly used to decode the audio code stream.
[0250] S504: Reconstruct the model based on the model information to obtain a decoding model.
[0251] Exemplarily, the decapsulation module may determine the model information based on the model information code stream, and then reconstruct the model based on the model information to obtain the decoding model. Specifically, the model may be reconstructed based on the model structure information and / or model parameters included in the model information to obtain the decoding model.
[0252] S505: parse the first code stream to obtain an audio code stream.
[0253] Exemplarily, the decapsulation module may further parse the first code stream to obtain an audio code stream, and send the audio code stream to the audio decoder. It should be understood that the present application does not limit the execution order of any of steps S505 and S502 to S504.
[0254] S506: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0255] Next, the audio decoder can read the reconstructed decoding model (also called audio decoder initialization), and then decode the audio code stream based on the reconstructed decoding model to obtain a reconstructed audio signal. Afterwards, the second electronic device can play the reconstructed audio signal.
[0256] Exemplarily, the audio decoder can perform entropy decoding and inverse quantization on the audio code stream based on the reconstructed decoding model to obtain second data; thereafter, the second data can be subjected to audio post-processing such as inverse transformation to obtain a reconstructed audio signal.
[0257] In one possible embodiment, when the first electronic device has not updated the coding model and decoding module, or has not switched the coding model used for encoding, the first code stream does not include a model identifier and does not include a model information code stream; further, when the second electronic device has not parsed the model identifier from the first code stream, it can use a default decoding model to decode the audio code stream parsed from the first code stream to obtain a reconstructed audio signal.
[0258] For example, the first electronic device may encapsulate the audio bitstream and the model information bitstream according to the target bitstream type (i.e., the type of the desired first bitstream) to obtain the first bitstream. Correspondingly, the second electronic device may also parse the first bitstream according to the target bitstream type to obtain the model information bitstream, the audio bitstream, and the value of the model identifier.
[0259] Exemplarily, the target stream type includes any one of the following: a first type, a second type, or a third type. The first type is the ES type; the second type is the stream type encapsulated according to the target container format (e.g., TS type, MP4 type, etc.); and the third type is the stream type encapsulated according to the target transport protocol (e.g., HLS type, RTP type, etc.).
[0260] Figure 6A is a schematic diagram illustrating an exemplary audio encoding process. In the embodiment of Figure 6A , the target bitstream type is the first type; correspondingly, the first bitstream can also be referred to as an ES stream. The model information bitstream and the audio bitstream can be encapsulated in the same ES packet to obtain the first bitstream.
[0261] S601: Acquire an audio signal.
[0262] Exemplarily, M frames of audio signals may be acquired, where M is a positive integer.
[0263] S602: Determine whether both the encoding model and the decoding model are updated.
[0264] S603: When both the encoding model and the decoding model are updated, encode the audio signal based on the updated encoding model to obtain an audio code stream.
[0265] Exemplarily, the audio encoder may encode M frames of audio signals based on the updated coding model to obtain M audio code streams.
[0266] S604: Generate a model information code stream based on the updated model information of the decoding model; wherein the updated decoding model corresponds to the updated encoding model.
[0267] Exemplarily, model information of a decoding model corresponding to an updated encoding model (i.e., an updated decoding model) may be obtained; the model information of the updated decoding model may be divided into N (N is a positive integer) information groups; then, based on each of the N information groups, a sub-codestream may be generated; and thus, N sub-codestreams may be obtained, i.e., the model information codestream may include multiple sub-codestreams.
[0268] Exemplarily, when N is equal to 1, that is, the model information is not divided or the model information is divided into an information group; in this case, the model information code stream may include the description information of the model information and the model information; or, the model information code stream may include the description information of the model information and the encoded data of the model information.
[0269] Figure 6B is a schematic diagram of an exemplary model information code stream. In Figure 6B (1), N=1, the description information of the model information is the length information of the model information, and the model information code stream may include the length information of the model information and the model information.
[0270] Exemplarily, when N is greater than 1, each sub-stream may include description information of an information group and an information group; or, each sub-stream may include description information of an information group and coded data of the information group.
[0271] Referring to FIG6B(2), illustratively, in FIG6B(2), N is greater than 1, and the description information of each information group may include but is not limited to: the total number N of information groups, the index of the current information group, the length information of the current information group, etc. The sub-code stream may include but is not limited to: the total number N of information groups, the index of the current information group, the length information of the current information group, the information group, etc.
[0272] It should be noted that the model information can be losslessly encoded to obtain the encoded data of the model information; and the information group can be losslessly encoded to obtain the encoded data of the information group; in this way, it can be ensured that the decoding model reconstructed by the second electronic device is consistent with the decoding model of the first electronic device.
[0273] S605: Encapsulate the model information code stream and the audio code stream into a first ES packet, and use the first ES packet as the first code stream.
[0274] Exemplarily, the model information code stream and the audio code stream may be encapsulated in the same ES packet, thereby obtaining a first ES packet; and then the first ES packet is used as the first code stream. In this case, the first code stream may also be referred to as an ES stream.
[0275] Exemplarily, there may be M first ES packets, and the M first ES packets may include N ninth ES packets and MN tenth ES packets, where M and N are positive integers, and M is greater than or equal to N.
[0276] Specifically, N substreams and N audio streams can be encapsulated into N ninth ES packets, and the remaining MN audio streams can be encapsulated into MN tenth ES packets. Furthermore, a substream and an audio stream can be encapsulated into a ninth ES packet, that is, the ninth ES packet can include a substream and an audio stream. Furthermore, one of the remaining MN audio streams can be encapsulated into a tenth ES packet, that is, the tenth ES packet includes an audio stream. It should be noted that an audio stream can be obtained by encoding a frame of audio signal based on a coding model.
[0277] FIG6C is a schematic diagram showing the structure of a first code stream.
[0278] Referring to Figure 6C(1), illustratively, when N is 1, the model information code stream and an audio code stream can be encapsulated in a single ES packet, resulting in a ninth ES packet. Specifically, the model information code stream and the first audio code stream are encapsulated in a single ES packet, resulting in a ninth ES packet; and the remaining M-1 audio code streams are encapsulated as M-1 tenth ES packets. Exemplarily, in the ninth ES packet, the model information code stream can be located after the first audio code stream.
[0279] Referring to FIG6C(2), illustratively, when N is greater than 1, a substream and an audio stream can be encapsulated in one ES packet, resulting in a ninth ES packet. Specifically, N substreams and N audio streams are encapsulated into N ninth ES packets; and the remaining MN audio streams are encapsulated into MN tenth ES packets. Exemplarily, in each ninth ES packet, the substream can be located after the audio stream.
[0280] It should be noted that N can be determined based on the data volume of the model information (or the encoded data of the model information), the network bandwidth and the encoded data of the audio signal to ensure the smoothness of the reconstructed audio signal played by the second electronic device.
[0281] For example, in the embodiment of FIG6A , the model identifier may be a field of an ES. For example, an ES encapsulation format may include multiple fields, and an ES field may refer to a field included in the ES encapsulation format. In other words, after data is encapsulated according to the ES encapsulation format, an ES package may be obtained. The ES package may include multiple fields, and an ES field may refer to a field in the ES package.
[0282] Exemplarily, the audio code stream output by the audio encoder may include a header (ie, a header of an ES packet) and encoded data of the audio data (also referred to as a payload of the audio data).
[0283] Exemplarily, the header of the audio code stream may include a field for indicating the type of decoding model (or the type of encoding model). When there is an unused identification value in this field, the audio encoder may use this field as a model identifier for indicating whether the first code stream contains model information code stream, and select a preset identification value from the unused identification values corresponding to the field value.
[0284] For example, in the Audio Vivid standard, the field used to indicate the type of decoding model (or encoding model) is the nn_type field; the nn_type field can be represented by 3 bits, that is, the nn_type field corresponds to 8 identification values: "0", "1", "2", "3", "4", "5", "6", and "7". Currently, "0" and "1" are used identification values, and "2", "3", "4", "5", "6", and "7" are reserved identification values, that is, unused identification values. The definition of the nn_type field can be shown in Table 1 below:
[0285] Table 1
[0286] Furthermore, the nn_type field can be used as a model identifier. For example, "2" can be used as the preset identifier value. In this case, the nn_type field definition is as shown in Table 2:
[0287] Table 2
[0288] For example, different identification values among the unused identification values corresponding to the nn_type field may be used to represent different types of coding models (or decoding models), that is, different preset identification values may be used to represent different types of coding models (or decoding models); as shown in Table 3 below:
[0289] Table 3
[0290] Referring to Table 3, the identification value "2" may be used to represent the custom model 1, the identification value "3" may be used to represent the custom model 3, and the identification value "4" may be used to represent the custom model 3.
[0291] For example, in other standards, the audio encoder may add an additional field to the header of the audio code stream, and use the additional field as a model identifier for indicating whether the first code stream includes a model information code stream; and set the field value of the additional field to a preset identifier value.
[0292] For example, only the model identifier in the first ninth ES packet (i.e., the ninth ES packet containing the first audio code stream) can be set to the preset identifier value, or the model identifier in some or all of the N ninth ES packets and MN tenth ES packets can be set to the preset identifier value. This application does not impose any restrictions on this.
[0293] This allows direct transmission of the ES packets after they are obtained, without further encapsulation according to the target container format or transport protocol. This reduces bitrate overhead and bandwidth requirements. Furthermore, by encapsulating the second and third streams in the same ES packet, they can be sent over the same channel, reducing transmission complexity.
[0294] Figure 7 is a schematic diagram illustrating an exemplary audio decoding process. Figure 7 is an audio decoding process corresponding to the audio encoding process of Figure 6A.
[0295] S701: Receive a first code stream.
[0296] For example, when the first electronic device encodes the audio signal using the audio encoding method in the embodiment of FIG. 6A , the first bitstream received by the second electronic device is essentially an ES stream. In this case, the first bitstream may include a first ES packet. Assuming that the reconstructed audio signal consists of M frames, there may be M first ES packets, and the M first ES packets may include N ninth ES packets and M min tenth ES packets.
[0297] S702: Parse the value of the model identifier from the field corresponding to the model identifier of the first ES packet.
[0298] Exemplarily, the model identifier is a field of ES.
[0299] Exemplarily, after receiving the first code stream, the second electronic device may have a decapsulation module parse each ES packet and extract the model identifier value from the ES packet. Among the multiple ES packets included in the first code stream, at least one ES packet includes a field corresponding to the model identifier; and the decapsulation module may parse the model identifier value from the field corresponding to the model identifier in at least one ES packet (the ninth ES packet or the tenth ES packet).
[0300] S703: When the value of the model identifier is a preset identifier value, read the model information code stream from the ninth ES packet.
[0301] Exemplarily, when the value of the model identifier is a preset identifier value, the decapsulation module can parse the header of each ES packet to determine the length of the encoded data of the audio signal; then, based on the length of the header, the length of the encoded data of the audio signal, and the total length of the ES packet, determine whether the ES packet contains a model information code stream (or sub-code stream). When the total length of the ES packet is greater than the sum of the length of the encoded data of the audio signal and the length of the header, it is determined that the ES packet contains a model information code stream (or sub-code stream), that is, the ES packet is the ninth ES packet. When the total length of the ES packet is equal to the sum of the length of the encoded data of the audio signal and the length of the header, it is determined that the ES packet does not contain a model information code stream (or sub-code stream), that is, the ES packet is the tenth ES packet.
[0302] For example, for a ninth ES packet, the coded data portion of the audio signal can be skipped based on the length of the coded data of the audio signal to read the model information code stream (or sub-code stream). When N is equal to 1, the model information code stream can be directly read from the ninth ES packet. When N is greater than 1, N sub-code streams can be read, and these N sub-code streams can constitute the model information code stream.
[0303] S704: Reconstruct the model based on the model information to obtain a decoding model.
[0304] Exemplarily, when N is equal to 1, the decapsulation module can parse the description information of the model information in the model information code stream to determine the length of the model information (or the encoded data of the model information); then, based on the length of the model information (or the encoded data of the model information), read out the model information (or the encoded data of the model information).
[0305] Exemplarily, when N is greater than 1, for each substream, the decapsulation module can parse the descriptive information of the information groups in the substream to determine the length of the current information group (or the encoded data of the current information group), the index corresponding to the current information group, and the total number of information groups N. Then, based on the length of the current information group (or the encoded data of the current information group), the information group (or the encoded data of the information group) is read from the substream. In this way, after obtaining N information groups, the N information groups can be concatenated based on the indexes corresponding to the respective information groups to obtain model information. Alternatively, after obtaining the encoded data of N information groups, the encoded data of the N information groups can be decoded to obtain N information groups; then, the N information groups can be concatenated based on the indexes corresponding to the respective information groups to obtain model information.
[0306] Exemplarily, when the encoded data of the model information (or information group) is read out, the decapsulation module can call the decoder to decode the encoded data of the model information (or information group) to determine the model information (or information group); thereafter, the model is reconstructed based on the model information (or the spliced information group) to obtain a decoding model.
[0307] S705 , reading the audio code stream from the ninth ES packet and using the tenth ES packet as the audio code stream.
[0308] In one possible approach, after parsing the header of the ninth ES packet and determining the length of the encoded audio data, the portion of the ninth ES packet corresponding to the encoded audio data can be determined based on the length of the encoded audio data. The header and the portion corresponding to the encoded audio data can then be extracted from the ninth ES packet to obtain an audio stream. The tenth ES packet can be used as the audio stream, which can then be sent to an audio decoder.
[0309] In one possible approach, after parsing the header of a ninth ES packet to determine the length of the encoded audio data, the encoded audio data can be read from the ninth ES packet based on the length of the encoded audio data. Furthermore, for each tenth ES packet, the header of the tenth ES packet can be parsed to determine the length of the encoded audio data, and then the encoded audio data can be read from the tenth ES packet based on the length of the encoded audio data. The encoded audio data can then be sent to an audio decoder.
[0310] S706: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0311] Exemplarily, when the decapsulation module sends an audio bitstream to the audio decoder, the audio decoder, after reading the reconstructed decoding model, can first read the encoded data of the audio signal from the audio bitstream, and then decode the encoded data of the audio signal based on the decoding model to obtain a reconstructed audio signal. When the decapsulation module sends the encoded data of the audio signal to the audio decoder, the audio decoder, after reading the reconstructed decoding model, can directly decode the encoded data of the audio signal based on the decoding model to obtain a reconstructed audio signal.
[0312] It should be understood that when N is 1, after parsing the model information stream from the first ninth ES packet, the model can be reconstructed to obtain a decoding model. Subsequently, the audio decoder can decode the first audio stream based on the decoding model to obtain the first frame of the reconstructed audio signal. Thereafter, upon receiving each tenth ES packet (the second first ES packet to the Mth first ES packet), the second electronic device can decode the tenth ES packet in real time to obtain the second to Mth frames of the reconstructed audio signal.
[0313] When N is greater than 1, the second electronic device needs to parse N sub-code streams from the 1st to Nth ES packets before it can reconstruct the model based on the N sub-code streams to obtain a decoding model; then, the audio decoder can decode the 1st to Nth audio code streams based on the decoding model to obtain the 1st to Nth frames of the reconstructed audio signal. After that, after the second electronic device receives each tenth ES packet (the M-N+1th first ES packet to the Mth first ES packet), the audio decoder can decode the tenth ES packet in real time to obtain the M-N+1th to Mth frames of the reconstructed audio signal. That is to say, in this case, the audio decoder's decoding of the audio code stream in the Nth frame ninth ES packet is delayed, or in other words, the decoding delay of the audio decoder is N frames.
[0314] Figure 8 is a schematic diagram illustrating an exemplary audio encoding process. In Figure 8, the target stream type is the second type; the audio stream and the model information stream can be encapsulated in the same ES packet, and the ES packet is encapsulated according to the target container format to obtain the first stream.
[0315] S801: Acquire an audio signal.
[0316] S802: Determine whether both the encoding model and the decoding model are updated.
[0317] S803: When both the encoding model and the decoding model are updated, encode the audio signal based on the updated encoding model to obtain an audio code stream.
[0318] S804: Generate a model information code stream based on the updated model information of the decoding model; wherein the updated decoding model corresponds to the updated encoding model.
[0319] S805: Encapsulate the model information code stream and the audio code stream into a second ES packet.
[0320] For example, S801 to S805 can refer to the description of S601 to S605 above, which will not be repeated here. The format of the second ES packet is similar to that of the first ES packet, that is, M second ES packets include N ninth ES packets and MN tenth ES packets, which will not be repeated here.
[0321] S806: Encapsulate the second ES packet according to the target container format to obtain a first code stream.
[0322] Exemplarily, the second ES packet may be encapsulated according to the target container format and the type of the second ES packet may be set to an audio type to obtain the first code stream.
[0323] In the process of encapsulating the second ES packet according to the target container format, the type identifier of the ES packet may be set to a first identifier value, where the first identifier value is used to indicate that the type of the ES packet is an audio type. The type identifier may be a field in the target container format used to indicate the type of the ES packet.
[0324] For example, in the embodiment of FIG8 , in one possible manner, the model identifier may be a field of ES; for details, reference may be made to the description in the embodiment of FIG6A above, which will not be repeated here.
[0325] In one possible approach, when encapsulating the second ES packet according to the target container format, a field in the additional information (the field being a field in the target container format) can be used as a model identifier, and the value of the model identifier can be set to a preset identifier value. For example, if the target container format is TS / MP4, a field in the additional information such as the nn_type field can be used as a model identifier, and the field value of the nn_type field can be set to a preset identifier value. For details, please refer to the description in the embodiment of FIG. 6A above, which will not be repeated here. In this way, the second electronic device can determine whether the decoding model needs to be reconstructed without parsing the ES packet.
[0326] Among them, the ES packet can be divided into multiple small packets and then encapsulated according to the target container format. This is conducive to the bit stream distribution device to perform bit stream distribution and audio and video synchronization control.
[0327] FIG9 is a schematic diagram illustrating an exemplary audio decoding process. FIG9 is an audio decoding process corresponding to the audio encoding process of FIG8 .
[0328] S901: Receive a first code stream.
[0329] For example, when the first electronic device encodes according to the audio encoding method in the embodiment of FIG8 , assuming the target container format is TS, the first code stream received by the second electronic device is substantially a TS stream; assuming the target container format is MP4, the first code stream received by the second electronic device is substantially an MP4 stream. This application uses the example of the first code stream being a TS stream for illustration.
[0330] Exemplarily, when the model identifier is a field of the target container format, the process of parsing the model identifier from the first code stream may refer to S902 to S903:
[0331] S902: Decapsulate the first code stream according to the target container format to obtain a field corresponding to the model identifier.
[0332] S903: Parse the field corresponding to the model identifier to obtain the value of the model identifier.
[0333] Exemplarily, after the second electronic device receives the first code stream, the decapsulation module can decapsulate the first code stream to obtain a field corresponding to the model identifier and M second ES packets (the M second ES packets include N ninth ES packets and MN tenth ES packets). Then, the decapsulation module can parse the field corresponding to the model identifier to obtain the value of the model identifier.
[0334] Exemplarily, when the model identifier is a field of ES, the value of the model identifier can be parsed from the field corresponding to the model identifier of the ES package; this can be referred to the description of S702 above and will not be repeated here.
[0335] S904: When the value of the model identifier is a preset identifier value, read the model information code stream from the ninth ES packet.
[0336] S905: Reconstruct the model based on the model information code stream to obtain a decoding model.
[0337] S906: Read the audio code stream from the ninth ES packet and use the tenth ES packet as the audio code stream.
[0338] S907: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0339] For example, S904 to S907 may refer to the description of S503 to S506 above, which will not be repeated here.
[0340] FIG10A is a schematic diagram illustrating an exemplary audio encoding process. In FIG10A , the target bitstream type is the second type. The audio bitstream and the model information bitstream can be encapsulated in separate ES packets, and the ES packets of the model information bitstream and the audio bitstream can be encapsulated together according to the target container format to obtain a first bitstream.
[0341] S1001: Acquire an audio signal.
[0342] S1002: Determine whether both the encoding model and the decoding model are updated.
[0343] S1003: When both the encoding model and the decoding model are updated, the audio signal is encoded based on the updated encoding model to obtain an audio code stream.
[0344] S1004 , generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.
[0345] For example, S1001 to S1004 may refer to the description of S601 to S604 above, which will not be repeated here.
[0346] S1005: Encapsulate the model information code stream into a third ES packet.
[0347] Exemplarily, the model information may be divided into N information groups; then, one information group is encapsulated into a third ES packet, thus obtaining N third ES packets.
[0348] Exemplarily, when N is equal to 1, there is one third ES packet, which includes a model information code stream and description information of the model information. Please refer to the description of the above embodiment and will not be repeated here.
[0349] Exemplarily, when N is greater than 1, there are N third ES packets, each of which includes a substream and description information of the substream. For details, please refer to the description of the above embodiment and will not be repeated here.
[0350] S1006: Use the audio code stream as the fourth ES packet.
[0351] Exemplarily, one audio code stream may be taken as a fourth ES packet, and thus N fourth ES packets may be obtained.
[0352] S1007 , encapsulating the third ES packet and the fourth ES packet according to the target container format, setting the type of the third ES packet to a preset type and the type of the fourth ES packet to an audio type, to obtain a first code stream.
[0353] Illustratively, in the process of encapsulating the fourth ES packet and the third ES packet according to the target container format, the type identifier of the fourth ES packet may be set to a first identifier value, where the first identifier value is used to indicate that the type of the ES packet is an audio type.
[0354] For example, during encapsulation of the fourth ES packet and the third ES packet according to the target container format, the type identifier of the third ES packet can be set to a second identifier value, where the second identifier value indicates that the ES packet type is a preset type. The model identifier indicates that the ES packet includes a model information stream or substream. For details, please refer to the description of the above embodiment and will not be repeated here.
[0355] In one possible way, the preset type can be a private type. In this case, the private type identifier corresponding to the third ES packet can include private_stream_id_1 and private_stream_id_2; accordingly, the private type identifier private_stream_id_1 corresponding to the third ES packet can be set to 0xBD, or the private type identifier private_stream_id_2 corresponding to the third ES packet can be set to 0xBF.
[0356] In one possible manner, the preset type may be a custom type, in which case the custom type identifier corresponding to the third ES packet may include stream_id and stream_type. The custom type identifiers stream_id and stream_type corresponding to the third ES packet may be set to the second identifier value.
[0357] For example, in the embodiment of FIG10A , the model identifier may be a field of the ES or a field of the target container format, and this application does not impose any limitation thereto; for details, please refer to the description above, which will not be repeated here.
[0358] Exemplarily, when the model identifier is a field of ES, the model identifier may be located in at least one third ES packet, or in at least one fourth ES packet.
[0359] Referring to FIG. 10B , illustratively, in FIG. 10B , the target container format is TS, N=1; the third ES packet and the fourth ES packet may be encapsulated into a TS stream. In this case, the first code stream is a TS stream.
[0360] FIG11 is a schematic diagram illustrating an exemplary audio decoding process. FIG11 is an audio decoding process corresponding to the audio encoding process of FIG10A .
[0361] S1101: Receive a first code stream.
[0362] Exemplarily, when the model identifier is a field of the target container format, the process of parsing the model identifier from the first code stream may refer to S1102 to S1103:
[0363] S1102: Decapsulate the first code stream according to the target container format to obtain the field corresponding to the model identifier, the third ES packet, and the fourth ES packet.
[0364] Exemplarily, by decapsulating the first code stream according to the target container format, a field corresponding to the model identifier, multiple ES packets, and the type of the ES packet can be obtained. The decapsulation module can parse the type identifier of each ES packet. When the type identifier of the ES packet is determined to be the second identifier value, the type of the ES packet can be determined to be a preset type. In this case, the ES packet can be referred to as a third ES packet. Exemplarily, when the decapsulation model determines that the type identifier of the ES packet is the first identifier value, the type of the ES packet can be determined to be an audio type. In this case, the ES packet can be referred to as a fourth ES packet.
[0365] S1103: Parse the field corresponding to the model identifier to obtain the value of the model identifier.
[0366] S1104: When the value of the model identifier is a preset identifier value, read the model information code stream from the third ES packet.
[0367] S1105: Reconstruct the model based on the model information to obtain a decoding model.
[0368] S1106: Read the audio stream from the fourth ES packet.
[0369] S1107: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0370] For example, S1104 to S1107 may refer to the description of S703 to S706 above, which will not be repeated here.
[0371] In this way, compared with encapsulating the second code stream and the third code stream in the same ES packet, the second electronic device does not need to split the ES packet to parse the second code stream and the third code stream, thereby reducing the complexity of the parsing process.
[0372] Figure 12A is a schematic diagram illustrating an exemplary audio encoding process. In Figure 12A , the target bitstream type is the second type. The model information bitstream can be encapsulated as metadata and separately encapsulated with the audio bitstream in separate ES packets. Subsequently, the ES packets for the model information bitstream and the audio bitstream are encapsulated together according to the target container format to obtain the first bitstream.
[0373] S1201: Acquire an audio signal.
[0374] S1202: Determine whether both the encoding model and the decoding model are updated.
[0375] S1203: When both the encoding model and the decoding model are updated, encode the audio signal based on the updated encoding model to obtain an audio code stream.
[0376] S1204: Generate a model information code stream based on the updated model information of the decoding model; wherein the updated decoding model corresponds to the updated encoding model.
[0377] For example, S1201 to S1204 may refer to the description of S601 to S604 above, which will not be repeated here.
[0378] S1205: Encapsulate the model information code stream as metadata to obtain a third ES packet.
[0379] Exemplarily, the model information may be divided into N information groups; then, one information group is encapsulated into a third ES packet, thus obtaining N third ES packets.
[0380] Referring to Figure 12B , the model information code stream can be exemplarily described as a Metadata AU (Access Unit). When N is 1, a Metadata AU can be treated as a Metadata AU Cell; then, the Metadata AU Cell is encapsulated into a third ES packet to obtain a third ES packet. When N is greater than 1, a Metadata AU can be split into N Metadata AU Cells; then, the Metadata AU Cell is encapsulated into a third ES packet to obtain N third ES packets.
[0381] S1206: Use the audio code stream as the fourth ES packet.
[0382] Exemplarily, one audio code stream may be taken as a fourth ES packet, and thus N fourth ES packets may be obtained.
[0383] S1207 , encapsulating the third ES packet and the fourth ES packet according to the target container format, setting the type of the third ES packet to a metadata type and the type of the fourth ES packet to an audio type, to obtain a first code stream.
[0384] Exemplarily, the metadata type is one of the preset types in the embodiment of FIG. 11 .
[0385] In FIG. 12B ( 2 ), the target container format is TS. Based on FIG. 12B ( 1 ), N third ES packets and M fourth ES packets are encapsulated into a TS stream. Thus, the obtained first code stream is a TS stream.
[0386] Exemplarily, in the process of encapsulating the fourth ES packet and the third ES packet according to the target container format, the type identifier of the fourth ES packet can be set to the first identification value; and the type identifier of the third ES packet can be set to the third identification value, and the third identification value is used to characterize that the type of the ES packet is a metadata type.
[0387] In the embodiment of FIG12A , the syntax of the Metadata AU Cell in the first code stream may also be modified. Assuming that the target container format is TS, that is, the first code stream is a TS stream, the syntax of the Metadata AU Cell in the TS stream may be as shown in Table 4 below:
[0388] Table 4
[0389] In Table 4, decoder_config_flag indicates whether the current Metadata AU contains decoder initialization information. In the embodiment of Figure 12A, the decoder_config_flag identifier in the Metadata AU Cell syntax of the Metadata AU corresponding to the model information code stream is set to 1 to indicate that the model update information is necessary for audio decoder initialization.
[0390] In the embodiment of FIG12A , the model identifier may be a field of the ES or a field of the target container format, and this application does not impose any limitation on this.
[0391] Exemplarily, when the model identifier is a field of ES, the model identifier may be located in at least one third ES packet, or in at least one fourth ES packet.
[0392] Figure 13 is a schematic diagram illustrating an exemplary audio decoding process. Figure 13 is an audio decoding process corresponding to the audio encoding process of Figure 12A.
[0393] S1301: Receive a first code stream.
[0394] S1302: Decapsulate the first code stream according to the target container format to obtain the field corresponding to the model identifier, the third ES packet, and the fourth ES packet.
[0395] Exemplarily, by decapsulating the first code stream according to the target container format, a field corresponding to the model identifier, multiple ES packets, and the type of the ES packet can be obtained. The decapsulation module can parse the type identifier of each ES packet. When the type identifier of the ES packet is determined to be the third identifier value, the type of the ES packet can be determined to be a metadata type. In this case, the ES packet can be referred to as a third ES packet. Exemplarily, when the decapsulation model determines that the type identifier of the ES packet is the first identifier value, the type of the ES packet can be determined to be an audio type. In this case, the ES packet can be referred to as a fourth ES packet.
[0396] S1303: Parse the field corresponding to the model identifier to obtain the value of the model identifier.
[0397] For example, S1301 to S1303 may refer to the description of S1101 to S1103 above, which will not be repeated here.
[0398] S1304: When the value of the model identifier is a preset identifier value, read the model information code stream from the third ES packet.
[0399] S1305: Reconstruct the model based on the model information to obtain a decoding model.
[0400] S1306: Read the audio stream from the fourth ES packet.
[0401] S1307: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0402] Exemplarily, the decapsulation module can also parse the decoder_config_flag in the syntax of the Metadata AU Cell corresponding to the third ES packet; if decoder_config_flag is "1", the decoder can read the decoding model (ie, initialize), and then decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0403] For example, S1305 to S1307 may refer to the description of S703 to S706 above, which will not be repeated here.
[0404] Figure 14A is a schematic diagram illustrating an exemplary audio encoding process. In Figure 14A , the target bitstream type is the second type; the audio bitstream and the model information bitstream can be encapsulated in different ES packets, and the ES packets of the model information bitstream and the audio bitstream can be encapsulated in the target container format to obtain the first bitstream.
[0405] S1401: Acquire an audio signal.
[0406] S1402: Determine whether both the encoding model and the decoding model are updated.
[0407] S1403: When both the encoding model and the decoding model are updated, encode the audio signal based on the updated encoding model to obtain an audio code stream.
[0408] S1404 , generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.
[0409] For example, S1401 to S1404 may refer to the description of S601 to S604 above, which will not be repeated here.
[0410] S1405: Encapsulate the model information code stream into a fifth ES packet.
[0411] S1406: Encapsulate the fifth ES packet according to the target container format and set the type of the fifth ES packet to a preset type to obtain a fourth code stream.
[0412] Illustratively, in the process of encapsulating the fifth ES packet according to the target container format, the type identifier of the ES packet may be set to the second identifier value.
[0413] For example, if the target container format is TS, the fifth ES packet is encapsulated into a TS stream, namely, the fourth code stream.
[0414] S1407: Use the audio code stream as the sixth ES packet.
[0415] S1408: Encapsulate the sixth ES packet according to the target container format and set the type of the sixth ES packet to an audio type to obtain a fifth code stream.
[0416] Exemplarily, one audio code stream may be taken as a sixth ES packet, and thus, N sixth ES packets may be obtained.
[0417] Illustratively, in the process of encapsulating the sixth ES packet according to the target container format, the type identifier of the sixth ES packet may be set to the first identifier value.
[0418] For example, if the target container format is TS, the sixth ES packet is encapsulated into a TS stream, namely, the fifth code stream.
[0419] S1409: Combine the fourth code stream and the fifth code stream to obtain a first code stream.
[0420] 14B , the target container format is TS, the first code stream is TS stream, and the first code stream may include two TS streams: TS stream 1 and TS stream 2, wherein TS stream 1 is the fourth code stream; TS stream 2 is the fifth code stream.
[0421] For example, in the embodiment of FIG14A , the identifier may be the field value of the ES field, or the field value of the corresponding field of the target container format, and this application does not impose any restrictions on this; for details, please refer to the description above, which will not be repeated here.
[0422] Exemplarily, when the model identifier is a field in the target container format, the model identifier may be located in the fourth codestream or the fifth codestream.
[0423] Exemplarily, when the model identifier is a field of ES, the model identifier may be located in at least one fifth ES packet, or in at least one sixth ES packet.
[0424] Figure 15 is a schematic diagram illustrating an exemplary audio decoding process. Figure 15 is an audio decoding process corresponding to the audio encoding process of Figure 14A.
[0425] S1501: Receive a first code stream.
[0426] Exemplarily, when the first electronic device encodes according to the audio encoding method in the embodiment of Figure 14A, assuming that the target container format is TS, the first code stream received by the second electronic device is essentially a TS stream; assuming that the target container format is MP4, the first code stream received by the second electronic device is essentially an MP4 stream.
[0427] In this case, the first code stream may include the fourth code stream and the fifth code stream.
[0428] Exemplarily, when the model identifier is a field in the target container format and the fourth codestream includes a field corresponding to the model identifier, the process of parsing the value of the model identifier from the first codestream may refer to S1502 to S1503:
[0429] S1502: Decapsulate the fourth code stream according to the target container format to obtain a field corresponding to the model identifier and a fifth ES packet.
[0430] The following description is made by taking the case where the first code stream is a TS stream as an example.
[0431] Exemplarily, after the second electronic device receives the first code stream, the decapsulation module can decapsulate each TS stream according to the target container format to obtain the ES packet and the type of ES packet; and decapsulate part of the TS stream to obtain the field corresponding to the model identifier.
[0432] For example, when the ES packet type is a preset type, the corresponding TS stream can be determined to be the fourth code stream and the corresponding ES packet can be determined to be the fifth ES packet. When the ES packet type is an audio type, the corresponding TS stream can be determined to be the fifth code stream and the corresponding ES packet can be determined to be the sixth ES packet.
[0433] S1503: Parse the field corresponding to the model identifier to obtain the value of the model identifier.
[0434] S1504: When the value of the model identifier is a preset identifier value, read the model information code stream from the fifth ES packet.
[0435] S1505: Reconstruct the model based on the model information to obtain a decoding model.
[0436] S1506: Decapsulate the fifth code stream according to the target container format to obtain a sixth ES packet.
[0437] S1507: Read the audio stream from the sixth ES packet.
[0438] S1508: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0439] For example, S1503 to S1508 may refer to the description of S703 to S706 above, which will not be repeated here.
[0440] In this way, compared with encapsulating the second code stream and the third code stream in the same TS stream, the second electronic device does not need to split the TS stream to parse the second code stream and the third code stream, thereby reducing the complexity of the parsing process.
[0441] Figure 16A is a schematic diagram illustrating an exemplary audio encoding process. In Figure 16A , the target bitstream type is the third type; the audio bitstream and the model information bitstream are encapsulated in separate ES packets, and the ES packets of the model information bitstream and the audio bitstream are encapsulated in the target container format to obtain a sixth bitstream and a seventh bitstream, respectively. The sixth and seventh bitstreams are then encapsulated in accordance with the target transport protocol to obtain a first bitstream.
[0442] S1601: Acquire an audio signal.
[0443] S1602: Determine whether both the encoding model and the decoding model are updated.
[0444] S1603: When both the encoding model and the decoding model are updated, encode the audio signal based on the updated encoding model to obtain an audio code stream.
[0445] S1604: Generate a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.
[0446] For example, S1601 to S1604 may refer to the description of S601 to S604 above, which will not be repeated here.
[0447] S1605: Encapsulate the model information code stream into the seventh ES packet.
[0448] S1606 , encapsulating a seventh ES packet according to the target container format and setting the type of the seventh ES packet to a preset type to obtain a sixth code stream.
[0449] S1607: Use the audio code stream as the eighth ES packet.
[0450] S1608: Encapsulate the eighth ES packet according to the target container format and set the type of the eighth ES packet to an audio type to obtain a seventh code stream.
[0451] S1609: Encapsulate the sixth code stream and the seventh code stream according to the target transmission protocol to obtain a first code stream.
[0452] Exemplarily, when the target transmission protocol is the Http (Hyper Text Transfer Protocol) transmission protocol, the first code stream may be an HLS stream. In this case, the first code stream may include: an index file, a sixth code stream, and a seventh code stream.
[0453] Referring to Figure 16B, the target transmission protocol is HLS, and the target container format is TS; the first code stream is the HLS stream, which can include two TS streams encapsulated according to HLS: TS stream 1 and TS stream 2, where TS stream 1 is the sixth code stream; TS stream 1 is the seventh code stream.
[0454] For example, during the decoding process, the information in the index file is usually parsed sequentially from the beginning to the end; and in order to ensure that the decoding model is reconstructed before the audio code stream is obtained, the index information of the sixth code stream can be set before the index information of all seventh code streams in the index file.
[0455] Exemplarily, the index file also includes multiple tags, such as an initialization tag, which indicates how to obtain media initialization information (Media Initialization Section). Media initialization information may refer to information that needs to be parsed before decoding a media stream (such as an audio stream). Furthermore, the index information of the sixth stream may be used as attribute information of the initialization tag.
[0456] Illustratively, the index information may be a Uniform Resource Locator (URL), which is not limited in this application.
[0457] For example, in the embodiment of FIG16A , the model identifier may be a field of the ES or a field of the target container format, and this application does not impose any limitation on this; for details, please refer to the description above and will not be repeated here.
[0458] Exemplarily, when the model identifier is a field in the target container format, the model identifier may be located in the sixth code stream or the seventh code stream.
[0459] Exemplarily, when the model identifier is a field of the ES, the model identifier may be located in at least one seventh ES packet, or in at least one eighth ES packet.
[0460] In the embodiment of FIG. 16A , the model identifier may also be a field of the target transmission protocol.
[0461] It should be noted that the first code stream in the embodiment of FIG. 16A differs from the first code stream in the embodiment of FIG. 14A . The first code stream in the embodiment of FIG. 14A includes only the sixth and seventh code streams, while the first code stream in the embodiment of FIG. 16A includes, in addition to the sixth and seventh code streams, a field indicating the target transmission protocol.
[0462] Among them, a second code stream with different reconstruction accuracy can be generated (which can be understood as the accuracy of the reconstructed signal obtained by decoding the second code stream); this can help the code stream distribution device distribute the first code stream containing the second code stream with different reconstruction accuracy according to the bandwidth, thereby ensuring the smoothness of the reconstructed signal playback.
[0463] Figure 17 is a schematic diagram illustrating an exemplary audio decoding process. Figure 17 is an audio decoding process corresponding to the audio encoding process of Figure 16A.
[0464] S1701: Receive a first code stream.
[0465] S1702: Decapsulate the first code stream according to the target transmission protocol to obtain a sixth code stream and a seventh code stream.
[0466] Exemplarily, when the target transmission protocol is the Http transmission protocol, the first code stream may be decapsulated according to the Http transmission protocol to obtain an index file and multiple code streams.
[0467] In one possible approach, index information of the sixth code stream may be read from an index file, and then the sixth code stream may be determined from multiple code streams based on the index information of the sixth code stream; and index information of the seventh code stream may be read from an index file, and then the seventh code stream may be determined from multiple code streams based on the index information of the seventh code stream.
[0468] In one possible approach, index information of the seventh code stream may be read from an index file, and then the seventh code stream may be determined from multiple code streams based on the index information of the seventh code stream; and index information of the sixth code stream may be read from attribute information of an initialization tag in the index file, and the sixth code stream may be obtained from multiple code streams based on the index information of the sixth code stream.
[0469] S1703: Decapsulate the sixth code stream according to the target container format to obtain the field corresponding to the model identifier and the seventh ES packet.
[0470] S1704: Parse the field corresponding to the model identifier to obtain the value of the model identifier.
[0471] S1705: When the value of the model identifier is a preset identifier value, read the model information code stream from the seventh ES packet.
[0472] S1706: Reconstruct the model based on the model information to obtain a decoding model.
[0473] S1707: Decapsulate the seventh code stream according to the target container format to obtain an eighth ES packet.
[0474] S1708: Read the audio stream from the eighth ES packet.
[0475] S1709: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.
[0476] For example, S1703 to S1709 may refer to the description of S1502 to S1508 above, which will not be repeated here.
[0477] In one possible manner, the first electronic device may transmit the index information corresponding to the model information of the decoding model to the second electronic device. Specifically, the index information corresponding to the model information of the decoding model may be determined, and then the first code stream may be obtained based on the second code stream and the index information corresponding to the model information of the decoding model. In this case, the preset identification value is used to characterize that the first code stream includes the index information corresponding to the model information of the decoding model. During the decoding process, the process for the second electronic device to obtain the model information of the decoding model may be as follows: parsing the index information of the model information from the first code stream; and obtaining the model information based on the index information of the model information. That is, in this case, the preset identification value instructs the second electronic device to obtain the model information of the decoding model based on the index information obtained from the first code stream.
[0478] For example, one possible method for obtaining the first bitstream based on the index information corresponding to the second bitstream and the model information of the decoding model is to use the second bitstream as a third ES packet, encapsulate the third ES packet according to the target container format to obtain the fifth bitstream, and then encapsulate the fifth bitstream according to the target transport protocol to obtain the first bitstream. In this case, the first bitstream can include an index file and the fifth bitstream, and the index information of the decoding model can be added to the index file. In one possible method, the index information of the decoding model can be added before the index information of the fifth bitstream. In another possible method, the index information corresponding to the model information of the decoding model can be used as attribute information of the initialization tag in the index file.
[0479] Compared with the code stream carrying the model information in the first code stream, the code rate overhead of only carrying the index information corresponding to the model information of the decoding model in the first code stream is lower; and the bandwidth requirement for the first electronic device can be reduced.
[0480] For example, one possible method for obtaining the first bitstream based on the second bitstream and the index information corresponding to the model information of the decoding model is to generate an eighth bitstream based on the index information corresponding to the model information of the decoding model; and encapsulate the eighth bitstream and the second bitstream to obtain the first bitstream. The method for encapsulating the eighth bitstream and the second bitstream to obtain the first bitstream can be similar to the method for encapsulating the third bitstream and the second bitstream to obtain the first bitstream, and will not be further described here.
[0481] In one possible approach, a first electronic device and a second electronic device may pre-agree on a preset storage location in a third device for storing an updated coding model and an updated decoding model; in this way, the first electronic device does not need to transmit a model information codestream of the decoding model or index information corresponding to the model information of the decoding model. In this case, after the decoding model and coding model are updated in the preset storage location in the third device, the value of the model identifier in the first codestream can be set to a preset identifier value; in this case, the preset identifier value is used to indicate that the model information of the decoding model is stored in the preset storage location in the third device. In this way, the second electronic device can obtain the model information of the decoding model from the preset storage location in the third device; that is, the preset identifier value instructs the second electronic device to obtain the model information of the decoding model from the preset storage location in the third device. In this case, the first codestream does not need to transmit the model information codestream or index information corresponding to the model information, which can reduce bitrate overhead and bandwidth requirements for the first electronic device.
[0482] It should be noted that, assuming that the method by which the second electronic device obtains model information from the first code stream is referred to as the first method; the method by which the second electronic device obtains model information based on the index information corresponding to the model information obtained from the first code stream is referred to as the second method; and the method by which the second electronic device obtains model information from a preset storage location of a third electronic device is referred to as the third method. In one possible method, the first electronic device and the second electronic device can pre-agree on the method by which the second electronic device obtains model information of the decoding model. During the encoding and decoding process, when the first electronic device and the second electronic device agree on the first method, the first electronic device can generate a third code stream based on the model information of the decoding model; and encapsulate the second code stream and the third code stream to obtain the first code stream. During the encoding and decoding process, when the first electronic device and the second electronic device agree on the second method, the first electronic device can generate the first code stream based on the second code stream and the index information corresponding to the model information of the decoding model. During the encoding and decoding process, when the first electronic device and the second electronic device agree on the third method, the first electronic device can use the second code stream to generate the first code stream.
[0483] In one possible approach, during the encoding and decoding process, the first electronic device may select one of the first, second, and third approaches as a target approach. When the target approach is the first approach, the first electronic device may generate a third code stream based on model information of the decoding model; and encapsulate the second and third code streams to obtain the first code stream. When the target approach is the second approach, the first electronic device may generate the first code stream based on the second code stream and index information corresponding to the model information of the decoding model. When the target approach is the third approach, the first electronic device may use the second code stream to generate the first code stream.
[0484] Figure 18 is a schematic diagram of an exemplary encoding device. The encoding device can be used to execute the encoding method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.
[0485] 18 , illustratively, the encoding device may include:
[0486] The signal acquisition module 1801 is used to acquire the signal to be encoded;
[0487] Coding module 1802 is configured to generate a first code stream, wherein the first code stream includes a model identifier and a second code stream, the second code stream being obtained by encoding a signal to be coded based on a coding model, the value of the model identifier being a preset identifier value, the preset identifier value being used to instruct the second electronic device to obtain model information of a decoding model, where the decoding model corresponds to the coding model.
[0488] It should be understood that the encoding device can also perform other steps in the above encoding method, which will not be described in detail here.
[0489] Figure 19 is a schematic diagram of an exemplary decoding device. The decoding device can be used to execute the decoding method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.
[0490] 19 , illustratively, the decoding apparatus may include:
[0491] A code stream receiving module 1901 is configured to receive a first code stream, the first code stream including a model identifier and a second code stream, the second code stream being generated based on the encoded information;
[0492] The decapsulation module 1902 is configured to obtain model information of the decoding model when the value of the model identifier is a preset identifier value;
[0493] The decapsulation module 1902 is further configured to reconstruct the model based on the model information to obtain a decoding model;
[0494] The decoder 1903 is configured to decode the second code stream based on the decoding model to obtain a reconstructed signal.
[0495] It should be understood that the decoding device can also perform other steps in the above decoding method, which will not be described in detail here.
[0496] In an example, FIG20 shows a schematic block diagram of a device 2000 according to an embodiment of the present application. The device 2000 may include: a processor 2001 and a transceiver / transceiver pin 2002 , and optionally, a memory 2003 .
[0497] The various components of the device 2000 are coupled together via a bus 2004, wherein the bus 2004 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the various buses are collectively referred to as bus 2004 in the figure.
[0498] Optionally, the memory 2003 may be used to store instructions in the aforementioned method embodiment. The processor 2001 may be used to execute the instructions in the memory 2003 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.
[0499] The apparatus 2000 may be the electronic device or a chip of the electronic device in the above method embodiment. The electronic device may include a first electronic device and a second electronic device.
[0500] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0501] The present application also provides a chip comprising one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the electronic device implements the aforementioned method steps in the aforementioned embodiment. The interface circuit is a transceiver / transceiver pin 2002.
[0502] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.
[0503] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.
[0504] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.
[0505] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.
[0506] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0507] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0508] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0509] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0510] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.
[0511] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0512] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.
[0513] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.
[0514] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0515] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.
Claims
1. A decoding method, characterized in that: The method comprises: receiving a first code stream, the first code stream including a model identifier and a second code stream, the second code stream being generated based on an encoded signal; When the value of the model identifier is a preset identifier value, obtaining model information; Reconstructing the model based on the model information to obtain a decoding model; The second code stream is decoded based on the decoding model to obtain a reconstructed signal.
2. The method according to claim 1, characterized in that When the value of the model identifier is the preset identifier value, the first code stream further includes a third code stream, and the third code stream is generated based on the model information of the decoding model; The obtaining of model information includes: Parsing the third code stream from the first code stream; Based on the third code stream, model information of the decoding model is obtained.
3. The method according to claim 1, characterized in that When the value of the model identifier is the preset identifier value, the first bitstream further includes index information corresponding to the model information of the decoding model; The obtaining of model information includes: Parsing the first bitstream to obtain index information corresponding to the model information of the decoding model; Based on the index information corresponding to the model information of the decoding model, the model information of the decoding model is obtained.
4. The method according to claim 1, characterized in that: The preset identification value is used to represent that the model information of the decoding model is stored in a preset storage location of the remote electronic device; The obtaining of model information includes: The model information of the decoding model is obtained from the preset storage location.
5. The method according to any one of claims 1 to 4, characterized in that: The model identifier is a field of an elementary stream ES, and the method further includes: The field corresponding to the model identifier in the first bitstream is parsed to obtain a value of the model identifier.
6. The method according to any one of claims 1 to 4, characterized in that: The target codestream type of the first codestream is a codestream type encapsulated according to a target container format or a codestream type encapsulated according to a target transmission protocol, the model identifier is a field of the target container format, and the method further includes: Decapsulating the first code stream according to the target container format to obtain a field corresponding to the model identifier; The field corresponding to the model identifier is parsed to obtain the value of the model identifier.
7. The method according to any one of claims 1 to 4, characterized in that: The target code stream type of the first code stream is a code stream type encapsulated according to a target transmission protocol, and the model identifier is a field of the target transmission protocol; the method further includes: Decapsulating the first code stream according to the target transmission protocol to obtain a field corresponding to the model identifier; The field corresponding to the model identifier is parsed to obtain the value of the model identifier.
8. The method according to any one of claims 1 to 7, characterized in that: The model identifier is used to characterize the type of the decoding model, and the method further includes: When the value of the model identifier is a preset identifier value, determining whether there is a decoding model of a type corresponding to the preset identifier value; The reconstructing the model based on the model information to obtain a decoding model includes: When there is no decoding model of the type corresponding to the preset identification value, model reconstruction is performed based on the model information to obtain a decoding model of the type corresponding to the preset identification value.
9. The method according to claim 2, characterized in that: The step of parsing the third code stream from the first code stream comprises: The first code stream is parsed according to the target code stream type of the first code stream to obtain the third code stream.
10. The method according to claim 9, characterized in that The target bitstream type is an ES type, the first bitstream further includes a first ES packet, and the first ES packet includes the second bitstream, the third bitstream and the model identifier; The step of parsing the first code stream according to the target code stream type of the first code stream to obtain the third code stream includes: Reading the third code stream from the first ES packet; The method further comprises: The second code stream is read from the first ES packet.
11. The method according to claim 9, characterized in that The target code stream type is a code stream type encapsulated according to the target container format; The step of parsing the first code stream according to the target code stream type of the first code stream to obtain the third code stream includes: Decapsulating the first code stream according to the target container format to obtain a second ES packet; Reading the third code stream from the second ES packet; The method further comprises: The second code stream is read from the second ES packet.
12. The method according to claim 9, characterized in that The target code stream type is a code stream type encapsulated according to the target container format; The step of parsing the first code stream according to the target code stream type of the first code stream to obtain the third code stream includes: Decapsulating the first code stream according to the target container format to obtain a third ES packet, a fourth ES packet, a type of the third ES packet, and a type of the fourth ES packet; Reading the third code stream from a third ES packet; wherein the type of the third ES packet is a preset type; The method further comprises: The second code stream is read from a fourth ES packet; wherein the type of the fourth ES packet is a media type.
13. The method according to claim 12, characterized in that The preset type is a metadata type.
14. The method according to claim 9, characterized in that The target codestream type is a codestream type encapsulated according to a target container format, the first codestream further includes a fourth codestream and a fifth codestream, the fifth codestream includes the second codestream, the fourth codestream includes the third codestream, and the model identifier is located in the fourth codestream or the fifth codestream; The step of parsing the first code stream according to the target code stream type of the first code stream to obtain the third code stream includes: Decapsulating the fourth code stream according to the target container format to obtain a fifth ES packet and a type of the fifth ES packet; Reading the third code stream from the fifth ES packet, wherein the type of the fifth ES packet is a preset type; The method further comprises: Decapsulating the fifth code stream according to the target container format to obtain a sixth ES packet and a type of the sixth ES packet; The second code stream is read from the sixth ES packet, wherein the type of the sixth ES packet is a media type.
15. The method according to claim 9, characterized in that The target code stream type is a code stream type encapsulated according to a target transmission protocol, the first code stream further includes a sixth code stream and a seventh code stream, the seventh code stream includes the second code stream, the sixth code stream includes the third code stream, and the model identifier is located in the sixth code stream or the seventh code stream; The step of parsing the first code stream according to the target code stream type of the first code stream to obtain the third code stream includes: Decapsulating the first bitstream according to the target transmission protocol to obtain the sixth bitstream; Decapsulating the sixth code stream according to the target container format to obtain a seventh ES packet and a type of the seventh ES packet; Reading the third code stream from the seventh ES packet, wherein the type of the seventh ES packet is a preset type; The method further comprises: Decapsulating the seventh code stream decapsulated from the first code stream according to the target container format to obtain an eighth ES packet and a type of the eighth ES packet; The second code stream is read from the eighth ES packet, wherein the type of the eighth ES packet is a media type.
16. The method according to claim 15, characterized in that The first code stream further includes an index file, the index file includes index information of the sixth code stream and index information of the seventh code stream, and the index information of the sixth code stream is located before the index information of the seventh code stream; Decapsulating the first code stream according to the target transmission protocol to obtain the sixth code stream includes: Decapsulating the first code stream according to the target transmission protocol to obtain the index file and multiple code streams; The index information of the sixth code stream is read from the index file, and the sixth code stream is determined from the multiple code streams according to the index information of the sixth code stream.
17. The method according to claim 15, characterized in that The first code stream further includes an index file, and the attribute information of the initialization tag in the index file includes the index information of the sixth code stream; Decapsulating the first code stream according to the target transmission protocol to obtain the sixth code stream includes: Decapsulating the first code stream according to the target transmission protocol to obtain the index file and multiple code streams; The index information of the sixth code stream is read from the attribute information of the initialization tag of the index file, and the sixth code stream is determined from the multiple code streams according to the index information of the sixth code stream.
18. The method according to claim 10, characterized in that The reconstructed signal is M frames, the number of the first ES packets is M, the M first ES packets include N ninth ES packets and MN tenth ES packets, the ninth ES packet includes a sub-code stream and a second code stream, the tenth ES packet includes a second code stream, M and N are positive integers, and M is greater than or equal to N; The reading the third code stream from the first ES packet includes: Reading N sub-code streams from the N ninth ES packets; splicing the N sub-code streams to obtain the third code stream; The reading the second code stream from the first ES packet includes: N second code streams are read from the N ninth ES packets, and the MN tenth ES packets are used as MN second code streams.
19. The method according to any one of claims 2 or 9 to 18, characterized in that The obtaining, based on the third bitstream, model information of the decoding model includes: Parsing the model information from the third bitstream; or, The encoded data of the model information is parsed from the third code stream, and the encoded data of the model information is decoded to obtain the model information.
20. The method according to any one of claims 1 to 19, characterized in that The model information includes model structure information or model parameters.
21. A decoding device, characterized in that: The device comprises: A code stream receiving module, configured to receive a first code stream, the first code stream including a model identifier and a second code stream, the second code stream being generated based on an encoded signal; A decapsulation module, used for obtaining model information when the value of the model identifier is a preset identifier value; The decapsulation module is further used to reconstruct the model based on the model information to obtain a decoding model; A decoder is used to decode the second code stream based on the decoding model to obtain a reconstructed signal.
22. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the decoding method according to any one of claims 1 to 20.
23. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method described in any one of claims 1 to claim 20 are executed.
24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the decoding method according to any one of claims 1 to 20.
25. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 20 are performed.
Citation Information
Patent Citations
Image decoding and encoding methods and apparatuses, and device thereof
WO2019205871A1
Video encoding method, video decoding method, encoder, decoder, and ai accelerator
WO2022116165A1
Coding method, decoding method, coding apparatus, and decoding apparatus
WO2022116207A1
Context-based image coding
WO2023278068A1
