Coding method and electronic equipment

The coding method enhances audio and video compression by using a model identifier in the code stream to update or switch encoding and decoding models synchronously, addressing the limitations of traditional technologies and optimizing performance.

CN119071533APending Publication Date: 2024-12-03HUAWEI TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202310630393.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Traditional audio and video compression technologies lack the ability to update or iterate AI models after coding standards are standardized, hindering long-term evolution and performance optimization.

Method used

A coding method that includes generating a code stream with a model identifier to instruct a second electronic device to obtain model information, enabling synchronous updating or switching of encoding and decoding models, and reducing bit rate overhead through index information or storage location references.

Benefits of technology

Facilitates long-term evolution and optimization of coding standards by allowing synchronized updates or switches of encoding and decoding models, while minimizing bandwidth requirements and reducing complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071533A_ABST
    Figure CN119071533A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a coding method and electronic equipment. The method comprises the following steps: firstly, first electronic equipment acquires a signal to be coded; then, generating a first code stream; the first code stream comprises a second code stream and a model identifier, the second code stream is obtained by encoding a to-be-encoded signal based on an encoding model, the value of the model identifier is a preset identifier value, the preset identifier value is used for indicating the second electronic equipment to obtain model information of a decoding model, and the decoding model corresponds to the encoding model. In this way, after the first code stream is transmitted to the second electronic device, the second electronic device can obtain the model information of the decoding model according to the preset identification value, and synchronous updating of the decoding model of the second electronic device and the encoding model of the first electronic device is achieved. Or realizing synchronous switching between the decoding model used for decoding by the second electronic equipment and the encoding model used for encoding by the first electronic equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding method and an electronic device. Background Art

[0002] With the development of high-definition audio and video and three-dimensional (3D) audio and video, people have higher and higher demands on audio and video quality, which has led to a rapid growth in the amount of audio and video data in media applications. This has also made audio and video compression an indispensable part of media applications such as media communications and media broadcasting.

[0003] Traditional audio and video compression technologies are based on the basic principles of signal processing, using the correlation of signals in time and space to compress the original audio and video data, reducing the amount of data and making it easier to transmit or store. With the continuous development and maturity of artificial intelligence (AI) technology, AI technology has also been introduced into the field of audio and video encoding and decoding, bringing about improvements in encoding performance.

[0004] Usually, after the coding standard is standardized, the AI ​​models involved will generally not be updated or iterated, which will affect the long-term evolution and performance optimization of the coding standard. Summary of the invention

[0005] In view of this, the present application provides a coding method and an electronic device.

[0006] In a first aspect, an embodiment of the present application provides a coding method applied to a first electronic device, the method comprising: first, obtaining a signal to be encoded; then, generating a first code stream, the first code stream comprising a second code stream and a model identifier, the second code stream being obtained by encoding the signal to be encoded based on the coding model, the value of the model identifier being a preset identifier value, the preset identifier value being used to instruct the second electronic device to obtain model information of a decoding model, the decoding model corresponding to the coding model.

[0007] In this way, the first electronic device instructs the second electronic device to obtain the model information of the decoding model by setting the value of the model identifier in the first code stream to the preset identifier value; after the first code stream is transmitted to the second electronic device, the second electronic device can obtain the model information of the decoding model according to the preset identifier value, and realize the synchronous update of the decoding model of the second electronic device and the encoding model of the first electronic device, or realize the synchronous switching of the decoding model used for decoding by the second electronic device and the encoding model used for encoding by the first electronic device. Thus, it is conducive to the long-term evolution of the coding standard and the optimization of coding performance.

[0008] Exemplarily, both the encoding model and the decoding model can be implemented by a neural network; therefore, the encoding model can also be called an AI encoding model, and the decoding model can also be called an AI decoding model.

[0009] In one possible approach, the encoding model and the decoding model are two independent models; in this case, the encoding model and the decoding model can be jointly trained. In one possible approach, the encoding model and the decoding model are two parts of a model.

[0010] It should be understood that when the encoding model and the decoding model are two parts of a model, the preset identification value can be used to instruct the second electronic device to obtain the model information of the encoding model and the model information of the decoding model; in this way, the second electronic device can reconstruct the encoding model according to the model information of the encoding model, and reconstruct the decoding model according to the model information of the decoding model. This application is described by taking the encoding model and the decoding model as two independent models as an example.

[0011] In one possible way, a new identifier (i.e., flag) or a new field can be added to the first bitstream as a model identifier. In one possible way, an existing field in the first bitstream can be reused and the existing field can be used as a model identifier, and this application does not limit this. When an existing field in the first bitstream is reused and the existing field is used as a model identifier, this application does not need to change the decoding syntax, that is, the model update or switch is implemented under the condition of achieving backward compatibility.

[0012] Exemplarily, the signal to be encoded may include but is not limited to: an audio signal, an image, and a video signal.

[0013] According to the first aspect, the first code stream also includes a third code stream, and the third code stream is generated based on the model information of the decoding model. The preset identification value can be used to indicate that the first code stream contains the third code stream. In this case, the preset identification value can be used to instruct the second electronic device to obtain the model information of the decoding model from the first code stream. In this way, the first electronic device can transmit the model information of the decoding model to the second electronic device, so that the second electronic device can reconstruct the decoding model for decoding based on the model information of the decoding model.

[0014] According to the first aspect, or any implementation of the first aspect above, the method further includes: generating a third code stream based on model information of the decoding model; generating the first code stream, including: encapsulating the second code stream and the third code stream to obtain the first code stream.

[0015] According to the first aspect, or any implementation of the first aspect above, the first code stream also includes index information corresponding to the model information of the decoding model. The preset identification value is used to indicate that the first code stream includes index information corresponding to the model information of the decoding model. In this case, the preset identification value can be used to instruct the second electronic device to obtain the model information of the decoding model from the third electronic device according to the index information parsed from the first code stream.

[0016] In this way, the first electronic device only needs to transmit the index information corresponding to the model information of the decoding model. Compared with transmitting the model information of the decoding model, the bit rate overhead is smaller and the bandwidth requirement can be reduced.

[0017] Exemplarily, the index information may be a Uniform Resource Locator (URL), which is not limited in the present application.

[0018] According to the first aspect, or any implementation of the first aspect, generating a first bitstream includes: obtaining the first bitstream based on the second bitstream and index information corresponding to model information of a decoding model.

[0019] According to the first aspect, or any implementation of the first aspect above, the preset identification value is also used to indicate that the model information of the decoding model is stored in a preset storage location of the third device. In this case, the preset identification value can be used to instruct the second electronic device to obtain the model information of the decoding model from the preset storage location of the third electronic device; wherein the preset storage location can be a location pre-agreed by the first electronic device and the second electronic device.

[0020] In this way, the first electronic device does not need to transmit index information corresponding to the model information of the decoding model, nor does it need to transmit the model information of the decoding model; thus, the bit rate overhead is smaller and the bandwidth requirement is lower.

[0021] It should be noted that, assuming that the way in which the second electronic device obtains the model information from the first code stream is called the first way; the way in which the second electronic device obtains the model information according to the index information corresponding to the model information obtained from the first code stream is called the second way; the way in which the second electronic device obtains the model information from the preset storage location of the third electronic device is called the third way. In one possible way, the first electronic device and the second electronic device can pre-agree on the way in which the second electronic device obtains the model information of the decoding model. In the encoding and decoding process, when the first electronic device and the second electronic device agree on the first way, the first electronic device can generate the third code stream based on the model information of the decoding model; and encapsulate the second code stream and the third code stream to obtain the first code stream. In the encoding and decoding process, when the first electronic device and the second electronic device agree on the second way, the first electronic device can generate the first code stream based on the index information corresponding to the second code stream and the model information of the decoding model. In the encoding and decoding process, when the first electronic device and the second electronic device agree on the third way, the first electronic device can use the second code stream to generate the first code stream.

[0022] In one possible manner, during the encoding and decoding process, the first electronic device may select one of the first manner, the second manner, and the third manner as the target manner. When the target manner is the first manner, the first electronic device may generate a third code stream based on the model information of the decoding model; and encapsulate the second code stream and the third code stream to obtain the first code stream. When the target manner is the second manner, the first electronic device may generate the first code stream based on the index information corresponding to the second code stream and the model information of the decoding model. When the target manner is the third manner, the first electronic device may use the second code stream to generate the first code stream.

[0023] According to the first aspect, or any implementation of the first aspect above, the model identifier is a field of an elementary stream (ES).

[0024] Exemplarily, the encapsulation format of ES may include multiple fields, and the field of ES may refer to the field included in the encapsulation format of ES. In other words, after data is encapsulated according to the encapsulation format of ES, an ES package may be obtained, and the ES package may include multiple fields, and the field of ES may refer to the field in the ES package.

[0025] According to the first aspect, or any implementation of the first aspect above, when the target code stream type of the first code stream is a code stream type encapsulated according to a target container format or a code stream type encapsulated according to a target transmission protocol, the model identifier is a field of the target container format. In this way, the second electronic device can obtain the value of the model identifier without parsing the ES packet.

[0026] Exemplarily, the target container format includes but is not limited to the transport stream (TS) format, MP4 format, etc., and the present application does not impose any limitation on this.

[0027] Exemplarily, the target container format may include multiple fields, and the fields of the target container format may refer to the fields included in the target container format. In other words, after data is encapsulated according to the target container format, an encapsulation package may be obtained, and the encapsulation package may include multiple fields. The fields of the target container format may refer to the fields in the encapsulation package obtained by encapsulating according to the target container format. For example, when the target container format is TS format / MP4 format, the fields of the target container format may refer to the fields in the TS package.

[0028] Exemplarily, the target transmission protocol may include multiple ones, such as a streaming media-based transmission protocol (HTTP Live Streaming, HLS), a real-time transport protocol (Real-time Transport Protocol, RTP), etc., and the present application does not impose any limitation on this.

[0029] According to the first aspect, or any implementation of the first aspect above, when the target code stream type of the first code stream is a code stream type encapsulated according to the target transmission protocol, the model identifier is a field of the target transmission protocol. In this way, the second electronic device does not need to decapsulate the first code stream according to the target container format or parse the ES packet to obtain the value of the model identifier.

[0030] Exemplarily, the encapsulation format of the target transmission protocol may include multiple fields, and the field of the target transmission protocol may refer to the field included in the encapsulation format of the target transmission protocol. In other words, after data is encapsulated according to the encapsulation format of the target transmission protocol, an encapsulation package may be obtained, and the encapsulation package may include multiple fields, and the field of the target transmission protocol may refer to the field in the encapsulation package obtained by encapsulating according to the target transmission protocol.

[0031] According to the first aspect, or any implementation of the first aspect above, the method further includes: determining whether the decoding model is an updated decoding model; when the decoding model is an updated decoding model, setting the value of the model identifier to a preset identifier value. At this time, the preset identifier value is used to instruct the second electronic device to obtain model information of the updated decoding model. In this case, the encoding model may not be updated, and only the decoding model (such as an optimized decoding model) is updated; at this time, the updated decoding model corresponds to the encoding model.

[0032] In one possible scenario, both the coding model and the decoding model are updated; the method further includes: determining whether both the coding model and the decoding model are updated; encoding the signal to be coded based on the coding model to obtain a second code stream, including: when it is determined that both the coding model and the decoding model are updated, encoding the signal to be coded based on the updated coding model to obtain a second code stream; at this time, the preset identification value is used to instruct the second electronic device to obtain model information of the updated decoding model, and the updated decoding model corresponds to the updated coding model.

[0033] In one possible scenario, the first electronic device includes multiple encoding models and multiple decoding models (the encoding models correspond one-to-one to the decoding models), and the method also includes: determining whether to switch the encoding model used for encoding; when it is determined that the encoding model used for encoding has been switched, encoding the to-be-encoded signal based on the switched encoding model to obtain a second code stream; at this time, the preset identification value is used to instruct the second electronic device to obtain model information of the switched decoding model, and the switched decoding model corresponds to the switched encoding model.

[0034] It should be understood that, regardless of whether the encoding model and the decoding model in the first electronic device are updated (or regardless of whether the decoding model is updated), and regardless of whether the encoder in the first electronic device switches the encoding model used for encoding, the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value. In this case, the preset identifier value can be used to characterize the type of the decoding model (or encoding model); correspondingly, after obtaining the preset identifier value, the second electronic device determines whether the second electronic device has a decoding model of the type corresponding to the preset identifier value; when there is no decoding model of the type corresponding to the preset identifier value, the model information of the decoding model is obtained.

[0035] According to the first aspect, or any implementation of the first aspect, encapsulating the second code stream and the third code stream to obtain the first code stream includes: encapsulating the second code stream and the third code stream according to the target code stream type to obtain the first code stream.

[0036] It should be noted that, in one possible manner, the first electronic device and the second electronic device may pre-agree on the target code stream type, so that the first electronic device encapsulates the second code stream and the third code stream according to the target code stream type, and the second electronic device decapsulates the first code stream according to the target code stream type.

[0037] In one possible manner, during the encoding process, the first electronic device can select a code stream type from multiple code stream types as the target code stream type as required; correspondingly, the second electronic device identifies the code stream type of the first code stream (i.e., the target code stream type), and then decapsulates the first code stream according to the target code stream type.

[0038] According to the first aspect, or any implementation of the first aspect above, the target code stream type is an ES type, and the second code stream and the third code stream are encapsulated according to the target code stream type to obtain the first code stream, including: encapsulating the third code stream and the second code stream into a first ES packet, and using the first ES packet as the first code stream. That is, the first code stream is an ES stream, and the second code stream and the third code stream are encapsulated in the same ES packet for transmission. In this way, after obtaining the ES packet, the ES packet can be directly transmitted without encapsulating the ES packet according to the target container format or the target transmission protocol, which can reduce the bit rate overhead and reduce the bandwidth requirement. In addition, the second code stream and the third code stream are encapsulated in the same ES packet, and the second code stream and the third code stream can be sent on the same channel, which can reduce the transmission complexity.

[0039] According to the first aspect, or any implementation of the first aspect above, the target bitstream type is a bitstream type encapsulated according to the target container format, and encapsulating the second bitstream and the third bitstream according to the target bitstream type to obtain the first bitstream includes: encapsulating the third bitstream and the second bitstream into a second ES packet; and encapsulating the second ES packet according to the target container format to obtain the first bitstream. The ES packet may be divided into a plurality of small packets and then encapsulated according to the target container format, which is beneficial for the bitstream distribution device to perform bitstream distribution and audio and video synchronization control.

[0040] Exemplarily, when the target container format is TS format, the first code stream is a TS stream, the second code stream and the third code stream can be encapsulated in the same ES packet, and the ES packet is encapsulated as a TS stream for transmission. When the target container format is MP4 format, the first code stream is an MP4 stream, the second code stream and the third code stream can be encapsulated in the same ES packet, and the ES packet is encapsulated as a TS stream of MP4.

[0041] According to the first aspect, or any implementation of the first aspect above, the target code stream type is a code stream type encapsulated according to the target container format, and the second code stream and the third code stream are encapsulated according to the target code stream type to obtain the first code stream, including: encapsulating the third code stream into a third ES packet, and using the second code stream as a fourth ES packet; encapsulating the third ES packet and the fourth ES packet according to the target container format, and setting the type of the third ES packet to a preset type and setting the type of the fourth ES packet to a media type to obtain the first code stream. In other words, the second code stream and the third code stream are encapsulated into different ES packets for transmission. In this way, compared with encapsulating the second code stream and the third code stream in the same ES packet, the second electronic device does not need to split the ES packet to parse out the second code stream and the third code stream, thereby reducing the complexity of the parsing process.

[0042] Exemplarily, the media type may include, but is not limited to: audio type, video type, and image type.

[0043] According to the first aspect, or any implementation of the first aspect above, the preset type is a metadata type, that is, the third code stream is encapsulated as metadata.

[0044] According to the first aspect, or any implementation of the first aspect above, the target code stream type is a code stream type encapsulated according to the target container format, and the second code stream and the third code stream are encapsulated according to the target code stream type to obtain the first code stream, including: encapsulating the third code stream into a fifth ES packet, and using the second code stream as a sixth ES packet; encapsulating the fifth ES packet according to the target container format and setting the type of the fifth ES packet to a preset type to obtain a fourth code stream; encapsulating the sixth ES packet according to the target container format and setting the type of the sixth ES packet to a media type to obtain a fifth code stream; combining the fourth code stream and the fifth code stream to obtain the first code stream. That is, the second code stream and the third code stream are respectively encapsulated according to the target container format, for example, the second code stream and the third code stream are encapsulated into two TS streams. In this way, compared with encapsulating the second code stream and the third code stream in the same TS stream, the second electronic device does not need to split the TS stream to parse out the second code stream and the third code stream, thereby reducing the complexity of the parsing process.

[0045] It should be noted that combining the fourth code stream and the fifth code stream may be understood as combining only the fourth code stream and the fifth code stream.

[0046] According to the first aspect, or any implementation of the first aspect above, the target code stream type is a code stream type encapsulated according to the target transmission protocol, and the second code stream and the third code stream are encapsulated according to the target code stream type to obtain the first code stream, including: encapsulating the third code stream into the seventh ES packet, and the second code stream as the eighth ES packet; encapsulating the seventh ES packet according to the target container format and setting the type of the seventh ES packet to the preset type to obtain the sixth code stream; encapsulating the eighth ES packet according to the target container format and setting the type of the eighth ES packet to the media type to obtain the seventh code stream; encapsulating the sixth code stream and the seventh code stream according to the target transmission protocol to obtain the first code stream. That is, after encapsulating the second code stream and the third code stream respectively according to the target container format, they are encapsulated according to the target transmission protocol. Among them, the second code stream with different reconstruction accuracy can be generated (which can be understood as the accuracy of the reconstructed signal obtained by decoding the second code stream); in this way, it is beneficial for the code stream distribution device to distribute the first code stream containing the second code stream with different reconstruction accuracy according to the bandwidth, so as to ensure the smoothness of the playback of the reconstructed signal.

[0047] It should be noted that the first code stream obtained by combining the fourth code stream and the fifth code stream only includes the fourth code stream and the fifth code stream; and the first code stream obtained by encapsulating the fourth code stream and the fifth code stream according to the target transmission protocol, in addition to including the sixth code stream and the seventh code stream, also includes the fields included in the encapsulation format of the target transmission protocol.

[0048] According to the first aspect, or any implementation of the first aspect above, the first code stream further includes an index file, the index file includes index information of the sixth code stream and index information of the seventh code stream, and the index information of the sixth code stream in the index file is located before the index information of the seventh code stream. In this way, the second electronic device can first obtain the model information for reconstructing the decoding model, which can shorten the delay of decoding the second code stream corresponding to each frame signal.

[0049] Exemplarily, the target code stream type is a code stream type encapsulated according to HLS, and the first code stream may be HLS.

[0050] According to the first aspect, or any implementation of the first aspect above, the first bitstream further includes an index file, and the attribute information of the initialization tag in the index file includes the index information of the sixth bitstream. In this way, it is ensured that the decoder can decode the second bitstream after determining that the reconstruction of the decoding model is completed and reading the decoding model (i.e., initialization of the decoder).

[0051] Exemplarily, the target code stream type is a code stream type encapsulated according to HLS, and the first code stream may be HLS.

[0052] According to the first aspect, or any implementation of the first aspect above, the signal to be encoded is M frames, the second code stream is M, the model information includes N information groups, the third code stream includes N sub-code streams, the N information groups correspond to the N sub-code streams one by one, the first ES packet includes N ninth ES packets and MN tenth ES packets, M and N are positive integers, and M is greater than or equal to N; encapsulating the third code stream and the second code stream into the first ES packet includes: encapsulating the N sub-code streams and the N second code streams into N ninth ES packets, and using the other MN second code streams as MN tenth ES packets; wherein the ninth ES packet includes one sub-code stream and one second code stream, and the tenth ES packet includes one second code stream.

[0053] Among them, a second code stream is obtained by encoding a frame of the signal to be encoded based on the encoding model. N can be determined according to the data volume of the model information (or the encoded data of the model information), the network bandwidth and the encoded data of the audio signal; in this way, the smoothness of the reconstructed signal played by the second electronic device can be guaranteed.

[0054] According to the first aspect, or any implementation of the first aspect above, the first code stream further includes an index file, and the index file includes index information corresponding to the model information of the decoding model.

[0055] Exemplarily, the target code stream type is a code stream type encapsulated according to HLS, and the first code stream may be HLS.

[0056] According to the first aspect, or any implementation of the first aspect above, obtaining the first bitstream based on the second bitstream and the index information corresponding to the model information of the decoding model includes: generating an eighth bitstream based on the index information corresponding to the model information of the decoding model; and encapsulating the second bitstream and the eighth bitstream to obtain the first bitstream. This is similar to the method of encapsulating the second bitstream and the third bitstream to obtain the first bitstream, and will not be repeated here.

[0057] According to the first aspect, or any implementation of the first aspect above, the third bitstream includes description information of the model information and the model information, or the third bitstream includes description information of the model information and encoded data of the model information.

[0058] According to the first aspect, or any implementation of the first aspect above, the model information includes model structure information or model parameters.

[0059] The model structure information may refer to the network structure information of the neural network, and the network structure information may include the connection relationship between network layers. The decoding model may include multiple types of network layers (e.g., convolutional layer, fully connected layer, RNN (Recurrent Neural Network) layer, activation layer, etc.), and each type of network layer may be one or more.

[0060] Exemplarily, the model parameters may refer to the network parameters of the neural network, and the network parameters may refer to the weight matrix and bias matrix corresponding to the network layer; wherein the weight matrix is ​​composed of weight vectors of multiple network layers, and the bias matrix is ​​composed of bias vectors of multiple network layers. Each network layer may be composed of one or more neurons, and a value in the weight vector of the network layer represents the weight value of a neuron in the network layer, and a value in the bias vector of the network layer represents the bias value of a neuron in the network layer.

[0061] In one possible manner, the model information of the decoding model may be a model file of the decoding model, that is, including connections between all network layers of the decoding model and network parameters of all network layers.

[0062] In one possible manner, the model information of the decoding model may be connections between updated network layers of the decoding model and / or network parameters of updated network layers.

[0063] According to the first aspect, or any implementation of the first aspect above, the preset identification value is also used to characterize the type of the decoding model. In this way, it is convenient for the second electronic device to know whether it is necessary to obtain model information of the decoding model and perform model reconstruction.

[0064] In a second aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a first code stream, the first code stream including a model identifier and a second code stream, the second code stream being generated based on an encoded signal; when the value of the model identifier is a preset identifier value, obtaining model information; then, reconstructing the model based on the model information to obtain a decoding model; thereafter, decoding the second code stream based on the decoding model to obtain a reconstructed signal.

[0065] The decoding method can be applied to the second electronic device, so that when the second electronic device determines that the model identifier is a preset identifier value, it obtains the model information and reconstructs the model, thereby realizing synchronous updating of the decoding model and the encoding model of the first electronic device, or realizing synchronous switching of the decoding model used for decoding and the encoding model used for encoding by the first electronic device. Thus, it is beneficial to the long-term evolution of the coding standard and the optimization of coding performance.

[0066] Exemplarily, the reconstructed signal may include, but is not limited to, a reconstructed audio signal, a reconstructed image, or a reconstructed video signal.

[0067] According to the second aspect, when the value of the model identifier is a preset identifier value, the first code stream also includes a third code stream, and the third code stream is generated based on the model information of the decoding model; obtaining the model information includes: parsing the third code stream from the first code stream; and obtaining the model information of the decoding model based on the third code stream.

[0068] According to the second aspect, or any implementation method of the above second aspect, when the value of the model identifier is a preset identifier value, the first code stream also includes index information corresponding to the model information of the decoding model; obtaining the model information includes: parsing the index information corresponding to the model information of the decoding model from the first code stream; and obtaining the model information of the decoding model based on the index information corresponding to the model information of the decoding model.

[0069] According to the second aspect, or any implementation method of the second aspect above, the preset identification value is used to represent the model information of the decoding model stored in the preset storage location of the remote electronic device; obtaining the model information includes: obtaining the model information of the decoding model from the preset storage location in the remote electronic device.

[0070] The remote electronic device may also refer to the third electronic device mentioned above.

[0071] According to the second aspect, or any implementation of the second aspect above, the model identifier is a field of ES; the method further includes: parsing the field corresponding to the model identifier in the first code stream to obtain the value of the model identifier.

[0072] According to the second aspect, or any implementation manner of the above second aspect, the target codestream type of the first codestream is a codestream type encapsulated according to a target container format or a codestream type encapsulated according to a target transmission protocol, and the model identifier is a field of the target container format; the method further includes: decapsulating the first codestream according to the target container format to obtain a field corresponding to the model identifier; and parsing the field corresponding to the model identifier to obtain a value of the model identifier.

[0073] According to the second aspect, or any implementation manner of the above second aspect, the target code stream type of the first code stream is a code stream type encapsulated according to the target transmission protocol, and the model identifier is a field of the target transmission protocol; the method further includes: decapsulating the first code stream according to the target transmission protocol to obtain a field corresponding to the model identifier; and parsing the field corresponding to the model identifier to obtain a value of the model identifier.

[0074] According to the second aspect, or any implementation of the second aspect above, the model identifier is used to characterize the type of the decoding model, and the method also includes: when the value of the model identifier is a preset identifier value, determining whether there is a decoding model of the type corresponding to the preset identifier value; reconstructing the model based on the model information to obtain the decoding model, including: when there is no decoding model of the type corresponding to the preset identifier value, reconstructing the model based on the model information to obtain a decoding model of the type corresponding to the preset identifier value.

[0075] According to the second aspect, or any implementation of the second aspect, parsing the third bitstream from the second bitstream includes: parsing the first bitstream according to the target bitstream type of the first bitstream to obtain the third bitstream.

[0076] According to the second aspect, or any implementation of the second aspect, the target bitstream type is an ES type, the first bitstream further includes a first ES packet, the first ES packet includes a second bitstream, a third bitstream, and a model identifier; parsing the first bitstream according to the target bitstream type of the first bitstream to obtain the third bitstream includes: reading the third bitstream from the first ES packet; the method also includes: reading the second bitstream from the first ES packet.

[0077] Exemplarily, the present application does not limit the execution order of parsing the second code stream from the first code stream and parsing the value of the model identifier from the first code stream; nor does it limit the execution order of parsing the second code stream from the first code stream and parsing the third code stream from the first code stream.

[0078] According to the second aspect, or any implementation of the second aspect, the target code stream type is a code stream type encapsulated according to a target container format; parsing the first code stream according to the target code stream type of the first code stream to obtain a third code stream includes: decapsulating the first code stream according to the target container format to obtain a second ES packet; reading the third code stream from the second ES packet; the method further includes: reading the second code stream from the second ES packet.

[0079] According to the second aspect, or any implementation of the second aspect above, the target codestream type is a codestream type encapsulated according to a target container format; parsing the first codestream according to the target codestream type of the first codestream to obtain a third codestream includes: decapsulating the first codestream according to the target container format to obtain a third ES packet, a fourth ES packet, a type of the third ES packet, and a type of the fourth ES packet; reading the third codestream from the third ES packet; wherein the type of the third ES packet is a preset type; the method further includes: reading the second codestream from the fourth ES packet; wherein the type of the fourth ES packet is a media type.

[0080] According to the second aspect, or any implementation of the second aspect, the preset type is a metadata type.

[0081] According to the second aspect, or any implementation of the second aspect, the target codestream type is a codestream type encapsulated according to a target container format, the first codestream further includes a fourth codestream and a fifth codestream, the fifth codestream includes the second codestream, the fourth codestream includes the third codestream, and the model identifier is located in the fourth codestream or the fifth codestream; parsing the first codestream according to the target codestream type of the first codestream to obtain the third codestream includes: decapsulating the fourth codestream according to the target container format to obtain a fifth ES packet and a type of the fifth ES packet; reading the third codestream from the fifth ES packet, wherein the type of the fifth ES packet is a preset type; the method further includes: decapsulating the fifth codestream according to the target container format to obtain a sixth ES packet and a type of the sixth ES packet; reading the second codestream from the sixth ES packet, wherein the type of the sixth ES packet is a media type.

[0082] According to the second aspect, or any implementation of the second aspect, the target code stream type is a code stream type encapsulated according to a target transmission protocol, the first code stream further includes a sixth code stream and a seventh code stream, the seventh code stream includes the second code stream, the sixth code stream includes the third code stream, and the model identifier is located in the sixth code stream or the seventh code stream; parsing the first code stream according to the target code stream type of the first code stream to obtain the third code stream includes: decapsulating the first code stream according to the target transmission protocol to obtain the sixth code stream; decapsulating the sixth code stream according to the target container format to obtain a seventh ES packet and a type of the seventh ES packet; reading the third code stream from the seventh ES packet, wherein the type of the seventh ES packet is a preset type; the method further includes: decapsulating the seventh code stream decapsulated from the first code stream according to the target container format to obtain an eighth ES packet and a type of the eighth ES packet; reading the second code stream from the eighth ES packet, wherein the type of the eighth ES packet is a media type.

[0083] According to the second aspect, or any implementation of the second aspect, the first code stream further includes an index file, the index file includes index information of the sixth code stream and index information of the seventh code stream, and the index information of the sixth code stream is located before the index information of the seventh code stream; decapsulating the first code stream according to the target transmission protocol to obtain the sixth code stream includes: decapsulating the first code stream according to the target transmission protocol to obtain the index file and multiple code streams; reading the index information of the sixth code stream from the index file, and determining the sixth code stream from the multiple code streams according to the index information of the sixth code stream.

[0084] According to the second aspect, or any implementation of the second aspect, the first code stream further includes an index file, and the attribute information of the initialization tag in the index file includes the index information of the sixth code stream; decapsulating the first code stream according to the target transmission protocol to obtain the sixth code stream includes: decapsulating the first code stream according to the target transmission protocol to obtain the index file and the multiple code streams; reading the index information of the sixth code stream from the attribute information of the initialization tag in the index file, and determining the sixth code stream from the multiple code streams according to the index information of the sixth code stream.

[0085] According to the second aspect, or any implementation of the second aspect above, the reconstructed signal is M frames, the number of first ES packets is M, the M first ES packets include N ninth ES packets and MN tenth ES packets, the ninth ES packet includes a substream and a second stream, the tenth ES packet includes a second stream, M and N are positive integers, and M is greater than or equal to N; reading the third stream from the first ES packet includes: reading N substreams from N ninth ES packets; splicing the N substreams to obtain the third stream; reading the second stream from the first ES packet includes: reading N second streams from N ninth ES packets, and using MN tenth ES packets as MN second streams.

[0086] According to the second aspect, or any implementation of the second aspect above, the first code stream also includes an index file, and the index file includes index information corresponding to the model information of the decoding model; parsing the index information corresponding to the model information of the decoding model from the first code stream includes: decapsulating the first code stream according to the target transmission format to obtain the index file; reading the index information corresponding to the model information of the decoding model from the index file.

[0087] According to the second aspect, or any implementation of the second aspect above, determining the model information based on the third bit stream includes: parsing the model information from the third bit stream; or parsing the encoded data of the model information from the third bit stream, and decoding the encoded data of the model information to obtain the model information.

[0088] According to the second aspect, or any implementation of the second aspect, the model information includes model structure information or model parameters.

[0089] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0090] In a third aspect, the present application provides an encoding device applied to a first electronic device, the device comprising:

[0091] A signal acquisition module, used to acquire a signal to be encoded;

[0092] The encoding module is used to generate a first code stream; wherein the first code stream includes a second code stream and a model identifier, the second code stream is obtained by encoding the information to be encoded based on the encoding model, the value of the model identifier is a preset identifier value, and the preset identifier value is used to instruct the second electronic device to obtain model information of a decoding model, and the decoding model corresponds to the encoding model.

[0093] It should be understood that the encoding device can execute the steps of the above-mentioned first aspect and any one of the implementation methods of the first aspect, which will not be repeated here.

[0094] In a fourth aspect, the present application provides a decoding device, the device comprising:

[0095] A code stream receiving module, configured to receive a first code stream, the first code stream including a model identifier and a second code stream, the second code stream being generated based on the encoded signal;

[0096] A decapsulation module, used for obtaining model information when the value of the model identifier is a preset identifier value;

[0097] The decapsulation module is also used to reconstruct the model based on the model information to obtain a decoding model;

[0098] The decoder is used to decode the second code stream based on the decoding model to obtain a reconstructed signal.

[0099] Exemplarily, the decoding device may be applied to a second electronic device.

[0100] It should be understood that the decoding device can execute the steps of the above-mentioned second aspect and any one of the implementation methods of the second aspect, which will not be repeated here.

[0101] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the encoding method executed by the first electronic device in the first aspect or any possible implementation of the first aspect.

[0102] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0103] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the decoding method executed by the second electronic device in the second aspect or any possible implementation of the second aspect.

[0104] The sixth aspect and any implementation of the sixth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0105] In the seventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the steps of the encoding method in the first aspect or any possible implementation of the first aspect are executed.

[0106] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0107] In an eighth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the decoding method in the second aspect or any possible implementation of the second aspect are executed.

[0108] The eighth aspect and any implementation of the eighth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0109] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0110] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0111] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0112] The tenth aspect and any implementation of the tenth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0113] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0114] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0115] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0116] The twelfth aspect and any implementation of the twelfth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0117] In a thirteenth aspect, an embodiment of the present application provides a method for generating a bitstream, which can be generated according to the encoding method of the first aspect and any implementation method of the first aspect.

[0118] The thirteenth aspect and any implementation of the thirteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0119] In a fourteenth aspect, an embodiment of the present application provides a device for storing a code stream, the device comprising: a receiver and at least one storage medium, the receiver being used to receive the code stream; the at least one storage medium being used to store the code stream; the code stream is a first code stream generated according to the first aspect and any one implementation method of the first aspect.

[0120] The fourteenth aspect and any implementation of the fourteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0121] In a fifteenth aspect, an embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium is used to store the code stream, the code stream is a first code stream generated according to the first aspect and any one of the implementation methods of the first aspect; the transmitter is used to obtain the code stream from the storage medium and send the code stream to the terminal side device through the transmission medium.

[0122] The fifteenth aspect and any implementation of the fifteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0123] In a sixteenth aspect, an embodiment of the present application provides a system for distributing a code stream, the system comprising: at least one storage medium, used to store at least one code stream, the at least one code stream is a first code stream generated according to the first aspect and any one of the implementation methods of the first aspect, a streaming media device, used to obtain a target code stream from the at least one storage medium and send the target code stream to a terminal side device, wherein the streaming media device includes a content server or a content distribution server.

[0124] The sixteenth aspect and any implementation of the sixteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixteenth aspect and any implementation of the sixteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0125] In a seventeenth aspect, an embodiment of the present application provides a compression system, the compression system comprising an encoding end and a decoding end, wherein:

[0126] The encoding end is used to obtain a signal to be encoded; generate a first code stream, the first code stream includes a second code stream and a model identifier, the second code stream is obtained by encoding the signal to be encoded based on the encoding model, the value of the model identifier is a preset identifier value, and the preset identifier value is used to instruct the second electronic device to obtain model information of a decoding model, and the decoding model corresponds to the encoding model;

[0127] The decoding end is used to receive the first code stream; when the value of the model identifier is a preset identifier value, obtain the model information; reconstruct the model based on the model information to obtain a decoding model; and decode the second code stream based on the decoding model to obtain a reconstructed signal.

[0128] The seventeenth aspect and any implementation of the seventeenth aspect correspond to the first aspect and any implementation of the first aspect, respectively, and correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the seventeenth aspect and any implementation of the seventeenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0129] Figure 1A is a schematic diagram of a system framework shown as an example;

[0130] Figure 1B is a schematic diagram of a system framework shown as an example;

[0131] Figure 1C is a schematic diagram of the structure of an encoding and decoding module shown as an example;

[0132] Figure 2 is a schematic diagram of an exemplary encoding process;

[0133] Figure 3 is a schematic diagram of an exemplary decoding process;

[0134] Figure 4 A schematic diagram of an audio encoding process is shown as an example;

[0135] Figure 5 is a schematic diagram of an audio decoding process shown as an example;

[0136] Fig. 6A A schematic diagram of an audio encoding process is shown as an example;

[0137] Figure 6B is a schematic diagram of an exemplary model information code stream;

[0138] Figure 6C is a schematic diagram of the structure of a first code stream shown as an example;

[0139] Figure 7 is a schematic diagram of an audio decoding process shown as an example;

[0140] Figure 8 A schematic diagram of an audio encoding process is shown as an example;

[0141] Fig. 9 is a schematic diagram of an audio decoding process shown as an example;

[0142] Fig. 10A A schematic diagram of an audio encoding process is shown as an example;

[0143] Fig. 10B is a schematic diagram of the structure of a first code stream shown as an example;

[0144] Fig.11 is a schematic diagram of an audio decoding process shown as an example;

[0145] Fig. 12A A schematic diagram of an audio encoding process is shown as an example;

[0146] Fig. 12B The figure is a schematic diagram of an exemplary metadata packaging process;

[0147] Fig.13 is a schematic diagram of an audio decoding process shown as an example;

[0148] Fig.14A A schematic diagram of an audio encoding process is shown as an example;

[0149] Fig. 14B is a schematic diagram of the structure of a first code stream shown as an example;

[0150] Fig.15 is a schematic diagram of an audio decoding process shown as an example;

[0151] Fig.16A A schematic diagram of an audio encoding process is shown as an example;

[0152] Fig. 16B is a schematic diagram of the structure of a first code stream shown as an example;

[0153] Fig.17 is a schematic diagram of an audio decoding process shown as an example;

[0154] Fig.18 is a schematic diagram of an exemplary encoding device;

[0155] Fig.19 is a schematic diagram of an exemplary decoding device;

[0156] Fig. 20 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0157] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0158] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0159] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0160] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0161] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0162] Figure 1A FIG. 1 is a schematic diagram of an exemplary system framework. Figure 1A The figure shows the coding and decoding framework of the media signal, wherein the media signal may include but is not limited to: audio signal, video signal and image.

[0163] Reference Figure 1A , exemplarily, the first electronic device may include a first acquisition module, a first encoding module, a first channel encoding module, a first channel decoding module, a first decoding module and a first playback module. It should be understood that the first electronic device may include Figure 1A The present application is not limited to more or fewer modules as shown.

[0164] Reference Figure 1A , exemplarily, the second electronic device may include a second acquisition module, a second encoding module, a second channel encoding module, a second channel decoding module, a second decoding module, and a second playback module. It should be understood that the second electronic device may include Figure 1A The present application is not limited to more or fewer modules as shown.

[0165] Exemplarily, the process of encoding and transmitting a media signal from a first electronic device to a second electronic device, and decoding and playing the signal by the second electronic device can be as follows: the first acquisition module can perform acquisition and output the media signal to the first encoding module. Then, the first encoding module can encode the media signal and output the code stream to the first channel encoding module. Afterwards, the first channel encoding module can perform channel encoding on the code stream, and transmit the channel-encoded code stream to the second electronic device through a wireless or wired network communication device. Then, the second channel decoding module of the second electronic device can perform channel decoding on the received data to obtain the code stream and output the code stream to the second decoding module. Then, the second decoding module can decode the code stream to obtain a reconstructed signal; then the reconstructed signal is output to the second playback module, and the second playback module plays the signal.

[0166] It should be understood that the process of the second electronic device encoding and transmitting the media signal to the first electronic device, which is decoded and played by the first electronic device, is similar to the process of the first electronic device transmitting the media signal to the second electronic device, which is played by the second electronic device, and will not be repeated here.

[0167] It should be understood that the first electronic device and the second electronic device can be directly connected without communicating through a wireless or wired network communication device, and this application does not limit this.

[0168] Exemplarily, the first electronic device and the second electronic device may include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.

[0169] For example, Figure 1A The codec framework can be applied to any scenario that requires codec, such as VR (Virtual Reality) / AR (Augmented Reality) scenarios. In VR and AR scenarios: In one possible way, the first electronic device is a server and the second electronic device is a VR / AR device; in one possible way, the second electronic device is a server and the first electronic device is a VR / AR device.

[0170] Exemplarily, the first electronic device may be referred to as an encoding end, and the second electronic device may be referred to as a decoding end; or, the second electronic device may be referred to as an encoding end, and the first electronic device may be referred to as a decoding end.

[0171] Figure 1B FIG. 1 is a schematic diagram of an exemplary system framework. Figure 1B Shown is a transcoded frame of a media signal.

[0172] Reference Figure 1B (1) Exemplarily, a wireless or core network device may include: a channel decoding module, other decoding modules, a coding module (that is, the coding module involved in this application) and a channel coding module. The wireless or core network device may be used for transcoding.

[0173] For example, Figure 1B A specific application scenario of (1) may be: when the first electronic device is not provided with an encoding module but only with other encoding modules, and the second electronic device is only provided with a decoding module but not with other decoding modules, in order to enable the second electronic device to decode and play the media signal encoded by the first electronic device using other encoding modules, a wireless or core network device may be used for transcoding.

[0174] Specifically, the first electronic device uses other encoding modules to encode the media signal to obtain code stream 1; and performs channel encoding on code stream 1 and sends it to the wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the code stream 1 obtained by channel decoding to other decoding modules. After that, other decoding modules decode code stream 1 to obtain a media signal and output the media signal to the encoding module. Then, the encoding module can encode the media signal to obtain code stream 2 and output code stream 2 to the channel encoding module, and the channel encoding module performs channel encoding on code stream 2 and sends it to the second electronic device. In this way, the second electronic device can call the decoding module, decode the code stream 2 obtained by channel decoding, and obtain a reconstructed signal; the reconstructed signal can be played later.

[0175] Reference Figure 1B (2) Exemplarily, the wireless or core network device may include: a channel decoding module, a decoding module (that is, the encoding module involved in this application), other encoding modules and a channel encoding module. The wireless or core network device may be used for transcoding.

[0176] For example, Figure 1B The specific application scenario of (2) may be: when the first electronic device is only provided with an encoding module but not other encoding modules, and the second electronic device is not provided with a decoding module but only other decoding modules, in order to enable the second electronic device to decode and play the media signal encoded by the encoding module of the first electronic device, a wireless or core network device may be used for transcoding.

[0177] Specifically, the first electronic device uses an encoding module to encode the media signal to obtain code stream 1; and performs channel encoding on code stream 1 and sends it to a wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the code stream 1 obtained by channel decoding to the decoding module. After that, the decoding module decodes code stream 1 to obtain a media signal and outputs the media signal to other encoding modules. Then, other encoding modules can encode the media signal to obtain code stream 2 and output code stream 2 to the channel encoding module, and the channel encoding module performs channel encoding on code stream 2 and sends it to the second electronic device. In this way, the second electronic device can call other decoding modules to decode code stream 2 obtained by channel decoding to obtain a reconstructed signal; the reconstructed signal can be played later.

[0178] Figure 1C FIG. 1 is a schematic diagram showing the structure of an exemplary encoding and decoding module. Figure 1C In the above, the encoding module may refer to Figure 1A The first encoding module, the second encoding module and the above Figure 1B The encoding module in Figure 1C In the above, the decoding module may refer to Figure 1A The first decoding module, the second decoding module and the above Figure 1B The decoding module in .

[0179] Reference Figure 1C (1), exemplarily, the encoding module may include an encoder and an encapsulation module; wherein the encoder is used to encode the media signal based on the encoding model and output a bitstream; and the encapsulation model is used to encapsulate the bitstream. It should be understood that Figure 1C The coding module in (1) is only an example of the present application. The coding module of the present application may include Figure 1C (1) More modules are shown, and this application does not limit this.

[0180] Reference Figure 1C (2) Exemplarily, the decoding module may include a decoder and a decapsulation module; wherein the decapsulation module is used to decapsulate and parse the received bitstream; the decoder may be used to decode the bitstream based on the decoding model to obtain a reconstructed signal. It should be understood that Figure 1C The decoding module in (2) is only an example of the present application. The decoding module of the present application may include Figure 1C (2) More modules are shown, but this application does not limit this.

[0181] Exemplarily, both the encoding model and the decoding model can be implemented by a neural network; therefore, the encoding model can also be called an AI (Artificial intelligence) encoding model, and the decoding model can also be called an AI decoding model.

[0182] In one possible approach, the encoding model and the decoding model are two independent models; in this case, the encoding model and the decoding model can be jointly trained. In one possible approach, the encoding model and the decoding model are two parts of a model.

[0183] The following Figure 1C The encoding and decoding process of the media signal is explained based on this.

[0184] Figure 2 FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 2 The encoding step in the embodiment may be performed by the first electronic device.

[0185] S201, obtaining a signal to be encoded.

[0186] Exemplarily, the signal to be encoded may be a media signal, wherein the media signal includes but is not limited to an audio signal, an image, and a video signal. The signal to be encoded may be M frames, where M may be a positive integer; when the signal to be encoded is an image, M is equal to 1.

[0187] S202, generating a first code stream, the first code stream including a second code stream and a model identifier, the second code stream is obtained by encoding the signal to be encoded based on the encoding model, the value of the model identifier is a preset identifier value, the preset identifier value is used to instruct the second electronic device to obtain model information of a decoding model, and the decoding model corresponds to the encoding model.

[0188] Exemplarily, the encoder in the encoding module may encode the signal to be encoded based on the encoding model to obtain a second bitstream. It should be noted that the second bitstream refers to the bitstream output by the encoder (in some cases, it may also be called a bare stream); the second bitstream may include the encoded data of the signal to be encoded (or the encoded data of the media signal).

[0189] Exemplarily, the encoder may perform preprocessing, such as transformation, on the signal to be encoded to obtain first data; thereafter, the first data may be quantized and entropy encoded to output a second code stream.

[0190] Then, based on the second code stream, the first code stream is generated. Exemplarily, the encapsulation module in the encoding module may generate the first code stream based on the second code stream.

[0191] In one possible scenario, if the first electronic device only stores one coding model and one decoding model; when the coding model and the decoding model in the first electronic device are updated, the coding module in the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value, and the preset identifier value can be used to instruct (or guide) the second electronic device to obtain the model information of the decoding model. In this way, the decoding model of the second electronic device can be guaranteed to be updated synchronously. (It should be understood that in one possible scenario, the coding model may not be updated, and only the decoding model (such as optimizing the decoding model) is updated; in this case, the updated decoding model corresponds to the coding model. In this case, when the decoding model in the first electronic device is updated, the coding module in the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value. This application is explained by taking the updating of both the coding module and the decoding model as an example.)

[0192] In a possible scenario, if a plurality of coding models and a plurality of decoding models are stored in the first electronic device (the coding models correspond one to one to the decoding models), in the process of encoding the signal to be encoded, the encoder of the first electronic device can switch the coding model used for encoding. After the encoder of the first electronic device switches the coding model used for encoding (for example, switching from coding model 1 to coding model 2), the encoding module in the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value, and the preset identifier value can be used to instruct the second electronic device to obtain the model information of the decoding model. In this way, it can be ensured that the decoder of the second electronic device can switch to the decoding model corresponding to the switched coding model (for example, switching from decoding model 1 to decoding model 2) for decoding.

[0193] In a possible scenario, regardless of whether the encoding model and the decoding model in the first electronic device are updated, and regardless of whether the encoder in the first electronic device switches the encoding model used for encoding, the encoding module in the first electronic device can set the value of the model identifier in the first code stream to a preset identifier value. In this case, the preset identifier value can be used to characterize the type of the decoding model (or encoding model); correspondingly, after obtaining the preset identifier value, the second electronic device determines whether the second electronic device has a decoding model of the type corresponding to the preset identifier value; when there is no decoding model of the type corresponding to the preset identifier value, the model information of the decoding model is obtained.

[0194] It should be understood that when the encoding model and the decoding model are two parts of a model, the preset identification value can be used to instruct the second electronic device to obtain the model information of the encoding model and the model information of the decoding model; in this way, the second electronic device can reconstruct the encoding model according to the model information of the encoding model, and reconstruct the decoding model according to the model information of the decoding model. This application is described by taking the encoding model and the decoding model as two independent models as an example.

[0195] In one possible way, a new identifier (i.e., flag) or a new field can be added to the first bitstream as a model identifier. In one possible way, an existing field in the first bitstream can be reused and the existing field can be used as a model identifier, and this application does not limit this. When an existing field in the first bitstream is reused and the existing field is used as a model identifier, this application does not need to change the decoding syntax, that is, the model update or switch is implemented under the condition of achieving backward compatibility.

[0196] Exemplarily, the first electronic device may include a device for transmitting a code stream; the device for transmitting a code stream may include: a transmitter and at least one storage medium, wherein the at least one storage medium is used to store the above Figure 2 The first code stream generated by the embodiment; the transmitter is used to obtain the first code stream from the storage medium and send the first code stream to the terminal side device through the transmission medium.

[0197] Exemplarily, after the first electronic device generates the first code stream, it can send the first code stream to the system for distributing code streams. The system for distributing code streams may include: at least one storage medium and a streaming media device; the storage medium is used to store the above Figure 2 At least one first code stream generated by the embodiment; a streaming media device, used to obtain a target code stream from at least one storage medium and send the target code stream to a terminal side device, wherein the streaming media device includes a content server or a content distribution server.

[0198] Figure 3 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 3 is with Figure 2 The decoding process corresponding to the encoding process is as follows: Figure 3 The decoding step in the embodiment may be performed by the second electronic device. Exemplarily, the second electronic device may include a device for storing a code stream. The device for storing a code stream may include: a receiver and at least one storage medium, the receiver is used to receive the first code stream; and the at least one storage medium is used to store the first code stream. Then, the first code stream may be decoded with reference to S302 to S304.

[0199] S301, receiving a first code stream.

[0200] Exemplarily, after the first electronic device sends the first code stream, the second electronic device can receive the first code stream.

[0201] Exemplarily, the decapsulation module in the decoding module of the second electronic device can parse the value of the model identifier from the first code stream; then, based on the parsed value of the model identifier, it can be determined whether the first electronic device updates the encoding module and the decoding model, or whether the encoder in the first electronic device switches the encoding model used for encoding.

[0202] S302: When the value of the model identifier is a preset identifier value, obtain model information of the decoding model.

[0203] Exemplarily, when the value of the model identifier is a preset identifier value, it indicates that the first electronic device has updated the encoding module and the encoding model, or that the encoder of the first electronic device has switched the encoding model used for encoding. At this time, the second electronic device can obtain the model information of the decoding model.

[0204] Exemplarily, when the value of the model identifier is another identifier value, it indicates that the first electronic device has not updated the encoding module and encoding model, or that the encoder of the first electronic device has not switched the encoding model used for encoding; at this time, the second electronic device does not need to obtain the model information of the decoding model. In this case, the decapsulation module in the decoding model of the second electronic device can parse the second code stream from the first code stream, and then the decoder can decode the second code stream based on the decoding model of the type corresponding to the other identifier value (or the default decoding model) to obtain a reconstructed signal.

[0205] Exemplarily, if the first electronic device sets the value of the model identifier to a preset identifier value regardless of whether the encoding model and decoding model of the first electronic device are updated, and regardless of whether the encoder of the first electronic device switches the encoding model used for encoding; then when the second electronic device parses the value of the model identifier as the preset identifier value, it can be determined whether the second electronic device has a decoding model of the type corresponding to the preset identifier value. When the second electronic device does not have a decoding model of the type corresponding to the preset identifier value, execute S303 to S304. When the second electronic device has a decoding model of the type corresponding to the preset identifier value, there is no need to execute S303. After executing S304, the decoding model of the type corresponding to the preset identifier value can be directly used to decode the second code stream.

[0206] S303: Reconstruct the model based on the model information to obtain a decoding model.

[0207] Exemplarily, the decapsulation module may reconstruct the model based on the model information to obtain a decoding model.

[0208] S304: Decode the second code stream based on the decoding model to obtain a reconstructed signal.

[0209] Exemplarily, the decapsulation module can also parse the second code stream from the first code stream and send the second code stream to the decoder. It should be understood that the present application does not limit the execution order of parsing the second code stream from the first code stream and obtaining the model information or parsing the model identifier value from the first code stream.

[0210] Next, the decoder can read the reconstructed decoding model (also called decoder initialization), and then decode the second bitstream based on the reconstructed decoding model to obtain a reconstructed signal. Afterwards, the decoding end can play the reconstructed signal.

[0211] Exemplarily, the decoder may perform entropy decoding and inverse quantization on the second bitstream based on the reconstructed decoding model to obtain second data; thereafter, the second data may be post-processed such as inverse transformation to obtain a reconstructed signal. The reconstructed signal may include but is not limited to: a reconstructed audio signal, a reconstructed image, or a reconstructed video signal.

[0212] In one possible manner, when the first electronic device does not update the encoding model and the decoding model, or does not switch the encoding model used for encoding, the first code stream may not include a model identifier; further, when the second electronic device does not parse the model identifier from the first code stream, the default decoding model may be used to decode the second code stream parsed from the first code stream to obtain a reconstructed signal.

[0213] In this way, the first electronic device instructs the second electronic device to obtain the model information of the decoding model by setting the value of the model identifier in the first code stream to a preset identifier value; after the first code stream is transmitted to the second electronic device, the second electronic device can obtain the model information of the decoding model according to the preset identifier value, thereby realizing the synchronous update of the decoding model or the synchronous switching of the decoding model used for decoding and the encoding model used for encoding.

[0214] In a possible manner, the first electronic device may transmit the model information of the decoding model to the second electronic device. Specifically, the third code stream may be generated based on the model information of the decoding model; then, the second code stream and the third code stream are encapsulated to obtain the first code stream; at this time, the preset identification value may be used to indicate that the first code stream contains the third code stream, so that the preset identification value may be used to instruct the second electronic device to obtain the model information of the decoding model from the first code stream.

[0215] The present application is described below by taking the signal to be encoded as an audio signal as an example.

[0216] exist Figures 4 to 17 In the embodiment of the present invention, the signal to be encoded is an audio signal, the second bit stream is an audio bit stream, and the third bit stream is a model information bit stream.

[0217] Figure 4 The figure is a schematic diagram of an exemplary audio encoding process.

[0218] S401, obtaining an audio signal.

[0219] Exemplarily, an audio signal to be encoded may be obtained; wherein the audio signal may be M frames, and M may be a positive integer.

[0220] S402: Encode the audio signal based on the encoding model to obtain an audio code stream.

[0221] Exemplarily, the audio signal may be encoded by an audio encoder based on the encoding model to obtain an audio bitstream. It should be noted that the audio bitstream refers to a bitstream output by the audio encoder; the audio bitstream may include encoded data of the audio signal.

[0222] Exemplarily, an audio encoder may perform audio preprocessing, such as transformation, on an audio signal to obtain first data; thereafter, the first data may be processed such as quantization and entropy coding to output an audio bit stream.

[0223] S403, generating a model information code stream based on the model information of the decoding model; wherein the decoding model corresponds to the encoding model.

[0224] S404, encapsulate the audio code stream and the model information code stream to obtain a first code stream; wherein the first code stream includes a model identifier, the value of the model identifier is a preset identifier value, and the preset identifier value is used to indicate that the first code stream includes the model information code stream.

[0225] In a possible scenario, the first electronic device stores only one coding model and one decoding model; when the coding model and the decoding model in the first electronic device are updated, in order to ensure that the decoding model of the second electronic device is updated synchronously, after the audio encoder encodes the audio signal for the first time based on the updated coding model to obtain the audio code stream, the first electronic device can generate a model information code stream based on the model information of the updated decoding model; then, encapsulate the audio code stream and the model information code stream to obtain the first code stream; and set the value of the model identifier in the first code stream to a preset identifier value (the preset identifier value is used to indicate that the model information code stream is included in the first code stream); then, the first code stream can be sent to the second electronic device. In this way, the model information can be transmitted to the second electronic device, and the second electronic device reconstructs the decoding model according to the model information to implement the update of the decoding model in the second electronic device.

[0226] In a possible scenario, a first electronic device stores multiple coding models and multiple decoding models (coding models correspond to decoding models one by one), and in the process of encoding an audio signal, the first electronic device can switch the coding model used for encoding. After the first electronic device switches the coding model used for encoding (for example, switching from coding model 1 to coding model 2), in order to ensure that the second electronic device can switch to the decoding model corresponding to the switched coding model (for example, switching from decoding model 1 to decoding model 2) for decoding, after the audio encoder encodes the audio signal based on the switched coding model (such as coding model 2), the first electronic device can generate a model information code stream based on the model information of the decoding model corresponding to the switched coding model (such as decoding model 2). Then, encapsulate the audio code stream and the model information code stream to obtain a first code stream; and set the value of the model identifier in the first code stream to a preset identifier value; then, the first code stream can be sent to the second electronic device. In this way, the model information can be transmitted to the second electronic device, and the second electronic device reconstructs the decoding model corresponding to the switched coding model (such as decoding model 2) according to the model information.

[0227] In a possible scenario, regardless of whether the encoding model and decoding model of the first electronic device are updated, and regardless of whether the first electronic device switches the encoding model used for encoding, the first electronic device can generate a model information code stream based on the model information of the decoding model corresponding to the encoding model used by the audio encoder to encode the audio signal; then, encapsulate the audio code stream and the model information code stream to obtain a first code stream; and set the value of the model identifier in the first code stream to a preset identifier value; then, the first code stream can be sent to the second electronic device. In this case, the preset identifier value can also be used to characterize the type of the encoding model (or decoding model); further, when the second electronic device determines that there is no decoding model of the type corresponding to the preset identifier value locally, it reconstructs the decoding model according to the model information.

[0228] It should be understood that the first electronic device can also generate a model information code stream of the coding model based on the model information of the updated coding model (or the switched coding model), encapsulate it into the first code stream, and send it to the second electronic device. In this way, the second electronic device can reconstruct the coding model based on the model information code stream of the coding model; when the second electronic device performs encoding later, it can encode the audio signal based on the updated coding model (or the switched coding model).

[0229] It should be understood that when the encoding model and the decoding model are two parts of a model, the first electronic device can encapsulate the model information code stream of the encoding model, the model information code stream of the decoding model and the audio code stream into a first code stream and send it to the second electronic device, and the second electronic device can reconstruct the encoding model according to the model information code stream of the encoding model, and reconstruct the decoding model according to the model information code stream of the decoding model. This application is described by taking the encoding model and the decoding model as two independent models as an example.

[0230] In one possible manner, an existing field in the first code stream may be used as a model identifier. When the first electronic device does not update the encoding model and the decoding model, or does not switch the encoding model used for encoding, that is, when the first code stream does not contain a model information code stream, the model identifier may be set to another identifier value; in this case, the other identifier value is used to indicate that the first code stream does not contain a model information code stream.

[0231] In addition, when the preset identification value is also used to characterize the type of decoding model (or the type of encoding model), other identification values ​​may also be used to characterize the type of decoding model (or the type of encoding model); wherein the type of decoding model (or encoding model) characterized by the other identification values ​​is different from the type of decoding model (or encoding model) characterized by the preset identification value.

[0232] In one possible manner, an additional field may be added to the first bitstream as a model identifier. When the first electronic device does not update the encoding model and the decoding model, or does not switch the encoding model used for encoding, that is, when the first bitstream does not include a model information bitstream, there is no need to add an additional field.

[0233] Exemplarily, the model information of the decoding model may include model structure information and / or model parameters.

[0234] The model structure information may refer to the network structure information of the neural network, and the network structure information may include the connection relationship between network layers. The decoding model may include multiple types of network layers (e.g., convolutional layer, fully connected layer, RNN (Recurrent Neural Network) layer, activation layer, etc.), and each type of network layer may be one or more.

[0235] Exemplarily, the model parameters may refer to the network parameters of the neural network, and the network parameters may refer to the weight matrix and bias matrix corresponding to the network layer; wherein the weight matrix is ​​composed of weight vectors of multiple network layers, and the bias matrix is ​​composed of bias vectors of multiple network layers. Each network layer may be composed of one or more neurons, and a value in the weight vector of the network layer represents the weight value of a neuron in the network layer, and a value in the bias vector of the network layer represents the bias value of a neuron in the network layer.

[0236] In one possible manner, the model information of the decoding model may be a model file of the decoding model, that is, including connections between all network layers of the decoding model and network parameters of all network layers.

[0237] In one possible manner, the model information of the decoding model may be connections between updated network layers of the decoding model and / or network parameters of updated network layers.

[0238] It should be noted that S404 can be executed by the encapsulation module, and S403 can be executed by other modules in the encoding module except the audio encoder and the encapsulation module.

[0239] Figure 5 FIG. 1 is a schematic diagram of an exemplary audio decoding process. Figure 5 is with Figure 4 The decoding process corresponding to the audio encoding process.

[0240] S501, receiving a first code stream.

[0241] Exemplarily, after the first electronic device sends the first code stream, the second electronic device can receive the first code stream.

[0242] S502: parse the first bitstream to obtain a value of the model identifier.

[0243] Exemplarily, the decapsulation module of the second electronic device can parse the value of the model identifier from the first code stream; then, based on the parsed value of the model identifier, it can be determined whether the first electronic device updates the encoding model and the decoding model, or whether the first electronic device switches the encoding model used for encoding.

[0244] S503: When the value of the model identifier is a preset identifier value, parse the first code stream to obtain a model information code stream.

[0245] Exemplarily, when the value of the model identifier is a preset identifier value, it indicates that the first electronic device has updated the encoding model and the decoding model, or it indicates that the first electronic device has switched the encoding model used for encoding, which means that the first code stream also includes the model information code stream. Furthermore, the decapsulation module of the second electronic device can parse the model information code stream from the first code stream.

[0246] Exemplarily, when the value of the model identifier is other identification values, it indicates that the first electronic device has not updated the encoding model and decoding model, or that the first electronic device has not switched the encoding model used for encoding, that is, the first code stream does not contain the model information code stream. In this case, the second electronic device can parse the audio code stream from the first code stream, and then decode the audio code stream based on the decoding model of the type corresponding to the other identification value (or the default decoding model) to obtain a reconstructed audio signal.

[0247] Exemplarily, when the first electronic device sends the model information code stream of the decoding model corresponding to the coding model used by the audio encoder to encode the audio signal regardless of whether the coding model and the decoding model of the first electronic device are updated, and regardless of whether the first electronic device switches the coding model used for encoding, if the value of the model identifier is a preset identifier value, it can be determined whether there is a decoding model of the type corresponding to the preset identifier value in the second electronic device. When there is no decoding model of the type corresponding to the preset identifier value in the second electronic device, S504 to S506 are executed. When the second electronic device has a decoding model of the type corresponding to the preset identifier value, there is no need to execute S504. After executing S505, the decoding model of the type corresponding to the preset identifier value can be directly used to decode the audio code stream.

[0248] S504: Reconstruct the model based on the model information to obtain a decoding model.

[0249] Exemplarily, the decapsulation module may determine the model information based on the model information code stream, and then reconstruct the model based on the model information to obtain the decoding model. Specifically, the model may be reconstructed based on the model structure information and / or model parameters included in the model information to obtain the decoding model.

[0250] S505: parse the first bit stream to obtain an audio bit stream.

[0251] Exemplarily, the decapsulation module may also parse the first code stream to obtain an audio code stream, and send the audio code stream to the audio decoder. It should be understood that the present application does not limit the execution order of any of steps S505 and S502 to S504.

[0252] S506: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.

[0253] Next, the audio decoder can read the reconstructed decoding model (also called initialization of the audio decoder), and then decode the audio code stream based on the reconstructed decoding model to obtain a reconstructed audio signal. Afterwards, the second electronic device can play the reconstructed audio signal.

[0254] Exemplarily, the audio decoder may perform entropy decoding and inverse quantization on the audio bitstream based on the reconstructed decoding model to obtain second data; thereafter, the second data may be subjected to audio post-processing such as inverse transformation to obtain a reconstructed audio signal.

[0255] In one possible manner, when the first electronic device has not updated the encoding model and the decoding module, or has not switched the encoding model used for encoding, the first bitstream does not contain a model identifier and does not contain a model information bitstream; further, when the second electronic device has not parsed the model identifier from the first bitstream, the default decoding model can be used to decode the audio bitstream parsed from the first bitstream to obtain a reconstructed audio signal.

[0256] Exemplarily, the first electronic device may encapsulate the audio code stream and the model information code stream according to the target code stream type (that is, the type of the expected first code stream) to obtain the first code stream. Correspondingly, the second electronic device may also parse the first code stream according to the target code stream type to obtain the model information code stream, the audio code stream, and the value of the model identifier.

[0257] Exemplarily, the target stream type includes any one of the following: a first type, a second type, or a third type. The first type is an ES type; the second type is a stream type encapsulated according to a target container format (e.g., a TS type, an MP4 type, etc.); and the third type is a stream type encapsulated according to a target transport protocol (e.g., an HLS type, an RTP type, etc.).

[0258] Fig. 6A FIG. 1 is a schematic diagram of an exemplary audio encoding process. Fig. 6A In the embodiment of , the target bitstream type is the first type; correspondingly, the first bitstream can also be called an ES stream. The model information bitstream and the audio bitstream can be encapsulated in the same ES package to obtain the first bitstream.

[0259] S601: Acquire an audio signal.

[0260] Exemplarily, M frames of audio signals may be acquired, where M is a positive integer.

[0261] S602: Determine whether both the encoding model and the decoding model are updated.

[0262] S603: When both the encoding model and the decoding model are updated, the audio signal is encoded based on the updated encoding model to obtain an audio bitstream.

[0263] Exemplarily, the audio encoder may encode M frames of audio signals based on the updated coding model to obtain M audio code streams.

[0264] S604, generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.

[0265] Exemplarily, model information of a decoding model corresponding to an updated coding model (that is, an updated decoding model) may be obtained; the model information of the updated decoding model may be divided into N (N is a positive integer) information groups; then, based on each of the N information groups, a sub-code stream may be generated; and then N sub-code streams may be obtained, that is, the model information code stream may include multiple sub-code streams.

[0266] Exemplarily, when N is equal to 1, that is, the model information is not divided or the model information is divided into an information group; in this case, the model information code stream may include the description information of the model information and the model information; or, the model information code stream may include the description information of the model information and the encoded data of the model information.

[0267] Figure 6B FIG. 1 is a schematic diagram of an exemplary model information code stream. Figure 6B In (1), N=1, the description information of the model information is the length information of the model information, and the model information code stream may include the length information of the model information and the model information.

[0268] Exemplarily, when N is greater than 1, each sub-stream may include description information of an information group and an information group; or, each sub-stream may include description information of an information group and coded data of the information group.

[0269] Reference Figure 6B (2), illustratively, in Figure 6B In (2), N is greater than 1, and the description information of each information group may include but is not limited to: the total number N of information groups, the index of the current information group, the length information of the current information group, etc. The sub-code stream may include but is not limited to: the total number N of information groups, the index of the current information group, the length information of the current information group, and the information group, etc.

[0270] It should be noted that the model information can be losslessly encoded to obtain the encoded data of the model information; and the information group can be losslessly encoded to obtain the encoded data of the information group; in this way, it can be ensured that the decoding model reconstructed by the second electronic device is consistent with the decoding model of the first electronic device.

[0271] S605: Encapsulate the model information code stream and the audio code stream into a first ES packet, and use the first ES packet as a first code stream.

[0272] Exemplarily, the model information code stream and the audio code stream may be encapsulated in the same ES packet, thereby obtaining a first ES packet; and then the first ES packet is used as the first code stream. In this case, the first code stream may also be called an ES stream.

[0273] Exemplarily, the number of first ES packets may be M, and the M first ES packets may include N ninth ES packets and MN tenth ES packets, where M and N are positive integers, and M is greater than or equal to N.

[0274] Specifically, N sub-streams and N audio streams can be encapsulated into N ninth ES packets, and the other MN audio streams can be used as MN tenth ES packets. Among them, a sub-stream and an audio stream can be encapsulated into a ninth ES packet, that is, the ninth ES packet can include a sub-stream and an audio stream. And one of the other MN audio streams can be used as a tenth ES packet, that is, the tenth ES packet includes an audio stream. It should be noted that an audio stream can be obtained by encoding a frame of audio signal based on the coding model.

[0275] Figure 6C FIG. 4 is a schematic diagram showing the structure of a first code stream.

[0276] Reference Figure 6C (1), exemplarily, when N is equal to 1, the model information code stream and an audio code stream can be encapsulated in one ES packet to obtain a ninth ES packet. Specifically, the model information code stream and the first audio code stream are encapsulated in one ES packet to obtain a ninth ES packet; and the other M-1 audio code streams are used as M-1 tenth ES packets. Exemplarily, in the ninth ES packet, the model information code stream can be located after the first audio code stream.

[0277] Reference Figure 6C (2) Exemplarily, when N is greater than 1, a substream and an audio stream can be encapsulated in one ES packet to obtain a ninth ES packet. Specifically, N substreams and N audio streams are encapsulated into N ninth ES packets; and the other MN audio streams are encapsulated into MN tenth ES packets. Exemplarily, in each ninth ES packet, the substream can be located after the audio stream.

[0278] It should be noted that N can be determined according to the data volume of the model information (or the encoded data of the model information), the network bandwidth and the encoded data of the audio signal to ensure the smoothness of the reconstructed audio signal played by the second electronic device.

[0279] For example, in Fig. 6AIn the embodiment of the present invention, the model identifier may be a field of ES. Exemplarily, the encapsulation format of ES may include multiple fields, and the field of ES may refer to the field included in the encapsulation format of ES. In other words, after data is encapsulated according to the encapsulation format of ES, an ES package may be obtained, and the ES package may include multiple fields, and the field of ES may refer to the field in the ES package.

[0280] Exemplarily, the audio code stream output by the audio encoder may include a header (ie, a header of an ES packet) and encoded data of the audio data (also referred to as a payload of the audio data).

[0281] Exemplarily, the header of the audio bitstream may include a field for indicating the type of decoding model (or the type of encoding model). When there is an unused identification value in the field, the audio encoder may use the field as a model identifier for indicating whether the first bitstream contains model information bitstream, and select a preset identification value from the unused identification values ​​corresponding to the field value.

[0282] For example, in the Audio Vivid standard, the field used to indicate the type of decoding model (or the type of encoding model) is the nn_type field; the nn_type field can be represented by 3 bits, that is, the nn_type field corresponds to 8 identification values: "0", "1", "2", "3", "4", "5", "6", "7". Currently, "0" and "1" are used identification values, and "2", "3", "4", "5", "6", and "7" are reserved identification values, that is, unused identification values. The definition of the nn_type field can be shown in Table 1 below:

[0283] Table 1

[0284] nn_type(3bit) meaning 0 Basic configuration default model 1 Low-complexity configuration default model 2-7 Reserve

[0285] Furthermore, the nn_type field can be used as a model identifier. For example, "2" can be used as a preset identifier value. In this case, the nn_type field definition is as shown in Table 2:

[0286] Table 2

[0287] nn_type(3bit) meaning 0 Basic configuration default model 1 Low-complexity configuration default model 2 Custom Model 3-7 Reserve

[0288] Exemplarily, different identification values ​​in the unused identification values ​​corresponding to the nn_type field may be used to represent different types of coding models (or decoding models), that is, different preset identification values ​​are used to represent different types of coding models (or decoding models); as shown in Table 3 below:

[0289] Table 3

[0290] nn_type(3bit) meaning 0 Basic configuration default model 1 Low-complexity configuration default model 2 Custom Model 1 3 Custom Model 2 4 Custom Model 3 5-7 Reserve

[0291] Referring to Table 3, the identification value “2” may be used to represent the custom model 1, the identification value “3” may be used to represent the custom model 3, and the identification value “4” may be used to represent the custom model 3.

[0292] Exemplarily, in other standards, the audio encoder may add an additional field in the header of the audio bitstream, and use the additional field as a model identifier for indicating whether the first bitstream includes a model information bitstream; and set the field value of the additional field to a preset identifier value.

[0293] Exemplarily, only the model identifier in the first ninth ES packet (i.e., the ninth ES packet containing the first audio code stream) can be set to a preset identifier value, or the model identifier in some or all of the N ninth ES packets and MN tenth ES packets can be set to a preset identifier value, and this application does not impose any restrictions on this.

[0294] In this way, after obtaining the ES packet, the ES packet can be directly transmitted without encapsulating the ES packet according to the target container format or the target transmission protocol, which can reduce the bit rate overhead and reduce the bandwidth requirement. In addition, the second code stream and the third code stream are encapsulated in the same ES packet, and the second code stream and the third code stream can be sent on the same channel, which can reduce the complexity of transmission.

[0295] Figure 7 FIG. 1 is a schematic diagram of an exemplary audio decoding process. Figure 7 is with Fig. 6A The audio decoding process corresponding to the audio encoding process.

[0296] S701, receiving a first code stream.

[0297] Exemplarily, when the first electronic device is in accordance with Fig. 6A In the embodiment, when the audio encoding method is encoded, the first code stream received by the second electronic device is substantially an ES stream; in this case, the first code stream may include a first ES packet. It is assumed that the reconstructed audio signal is M frames, the first ES packets may be M, and the M first ES packets may include N ninth ES packets and MN tenth ES packets.

[0298] S702: parse the value of the model identifier from the field corresponding to the model identifier of the first ES packet.

[0299] Exemplarily, the model identifier is a field of ES.

[0300] Exemplarily, after the second electronic device receives the first code stream, the decapsulation module may parse each ES packet and parse the value of the model identifier from the ES packet. Among the multiple ES packets included in the first code stream, at least one ES packet includes a field corresponding to the model identifier; and then the decapsulation module may parse the value of the model identifier from the field corresponding to the model identifier of at least one ES packet (the ninth ES packet or the tenth ES packet).

[0301] S703, when the value of the model identifier is a preset identifier value, read the model information code stream from the ninth ES packet.

[0302] Exemplarily, when the value of the model identifier is a preset identifier value, the decapsulation module can parse the header of each ES packet to determine the length of the encoded data of the audio signal; then, based on the length of the header, the length of the encoded data of the audio signal, and the total length of the ES packet, determine whether the ES packet contains a model information code stream (or sub-code stream). When the total length of the ES packet is greater than the sum of the length of the encoded data of the audio signal and the length of the header, it is determined that the ES packet contains a model information code stream (or sub-code stream), that is, the ES packet is the ninth ES packet. When the total length of the ES packet is equal to the sum of the length of the encoded data of the audio signal and the length of the header, it is determined that the ES packet does not contain a model information code stream (or sub-code stream), that is, the ES packet is the tenth ES packet.

[0303] Exemplarily, for a ninth ES packet, the coded data portion of the audio signal can be skipped according to the length of the coded data of the audio signal, and the model information code stream (or sub-code stream) can be read. When N is equal to 1, the model information code stream can be directly read from a ninth ES packet. When N is greater than 1, N sub-code streams can be read, and these N sub-code streams can constitute the model information code stream.

[0304] S704: Reconstruct the model based on the model information to obtain a decoding model.

[0305] Exemplarily, when N is equal to 1, the decapsulation module can parse the description information of the model information in the model information code stream to determine the length of the model information (or the encoded data of the model information); then, based on the length of the model information (or the encoded data of the model information), read out the model information (or the encoded data of the model information).

[0306] Exemplarily, when N is greater than 1, for each sub-code stream, the decapsulation module can parse the description information of the information group in the sub-code stream to determine the length of the current information group (or the encoded data of the current information group), the index corresponding to the current information group, and the total number of information groups N; then, according to the length of the current information group (or the encoded data of the current information group), read the information group (or the encoded data of the information group) from the sub-code stream. In this way, after obtaining N information groups, the N information groups can be spliced ​​according to the indexes corresponding to each information group to obtain model information. Alternatively, after obtaining the encoded data of N information groups, the encoded data of N information groups can be decoded to obtain N information groups; then, according to the indexes corresponding to each information group, the N information groups can be spliced ​​to obtain model information.

[0307] Exemplarily, when the encoded data of the model information (or information group) is read out, the decapsulation module can call the decoder to decode the encoded data of the model information (or information group) to determine the model information (or information group); thereafter, the model is reconstructed based on the model information (or the spliced ​​information group) to obtain a decoding model.

[0308] S705, read the audio code stream from the ninth ES packet and use the tenth ES packet as the audio code stream.

[0309] In one possible way, after parsing the header of the ninth ES packet and determining the length of the coded data of the audio signal, the portion corresponding to the coded data of the audio signal in the ninth ES packet can be determined according to the length of the coded data of the audio signal; then, the header and the portion corresponding to the coded data of the audio signal can be extracted from the ninth ES packet to obtain an audio bitstream. The tenth ES packet can be used as an audio bitstream; and then the audio bitstream is sent to an audio decoder.

[0310] In a possible manner, after parsing the header of a ninth ES packet and determining the length of the coded data of the audio signal, the coded data of the audio signal can be read from the ninth ES packet according to the length of the coded data of the audio signal. And for each tenth ES packet, the header of the tenth ES packet can be parsed and the length of the coded data of the audio signal can be determined. After that, the coded data of the audio signal is sent to the audio decoder.

[0311] S706: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.

[0312] Exemplarily, when the decapsulation module sends an audio code stream to the audio decoder, after the audio decoder reads the reconstructed decoding model, it can first read the encoded data of the audio signal from the audio code stream, and then decode the encoded data of the audio signal based on the decoding model to obtain a reconstructed audio signal. When the decapsulation module sends the encoded data of the audio signal to the audio decoder, after the audio decoder reads the reconstructed decoding model, it can directly decode the encoded data of the audio signal based on the decoding model to obtain a reconstructed audio signal.

[0313] It should be understood that when N is equal to 1, after parsing the model information code stream from the 1st 9th ES packet, the model reconstruction can be performed to obtain the decoding model; then, the audio decoder can decode the 1st audio code stream based on the decoding model to obtain the 1st frame reconstructed audio signal. Afterwards, after the second electronic device receives each 10th ES packet (the 2nd first ES packet to the Mth first ES packet), the audio decoder can decode the 10th ES packet in real time to obtain the 2nd frame to the Mth frame reconstructed audio signal.

[0314] When N is greater than 1, the second electronic device needs to parse out N sub-code streams from the 1st to the Nth ES packets before it can reconstruct the model based on the N sub-code streams to obtain a decoding model; then, the audio decoder can decode the 1st to the Nth audio code streams based on the decoding model to obtain the 1st frame to the Nth frame to reconstruct the audio signal. After that, after the second electronic device receives each tenth ES packet (the M-N+1th first ES packet to the Mth first ES packet), the audio decoder can decode the tenth ES packet in real time to obtain the M-N+1th frame to the Mth frame to reconstruct the audio signal. That is to say, in this case, the audio decoder's decoding of the audio code stream in the Nth frame ninth ES packet is delayed, or in other words, the decoding delay of the audio decoder is N frames.

[0315] Figure 8 FIG. 1 is a schematic diagram of an exemplary audio encoding process. Figure 8 The target code stream type is the second type; wherein the audio code stream and the model information code stream can be encapsulated in the same ES package, and the ES package is encapsulated according to the target container format to obtain the first code stream.

[0316] S801: Acquire an audio signal.

[0317] S802: Determine whether both the encoding model and the decoding model are updated.

[0318] S803: When both the encoding model and the decoding model are updated, the audio signal is encoded based on the updated encoding model to obtain an audio bitstream.

[0319] S804, generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.

[0320] S805, encapsulate the model information code stream and the audio code stream into a second ES packet.

[0321] Exemplarily, S801 to S805 can refer to the description of S601 to S605 above, which will not be repeated here. The format of the second ES packet is similar to that of the first ES packet, that is, M second ES packets include N ninth ES packets and MN tenth ES packets, which will not be repeated here.

[0322] S806: Encapsulate the second ES packet according to the target container format to obtain a first code stream.

[0323] Exemplarily, the second ES packet may be encapsulated according to the target container format and the type of the second ES packet may be set to an audio type to obtain the first code stream.

[0324] In the process of encapsulating the second ES packet according to the target container format, the type identifier of the ES packet can be set to a first identifier value, and the first identifier value is used to indicate that the type of the ES packet is an audio type. The type identifier can be a field in the target container format used to indicate the type of the ES packet.

[0325] For example, in Figure 8 In one possible embodiment, the model identifier may be a field of ES; for details, please refer to the above Fig. 6A The description in the embodiments will not be repeated here.

[0326] In one possible manner, when encapsulating the second ES packet according to the target container format, a field in the additional information (the field is a field in the target container format) can be used as a model identifier, and the value of the model identifier is set to a preset identifier value. For example, if the target container format is TS / MP4, a field in the additional information such as the nn_type field can be used as a model identifier, and the field value of the nn_type field can be set to a preset identifier value. For details, refer to the above Fig. 6A The description in the embodiment will not be repeated here. In this way, the second electronic device can determine whether the decoding model needs to be reconstructed without parsing the ES packet.

[0327] Among them, the ES packet can be divided into multiple small packets and then encapsulated according to the target container format, which is conducive to the bit stream distribution device to perform bit stream distribution and audio and video synchronization control.

[0328] Fig. 9 FIG. 1 is a schematic diagram of an exemplary audio decoding process. Fig. 9 is with Figure 8 The audio decoding process corresponding to the audio encoding process.

[0329] S901, receiving a first code stream.

[0330] Exemplarily, when the first electronic device is in accordance with Figure 8 In the embodiment, when the audio encoding method is encoded, assuming that the target container format is TS format, the first code stream received by the second electronic device is substantially a TS stream; assuming that the target container format is MP4 format, the first code stream received by the second electronic device is substantially an MP4 stream. This application is described by taking the first code stream as a TS stream as an example.

[0331] Exemplarily, when the model identifier is a field of the target container format, the process of parsing the model identifier from the first code stream may refer to S902 to S903:

[0332] S902: Decapsulate the first code stream according to the target container format to obtain a field corresponding to the model identifier.

[0333] S903: Parse the field corresponding to the model identifier to obtain the value of the model identifier.

[0334] Exemplarily, after the second electronic device receives the first code stream, the decapsulation module can decapsulate the first code stream to obtain the field corresponding to the model identifier and M second ES packets (M second ES packets include N ninth ES packets and MN tenth ES packets). Then, the decapsulation module can parse the field corresponding to the model identifier to obtain the value of the model identifier.

[0335] Exemplarily, when the model identifier is a field of ES, the value of the model identifier can be parsed from the field corresponding to the model identifier of the ES package; this can be referred to the description of S702 above, which will not be repeated here.

[0336] S904: When the value of the model identifier is a preset identifier value, read the model information code stream from the ninth ES packet.

[0337] S905, reconstructing the model based on the model information code stream to obtain a decoding model.

[0338] S906, read the audio code stream from the ninth ES packet and use the tenth ES packet as the audio code stream.

[0339] S907: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.

[0340] Illustratively, S904 to S907 may refer to the description of S503 to S506 above, which will not be repeated here.

[0341] Fig. 10A FIG. 1 is a schematic diagram of an exemplary audio encoding process. Fig. 10A The target bitstream type is the second type. The audio bitstream and the model information bitstream may be respectively encapsulated in different ES packets, and the ES packet of the model information bitstream and the ES packet of the audio bitstream may be encapsulated together according to the target container format to obtain the first bitstream.

[0342] S1001, obtaining an audio signal.

[0343] S1002: Determine whether both the encoding model and the decoding model are updated.

[0344] S1003: When both the encoding model and the decoding model are updated, the audio signal is encoded based on the updated encoding model to obtain an audio bitstream.

[0345] S1004, generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.

[0346] Illustratively, S1001 to S1004 may refer to the description of S601 to S604 above, which will not be repeated here.

[0347] S1005, encapsulate the model information code stream into a third ES packet.

[0348] Exemplarily, the model information may be divided into N information groups; then, one information group is encapsulated as a third ES packet, thus, N third ES packets may be obtained.

[0349] Exemplarily, when N is equal to 1, there is one third ES packet, and the third ES packet includes a model information code stream and description information of the model information, which can be referred to the description of the above embodiment and will not be repeated here.

[0350] Exemplarily, when N is greater than 1, there are N third ES packets, and each third ES packet includes a sub-stream and description information of the sub-stream. The description of the above embodiment may be referred to and will not be repeated here.

[0351] S1006, taking the audio code stream as the fourth ES packet.

[0352] Exemplarily, one audio code stream may be taken as a fourth ES packet, thus, N fourth ES packets may be obtained.

[0353] S1007: Encapsulate the third ES packet and the fourth ES packet according to the target container format, set the type of the third ES packet to a preset type, and set the type of the fourth ES packet to an audio type, so as to obtain a first code stream.

[0354] Exemplarily, in the process of encapsulating the fourth ES packet and the third ES packet according to the target container format, the type identifier of the fourth ES packet may be set to a first identifier value, where the first identifier value is used to indicate that the type of the ES packet is an audio type.

[0355] Exemplarily, in the process of encapsulating the fourth ES packet and the third ES packet according to the target container format, the type identifier of the third ES packet can be set to a second identifier value, and the second identifier value is used to indicate that the type of the ES packet is a preset type. The model identifier is used to indicate that the ES packet includes a model information code stream or a sub-code stream. For details, please refer to the description of the above embodiment, which will not be repeated here.

[0356] In one possible manner, the preset type may be a private type. In this case, the private type identifier corresponding to the third ES packet may include private_stream_id_1 and private_stream_id_2; correspondingly, the private type identifier private_stream_id_1 corresponding to the third ES packet may be set to 0xBD, or the private type identifier private_stream_id_2 corresponding to the third ES packet may be set to 0xBF.

[0357] In a possible manner, the preset type may be a custom type, and in this case, the custom type identifier corresponding to the third ES packet may include stream_id and stream_type. The custom type identifiers stream_id and stream_type corresponding to the third ES packet may be set to the second identifier value.

[0358] For example, in Fig. 10A In the embodiment, the model identifier can be a field of ES or a field of the target container format, and this application does not limit this. For details, please refer to the description above and will not be repeated here.

[0359] Exemplarily, when the model identifier is a field of the ES, the model identifier may be located in at least one third ES packet, or in at least one fourth ES packet.

[0360] Please refer to Fig. 10B , for example, in Fig. 10B In the example, the target container format is TS, N=1; the third ES packet and the fourth ES packet can be encapsulated into a TS stream. In this case, the first code stream is a TS stream.

[0361] Fig.11 FIG. 1 is a schematic diagram of an exemplary audio decoding process. Fig.11 is with Fig. 10A The audio decoding process corresponding to the audio encoding process.

[0362] S1101, receiving a first code stream.

[0363] Exemplarily, when the model identifier is a field of the target container format, the process of parsing the model identifier from the first code stream may refer to S1102 to S1103:

[0364] S1102, decapsulate the first code stream according to the target container format to obtain a field corresponding to the model identifier, a third ES packet, and a fourth ES packet.

[0365] Exemplarily, the first code stream is decapsulated according to the target container format, and the field corresponding to the model identifier, multiple ES packets and the type of ES packets can be obtained. The decapsulation module can parse the type identifier of each ES packet. When it is determined that the type identifier of the ES packet is the second identification value, it can be determined that the type of the ES is a preset type. At this time, the ES packet can be called a third ES packet. Exemplarily, when the decapsulation model determines that the type identifier of the ES packet is the first identification value, it can be determined that the type of the ES is an audio type. At this time, the ES packet can be called a fourth ES packet.

[0366] S1103, parsing the field corresponding to the model identifier to obtain the value of the model identifier.

[0367] S1104: When the value of the model identifier is a preset identifier value, read the model information code stream from the third ES packet.

[0368] S1105, reconstructing the model based on the model information to obtain a decoding model.

[0369] S1106, read the audio code stream from the fourth ES packet.

[0370] S1107, decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.

[0371] Illustratively, S1104 to S1107 may refer to the description of S703 to S706 above, which will not be repeated here.

[0372] In this way, compared with encapsulating the second code stream and the third code stream in the same ES packet, the second electronic device does not need to split the ES packet to parse the second code stream and the third code stream, thereby reducing the complexity of the parsing process.

[0373] Fig. 12A FIG. 1 is a schematic diagram of an exemplary audio encoding process. Fig. 12A The target bitstream type is the second type. The model information bitstream can be encapsulated as metadata and encapsulated in different ES packets with the audio bitstream; then, the ES packet of the model information bitstream and the ES packet of the audio bitstream are encapsulated together according to the target container format to obtain the first bitstream.

[0374] S1201: Acquire an audio signal.

[0375] S1202: Determine whether both the encoding model and the decoding model are updated.

[0376] S1203: When both the encoding model and the decoding model are updated, the audio signal is encoded based on the updated encoding model to obtain an audio bitstream.

[0377] S1204, generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.

[0378] Illustratively, S1201 to S1204 may refer to the description of S601 to S604 above, which will not be repeated here.

[0379] S1205, encapsulate the model information code stream as metadata to obtain a third ES package.

[0380] Exemplarily, the model information may be divided into N information groups; then, one information group is encapsulated as a third ES packet, thus, N third ES packets may be obtained.

[0381] Reference Fig. 12B , exemplarily, the model information code stream can be taken as a Metadata AU (Access Unit). When N is equal to 1, a MetadataAU can be taken as a MetadataAU Cell; then, a MetadataAU Cell is encapsulated as a third ES packet to obtain a third ES packet. When N is greater than 1, a MetadataAU can be split into N MetadataAU Cells; then, a MetadataAUCell is encapsulated as a third ES packet to obtain N third ES packets.

[0382] S1206: Use the audio code stream as the fourth ES packet.

[0383] Exemplarily, one audio code stream may be taken as a fourth ES packet, thus, N fourth ES packets may be obtained.

[0384] S1207: encapsulate the third ES packet and the fourth ES packet according to the target container format, set the type of the third ES packet to the metadata type, and set the type of the fourth ES packet to the audio type, so as to obtain a first code stream.

[0385] For example, the metadata type is Fig.11 One of the preset types in the embodiment.

[0386] exist Fig. 12B (2) In the target container format, TS is used. Fig. 12B Based on (1), N third ES packets and M fourth ES packets are encapsulated into a TS stream; thus, the first code stream obtained is a TS stream.

[0387] Exemplarily, in the process of encapsulating the fourth ES packet and the third ES packet according to the target container format, the type identifier of the fourth ES packet can be set to the first identification value; and the type identifier of the third ES packet can be set to the third identification value, and the third identification value is used to characterize that the type of the ES packet is a metadata type.

[0388] exist Fig. 12A In the embodiment of the present invention, the syntax of the MetadataAU Cell in the first code stream may also be modified; assuming that the target container format is TS, that is, the first code stream is a TS stream, the syntax of the MetadataAU Cell in the TS stream may be as shown in Table 4 below:

[0389] Table 4

[0390]

[0391] In Table 4, decoder_config_flag: indicates whether the current MetadataAU contains decoder initialization information. Fig. 12A In the embodiment, the identifier value of decoder_config_flag in the Metadata AU Cell syntax split by the MetadataAU corresponding to the model information code stream is set to 1; to indicate that the model update information is necessary information for initializing the audio decoder.

[0392] exist Fig. 12A In the embodiment, the model identifier can be a field of ES or a field of the target container format, and this application does not limit this.

[0393] Exemplarily, when the model identifier is a field of the ES, the model identifier may be located in at least one third ES packet, or in at least one fourth ES packet.

[0394] Fig.13 FIG. 1 is a schematic diagram of an exemplary audio decoding process. Fig.13 is with Fig. 12A The audio decoding process corresponding to the audio encoding process.

[0395] S1301: Receive a first code stream.

[0396] S1302, decapsulate the first code stream according to the target container format to obtain the field corresponding to the model identifier, the third ES packet and the fourth ES packet.

[0397] Exemplarily, the first code stream is decapsulated according to the target container format, and the fields corresponding to the model identifier, multiple ES packets, and the type of the ES packet can be obtained. The decapsulation module can parse the type identifier of each ES packet. When it is determined that the type identifier of the ES packet is the third identification value, it can be determined that the type of the ES is a metadata type. At this time, the ES packet can be called a third ES packet. Exemplarily, when the decapsulation model determines that the type identifier of the ES packet is the first identification value, it can be determined that the type of the ES is an audio type. At this time, the ES packet can be called a fourth ES packet.

[0398] S1303: parse the field corresponding to the model identifier to obtain the value of the model identifier.

[0399] Exemplarily, S1301 to S1303 may refer to the description of S1101 to S1103 above, which will not be repeated here.

[0400] S1304: When the value of the model identifier is a preset identifier value, read the model information code stream from the third ES packet.

[0401] S1305: Reconstruct the model based on the model information to obtain a decoding model.

[0402] S1306, read the audio code stream from the fourth ES packet.

[0403] S1307: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.

[0404] Exemplarily, the decapsulation module can also parse the decoder_config_flag in the syntax of the Metadata AU Cell corresponding to the third ES packet; if decoder_config_flag is "1", the decoder can read the decoding model (ie, initialize), and then decode the audio bitstream based on the decoding model to obtain a reconstructed audio signal.

[0405] Illustratively, S1305 to S1307 may refer to the description of S703 to S706 above, which will not be repeated here.

[0406] Fig.14A FIG. 1 is a schematic diagram of an exemplary audio encoding process. Fig.14A In the embodiment, the target bitstream type is the second type; wherein the audio bitstream and the model information bitstream can be respectively encapsulated in different ES packets, and the ES packets of the model information bitstream and the ES packets of the audio bitstream can be respectively encapsulated according to the target container format to obtain the first bitstream.

[0407] S1401: Acquire an audio signal.

[0408] S1402: Determine whether both the encoding model and the decoding model are updated.

[0409] S1403: When both the encoding model and the decoding model are updated, the audio signal is encoded based on the updated encoding model to obtain an audio bitstream.

[0410] S1404, generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.

[0411] Illustratively, S1401 to S1404 may refer to the description of S601 to S604 above, which will not be repeated here.

[0412] S1405, encapsulate the model information code stream into a fifth ES packet.

[0413] S1406: Encapsulate the fifth ES packet according to the target container format and set the type of the fifth ES packet to a preset type to obtain a fourth code stream.

[0414] Exemplarily, in the process of encapsulating the fifth ES packet according to the target container format, the type identifier of the ES packet may be set to the second identifier value.

[0415] For example, if the target container format is TS, the fifth ES packet is encapsulated into a TS stream, that is, the fourth code stream.

[0416] S1407, use the audio code stream as the sixth ES packet.

[0417] S1408: Encapsulate the sixth ES packet according to the target container format and set the type of the sixth ES packet to an audio type to obtain a fifth code stream.

[0418] Exemplarily, one audio code stream may be taken as a sixth ES packet, thus, N sixth ES packets may be obtained.

[0419] Exemplarily, in the process of encapsulating the sixth ES packet according to the target container format, the type identifier of the sixth ES packet may be set to the first identifier value.

[0420] For example, if the target container format is TS, the sixth ES packet is encapsulated into a TS stream, that is, the fifth code stream.

[0421] S1409: Combine the fourth code stream and the fifth code stream to obtain a first code stream.

[0422] Reference Fig. 14B , the target container format is TS, the first code stream is the TS stream, and the first code stream may include two TS streams: TS stream 1 and TS stream 2, wherein TS stream 1 is the fourth code stream; TS stream 1 is the fifth code stream.

[0423] For example, in Fig.14A In the embodiment, the identifier can be the field value of the ES field, or the field value of the corresponding field of the target container format, and this application does not limit this; the details can be referred to the description above, which will not be repeated here.

[0424] Exemplarily, when the model identifier is a field in the target container format, the model identifier may be located in the fourth code stream or in the fifth code stream.

[0425] Exemplarily, when the model identifier is a field of the ES, the model identifier may be located in at least one fifth ES packet, or in at least one sixth ES packet.

[0426] Fig.15 FIG. 1 is a schematic diagram of an exemplary audio decoding process. Fig.15 is with Fig.14A The audio decoding process corresponding to the audio encoding process.

[0427] S1501, receiving a first code stream.

[0428] Exemplarily, when the first electronic device is in accordance with Fig.14A When encoding by the audio encoding method in the embodiment, assuming that the target container format is TS, the first code stream received by the second electronic device is substantially a TS stream; assuming that the target container format is MP4, the first code stream received by the second electronic device is substantially an MP4 stream.

[0429] In this case, the first code stream may include the fourth code stream and the fifth code stream.

[0430] Exemplarily, when the model identifier is a field of the target container format and the fourth codestream includes a field corresponding to the model identifier, the process of parsing the value of the model identifier from the first codestream may refer to S1502 to S1503:

[0431] S1502, decapsulate the fourth code stream according to the target container format to obtain a field corresponding to the model identifier and a fifth ES packet.

[0432] The following description is made by taking the case where the first code stream is a TS stream as an example.

[0433] Exemplarily, after the second electronic device receives the first code stream, the decapsulation module can decapsulate each TS stream according to the target container format to obtain ES packets and ES packet types; and decapsulate part of the TS stream to obtain the field corresponding to the model identifier.

[0434] Exemplarily, when the type of the ES packet is a preset type, it can be determined that the corresponding TS stream is the fourth code stream, and the corresponding ES packet is the fifth ES packet. When the type of the ES packet is an audio type, it can be determined that the corresponding TS stream is the fifth code stream, and the corresponding ES packet is the sixth ES packet.

[0435] S1503: Parse the field corresponding to the model identifier to obtain the value of the model identifier.

[0436] S1504: When the value of the model identifier is a preset identifier value, read the model information code stream from the fifth ES packet.

[0437] S1505: Reconstruct the model based on the model information to obtain a decoding model.

[0438] S1506: decapsulate the fifth code stream according to the target container format to obtain a sixth ES packet.

[0439] S1507, read the audio code stream from the sixth ES packet.

[0440] S1508: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.

[0441] Illustratively, S1503 to S1508 may refer to the description of S703 to S706 above, which will not be repeated here.

[0442] In this way, compared with encapsulating the second code stream and the third code stream in the same TS stream, the second electronic device does not need to split the TS stream to parse the second code stream and the third code stream, thereby reducing the complexity of the parsing process.

[0443] Fig.16A FIG. 1 is a schematic diagram of an exemplary audio encoding process. Fig.16A The target bitstream type is the third type; wherein the audio bitstream and the model information bitstream are respectively encapsulated in different ES packets, and the ES packets of the model information bitstream and the ES packets of the audio bitstream are respectively encapsulated according to the target container format to obtain a sixth bitstream and a seventh bitstream, and then the sixth bitstream and the seventh bitstream are encapsulated according to the target transmission protocol to obtain a first bitstream.

[0444] S1601: Acquire an audio signal.

[0445] S1602: Determine whether both the encoding model and the decoding model are updated.

[0446] S1603: When both the encoding model and the decoding model are updated, the audio signal is encoded based on the updated encoding model to obtain an audio bitstream.

[0447] S1604, generating a model information code stream based on the model information of the updated decoding model; wherein the updated decoding model corresponds to the updated encoding model.

[0448] Illustratively, S1601 to S1604 may refer to the description of S601 to S604 above, which will not be repeated here.

[0449] S1605, encapsulate the model information code stream into the seventh ES packet.

[0450] S1606: Encapsulate the seventh ES packet according to the target container format and set the type of the seventh ES packet to a preset type to obtain a sixth code stream.

[0451] S1607: Use the audio code stream as the eighth ES packet.

[0452] S1608: Encapsulate the eighth ES packet according to the target container format and set the type of the eighth ES packet to an audio type to obtain a seventh code stream.

[0453] S1609: Encapsulate the sixth code stream and the seventh code stream according to the target transmission protocol to obtain a first code stream.

[0454] Exemplarily, when the target transmission protocol is the Http (Hyper Text Transfer Protocol) transmission protocol, the first code stream may be an HLS stream. In this case, the first code stream may include: an index file, a sixth code stream, and a seventh code stream.

[0455] Reference Fig. 16B , the target transmission protocol is HLS, the target container format is TS; the first code stream is the HLS stream, and the first code stream may include two TS streams encapsulated according to HLS: TS stream 1 and TS stream 2, wherein TS stream 1 is the sixth code stream; TS stream 1 is the seventh code stream.

[0456] Exemplarily, during the decoding process, the information in the index file is usually parsed in sequence from the beginning to the end; and in order to ensure that the decoding model is reconstructed before the audio bitstream is obtained, the index information of the sixth bitstream can be set before the index information of all seventh bitstreams in the index file.

[0457] Exemplarily, the index file also includes multiple tags such as an initialization tag, which is used to indicate how to obtain media initialization information (Media Initialization Section), which may refer to information that needs to be parsed first before decoding a media stream (such as an audio stream). Furthermore, the index information of the sixth stream may be used as attribute information of the initialization tag.

[0458] Illustratively, the index information may be a Uniform Resource Locator (URL), which is not limited in the present application.

[0459] For example, in Fig.16A In the embodiment, the model identifier can be a field of ES or a field of the target container format, and this application does not limit this. For details, please refer to the description above and will not be repeated here.

[0460] Exemplarily, when the model identifier is a field in the target container format, the model identifier may be located in the sixth code stream or in the seventh code stream.

[0461] Exemplarily, when the model identifier is a field of the ES, the model identifier may be located in at least one seventh ES packet, or in at least one eighth ES packet.

[0462] exist Fig.16A In the embodiment, the model identifier may also be a field of the target transmission protocol.

[0463] It should be noted that Fig.16A The first code stream in the embodiment is Fig.14A There are differences in the first code stream in the embodiments; Fig.14A In the embodiment, the first code stream only includes the sixth code stream and the seventh code stream, and Fig.16A In the embodiment, the first code stream includes not only the sixth code stream and the seventh code stream, but also a field of the target transmission protocol.

[0464] Among them, a second code stream with different reconstruction accuracy can be generated (which can be understood as the accuracy of the reconstructed signal obtained by decoding the second code stream); in this way, it can be beneficial for the code stream distribution device to distribute the first code stream containing the second code stream with different reconstruction accuracy according to the bandwidth, thereby ensuring the smoothness of the reconstructed signal playback.

[0465] Fig.17 FIG. 1 is a schematic diagram of an exemplary audio decoding process. Fig.17 is with Fig.16A The audio decoding process corresponding to the audio encoding process.

[0466] S1701: Receive a first code stream.

[0467] S1702: Decapsulate the first bitstream according to the target transmission protocol to obtain a sixth bitstream and a seventh bitstream.

[0468] Exemplarily, when the target transmission protocol is the Http transmission protocol, the first code stream may be decapsulated according to the Http transmission protocol to obtain an index file and multiple code streams.

[0469] In one possible manner, index information of the sixth code stream may be read from an index file, and then the sixth code stream may be determined from multiple code streams according to the index information of the sixth code stream; and index information of the seventh code stream may be read from an index file, and then the seventh code stream may be determined from multiple code streams according to the index information of the seventh code stream.

[0470] In one possible manner, the index information of the seventh code stream can be read from the index file, and then the seventh code stream can be determined from multiple code streams according to the index information of the seventh code stream; and the index information of the sixth code stream can be read from the attribute information of the initialization tag of the index file, and the sixth code stream can be obtained from multiple code streams according to the index information of the sixth code stream.

[0471] S1703, decapsulate the sixth code stream according to the target container format to obtain the field corresponding to the model identifier and the seventh ES packet.

[0472] S1704, parse the field corresponding to the model identifier to obtain the value of the model identifier.

[0473] S1705: When the value of the model identifier is a preset identifier value, read the model information code stream from the seventh ES packet.

[0474] S1706: Reconstruct the model based on the model information to obtain a decoding model.

[0475] S1707, decapsulate the seventh code stream according to the target container format to obtain an eighth ES packet.

[0476] S1708, read the audio code stream from the eighth ES packet.

[0477] S1709: Decode the audio code stream based on the decoding model to obtain a reconstructed audio signal.

[0478] Exemplarily, S1703 to S1709 may refer to the description of S1502 to S1508 above, which will not be repeated here.

[0479] In one possible manner, the first electronic device may transmit index information corresponding to the model information of the decoding model to the second electronic device. Specifically, the index information corresponding to the model information of the decoding model may be determined, and then the first code stream may be obtained based on the second code stream and the index information corresponding to the model information of the decoding model. In this case, the preset identification value is used to characterize that the first code stream includes the index information corresponding to the model information of the decoding model. During the decoding process, the process for the second electronic device to obtain the model information of the decoding model may be as follows: parsing the index information of the model information from the first code stream; and obtaining the model information based on the index information of the model information. That is, in this case, the preset identification value instructs the second electronic device to obtain the model information of the decoding model based on the index information obtained from the first code stream.

[0480] Exemplarily, in a possible way to obtain the first code stream based on the index information corresponding to the model information of the second code stream and the decoding model, the second code stream can be used as the third ES packet, and the third ES packet can be encapsulated according to the target container format to obtain the fifth code stream; the fifth code stream can be encapsulated according to the target transmission protocol to obtain the first code stream; at this time, the first code stream can include an index file and the fifth code stream, and the index information of the decoding model can be added to the index file. In a possible way, the index information of the decoding model can be added before the index information of the fifth code stream. In a possible way, the index information corresponding to the model information of the decoding model can be used as the attribute information of the initialization tag in the index file.

[0481] Compared with the code stream carrying the model information in the first code stream, the code rate overhead of only carrying the index information corresponding to the model information of the decoding model in the first code stream is less; and the bandwidth requirement for the first electronic device can be reduced.

[0482] Exemplarily, in a possible way to obtain the first code stream based on the index information corresponding to the model information of the second code stream and the decoding model, the eighth code stream can be generated based on the index information corresponding to the model information of the decoding model; the eighth code stream and the second code stream are encapsulated to obtain the first code stream. The way to encapsulate the eighth code stream and the second code stream to obtain the first code stream can refer to the way to encapsulate the third code stream and the second code stream to obtain the first code stream, which will not be repeated here.

[0483] In one possible manner, the first electronic device and the second electronic device may pre-agree on a preset storage location in the third device for storing an updated coding model and an updated decoding model; in this way, the first electronic device does not need to send a model information code stream of the decoding model or index information corresponding to the model information of the decoding model. In this case, after the preset storage location in the third device updates the decoding model and the coding model, the value of the model identifier in the first code stream may be set to a preset identifier value; at this time, the preset identifier value is used to indicate that the model information of the decoding model is stored in the preset storage location in the third device. In this way, the second electronic device may obtain the model information of the decoding model from the preset storage location in the third device; that is, the preset identifier value indicates that the second electronic device obtains the model information of the decoding model from the preset storage location in the third device. In this case, the first code stream does not need to transmit the model information code stream or the index information corresponding to the model information, which can reduce the code rate overhead; and can reduce the bandwidth requirements for the first electronic device.

[0484] It should be noted that, assuming that the way in which the second electronic device obtains the model information from the first code stream is called the first way; the way in which the second electronic device obtains the model information according to the index information corresponding to the model information obtained from the first code stream is called the second way; the way in which the second electronic device obtains the model information from the preset storage location of the third electronic device is called the third way. In one possible way, the first electronic device and the second electronic device can pre-agree on the way in which the second electronic device obtains the model information of the decoding model. In the encoding and decoding process, when the first electronic device and the second electronic device agree on the first way, the first electronic device can generate the third code stream based on the model information of the decoding model; and encapsulate the second code stream and the third code stream to obtain the first code stream. In the encoding and decoding process, when the first electronic device and the second electronic device agree on the second way, the first electronic device can generate the first code stream based on the index information corresponding to the second code stream and the model information of the decoding model. In the encoding and decoding process, when the first electronic device and the second electronic device agree on the third way, the first electronic device can use the second code stream to generate the first code stream.

[0485] In one possible manner, during the encoding and decoding process, the first electronic device may select one of the first manner, the second manner, and the third manner as the target manner. When the target manner is the first manner, the first electronic device may generate a third code stream based on the model information of the decoding model; and encapsulate the second code stream and the third code stream to obtain the first code stream. When the target manner is the second manner, the first electronic device may generate the first code stream based on the index information corresponding to the second code stream and the model information of the decoding model. When the target manner is the third manner, the first electronic device may use the second code stream to generate the first code stream.

[0486] Fig.18The schematic diagram of the coding device is shown as an example. The coding device can be used to execute the coding method of the above embodiment, so the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0487] Reference Fig.18 , exemplarily, the encoding device may include:

[0488] The signal acquisition module 1801 is used to acquire the signal to be encoded;

[0489] The encoding module 1802 is used to generate a first code stream; wherein the first code stream includes a model identifier and a second code stream, the second code stream is obtained by encoding the coded signal based on the coding model, the value of the model identifier is a preset identifier value, and the preset identifier value is used to instruct the second electronic device to obtain model information of a decoding model, and the decoding model corresponds to the coding model.

[0490] It should be understood that the encoding device can also perform other steps in the above encoding method, which will not be described in detail here.

[0491] Fig.19 The decoding device can be used to execute the decoding method of the above-mentioned embodiment, so the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0492] Reference Fig.19 , exemplary, the decoding device may include:

[0493] A code stream receiving module 1901 is used to receive a first code stream, the first code stream includes a model identifier and a second code stream, and the second code stream is generated based on the encoded information;

[0494] The decapsulation module 1902 is configured to obtain model information of a decoding model when the value of the model identifier is a preset identifier value;

[0495] The decapsulation module 1902 is further used to reconstruct the model based on the model information to obtain a decoding model;

[0496] The decoder 1903 is used to decode the second code stream based on the decoding model to obtain a reconstructed signal.

[0497] It should be understood that the decoding device can also perform other steps in the above decoding method, which will not be described in detail here.

[0498] In one example, Fig. 20A schematic block diagram of a device 2000 according to an embodiment of the present application is shown. The device 2000 may include: a processor 2001 and a transceiver / transceiver pin 2002 , and optionally, a memory 2003 .

[0499] The components of the device 2000 are coupled together via a bus 2004, wherein the bus 2004 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 2004 in the figure.

[0500] Optionally, the memory 2003 may be used to store instructions in the aforementioned method embodiment. The processor 2001 may be used to execute instructions in the memory 2003, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0501] The apparatus 2000 may be the electronic device or a chip of the electronic device in the above method embodiment. The electronic device may include a first electronic device and a second electronic device.

[0502] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0503] The embodiment of the present application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the electronic device implements the method in the above embodiment. The interface circuit is a transceiver / transceiver pin 2002.

[0504] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0505] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0506] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0507] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0508] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0509] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0510] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0511] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0512] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0513] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0514] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0515] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0516] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0517] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A coding method applied to a first electronic device, characterized in that: The method comprises: Obtaining a signal to be encoded; Generate a first code stream, the first code stream includes a second code stream and a model identifier, the second code stream is obtained by encoding the signal to be encoded based on the encoding model, the value of the model identifier is a preset identifier value, and the preset identifier value is used to instruct the second electronic device to obtain model information of a decoding model, and the decoding model corresponds to the encoding model.

2. The method according to claim 1, characterized in that The first code stream also includes a third code stream, and the third code stream is generated based on the model information of the decoding model.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Based on the model information of the decoding model, generating a third bitstream; The generating of the first code stream comprises: The second code stream and the third code stream are encapsulated to obtain the first code stream.

4. The method according to claim 1, characterized in that: The first code stream also includes index information corresponding to the model information of the decoding model.

5. The method according to claim 1 or 4, characterized in that: The generating of the first code stream comprises: The first bitstream is obtained based on the second bitstream and the index information corresponding to the model information of the decoding model.

6. The method according to claim 1, characterized in that The preset identification value is also used to indicate that the model information of the decoding model is stored in a preset storage location of the third electronic device.

7. The method according to any one of claims 1 to 6, characterized in that: The model identifier is a field of the elementary stream ES.

8. The method according to any one of claims 1 to 6, characterized in that: When the target codestream type of the first codestream is a codestream type encapsulated according to a target container format or a codestream type encapsulated according to a target transmission protocol, the model identifier is a field of the target container format.

9. The method according to any one of claims 1 to 6, characterized in that: When the target code stream type of the first code stream is a code stream type encapsulated according to a target transmission protocol, the model identifier is a field of the target transmission protocol.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Determining whether the decoding model is an updated decoding model; When the decoding model is an updated decoding model, the value of the model identifier is set to the preset identifier value.

11. The method according to claim 3, characterized in that The encapsulating the second code stream and the third code stream to obtain the first code stream includes: The second code stream and the third code stream are encapsulated according to the target code stream type to obtain the first code stream.

12. The method according to claim 11, characterized in that The target code stream type is an ES type, and encapsulating the second code stream and the third code stream according to the target code stream type to obtain the first code stream includes: The third code stream and the second code stream are encapsulated into a first ES packet, and the first ES packet is used as the first code stream.

13. The method according to claim 11, characterized in that The target codestream type is a codestream type encapsulated according to a target container format, and encapsulating the second codestream and the third codestream according to the target codestream type to obtain the first codestream includes: Encapsulating the third code stream and the second code stream into a second ES packet; The second ES packet is encapsulated according to the target container format to obtain the first code stream.

14. The method according to claim 11, characterized in that The target codestream type is a codestream type encapsulated according to a target container format, and encapsulating the second codestream and the third codestream according to the target codestream type to obtain the first codestream includes: Encapsulating the third code stream into a third ES packet, and using the second code stream as a fourth ES packet; The third ES packet and the fourth ES packet are encapsulated according to the target container format, and the type of the third ES packet is set to a preset type and the type of the fourth ES packet is set to a media type to obtain the first code stream.

15. The method according to claim 14, characterized in that The preset type is a metadata type.

16. The method according to claim 11, characterized in that The target codestream type is a codestream type encapsulated according to a target container format, and encapsulating the second codestream and the third codestream according to the target codestream type to obtain the first codestream includes: Encapsulating the third code stream into a fifth ES packet, and encapsulating the second code stream as a sixth ES packet; Encapsulating the fifth ES packet according to the target container format and setting the type of the fifth ES packet to a preset type to obtain a fourth code stream; Encapsulate the sixth ES packet according to the target container format and set the type of the sixth ES packet to a media type to obtain a fifth code stream; The fourth code stream and the fifth code stream are combined to obtain the first code stream.

17. The method according to claim 11, characterized in that The target code stream type is a code stream type encapsulated according to a target transmission protocol, and encapsulating the second code stream and the third code stream according to the target code stream type to obtain the first code stream includes: Encapsulating the third code stream into a seventh ES packet, and encapsulating the second code stream as an eighth ES packet; Encapsulating the seventh ES packet according to the target container format and setting the type of the seventh ES packet to a preset type to obtain a sixth code stream; Encapsulating the eighth ES packet according to the target container format and setting the type of the eighth ES packet as a media type to obtain a seventh code stream; The sixth code stream and the seventh code stream are encapsulated according to the target transmission protocol to obtain the first code stream.

18. The method according to claim 17, characterized in that The first code stream further includes an index file, the index file includes index information of the sixth code stream and index information of the seventh code stream, and the index information of the sixth code stream is located before the index information of the seventh code stream.

19. The method according to claim 17, characterized in that The first code stream also includes an index file; The attribute information of the initialization tag in the index file includes the index information of the sixth code stream.

20. The method according to claim 12, characterized in that The signal to be encoded is M frames, the second code stream is M, the first ES packets are M, the model information includes N information groups, the third code stream includes N sub-code streams, the N information groups correspond to the N sub-code streams one by one, the M first ES packets include N ninth ES packets and MN tenth ES packets, M and N are positive integers, and M is greater than or equal to N; The encapsulating the third code stream and the second code stream into a first ES packet includes: Encapsulating the N sub-streams and the N second streams into N ninth ES packets, and using the other MN second streams as MN tenth ES packets; The ninth ES packet includes a sub-stream and a second stream, and the tenth ES packet includes a second stream.

21. The method according to any one of claims 2 or 3 or 11 to 20, characterized in that The third code stream includes the description information of the model information and the model information, or, The third code stream includes description information of the model information and encoded data of the model information.

22. The method according to any one of claims 1 to 21, characterized in that The model information includes model structure information or model parameters.

23. The method according to any one of claims 1 to 22, characterized in that The preset identification value is also used to characterize the type of the decoding model.

24. A coding device applied to a first electronic device, characterized in that: The device comprises: A signal acquisition module, used to acquire a signal to be encoded; The encoding module is used to generate a first code stream, the first code stream includes a second code stream and a model identifier, the second code stream is obtained by encoding the signal to be encoded based on the encoding model, the value of the model identifier is a preset identifier value, and the preset identifier value is used to instruct the second electronic device to obtain model information of a decoding model, and the decoding model corresponds to the encoding model.

25. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the encoding method executed by any first electronic device as claimed in claim 1 to claim 23.

26. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method described in any one of claims 1 to claim 23 are executed.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the encoding method executed by any first electronic device as claimed in claim 1 to claim 23.

28. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 23 are performed.

29. A device for storing a code stream, characterized in that: The device comprises: a receiver and at least one storage medium, The receiver is used to receive a code stream; The at least one storage medium is used to store the code stream; The code stream is a first code stream generated according to the encoding method according to any one of claims 1 to claim 23.

30. A device for transmitting a code stream, characterized in that: The device comprises: a transmitter and at least one storage medium, The at least one storage medium is used to store a code stream, wherein the code stream is a first code stream generated according to the encoding method according to any one of claims 1 to 23; The transmitter is used to obtain the code stream from the storage medium and send the code stream to the terminal side device through the transmission medium.

31. A system for distributing code streams, characterized in that: The system comprises: at least one storage medium, used to store at least one code stream, wherein the at least one code stream is a first code stream generated according to the encoding method according to any one of claims 1 to 23, The streaming media device is used to obtain a target code stream from the at least one storage medium and send the target code stream to the terminal side device, wherein the streaming media device includes a content server or a content distribution server.

Citation Information

Cited By

  • Encoding method and electronic device

    EP4683320A1

  • Encoding method and electronic device

    WO2024244902A1