Information transmission method and device, storage medium, equipment and program product

Abstract

The application discloses an information transmission method and device, a storage medium, equipment and a program product, which are applied to audio live broadcast, video live broadcast, interactive live broadcast, virtual gift and scroll, online education live broadcast and e-commerce live broadcast scenes. The method comprises the following steps: obtaining extended information to be transmitted, the extended information being additional information to be transmitted together with live audio and / or live video; encapsulating the extended information into an information carrier to obtain a target data frame corresponding to the extended information, the information carrier being a low-code-rate medium; inserting the target data frame into a data stream corresponding to the live audio and / or live video to obtain a target data stream; and transmitting the target data stream to a second terminal. According to the application, the extended information to be transmitted together with the live audio and / or live video is encapsulated into a low-code-rate information carrier, and then inserted into a corresponding data stream and transmitted, so that the extended information is efficiently transmitted without significantly increasing bandwidth occupation, and the live broadcast fluency and user experience are ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an information transmission method, apparatus, storage medium, device, and program product. Background Technology

[0002] With the rapid development of live streaming technology, audio and video live streaming has become the mainstream interactive method. However, existing live streaming technology mainly relies on the transmission of audio and video, lacking effective support for additional information. For example, during a live stream, the host may need to transmit some additional information to the audience (such as quiz results, interactive instructions, etc.), but this information usually cannot be transmitted synchronously with the audio and video streams, resulting in a poor user experience. Summary of the Invention

[0003] This application provides an information transmission method, apparatus, storage medium, device, and program product. By encapsulating extended information transmitted along with live audio and / or live video in a low-bitrate information carrier, and then inserting the corresponding data stream and transmitting it, the extended information can be transmitted efficiently without significantly increasing bandwidth usage, thus ensuring live streaming smoothness and user experience.

[0004] On one hand, embodiments of this application provide an information transmission method applied to a first terminal, the method comprising:

[0005] Obtain the extended information to be transmitted, which is additional information transmitted along with the live audio and / or live video;

[0006] The extended information is encapsulated into an information carrier to obtain the target data frame corresponding to the extended information, wherein the information carrier is a low bit rate medium.

[0007] The target data frame is inserted into the data stream corresponding to the live audio and / or live video to obtain the target data stream;

[0008] The target data stream is sent to the second terminal.

[0009] On the other hand, embodiments of this application provide an information transmission device applied to a first terminal, the device comprising:

[0010] An acquisition unit is used to acquire extended information to be transmitted, wherein the extended information is additional information transmitted together with live audio and / or live video;

[0011] An encapsulation unit is used to encapsulate the extended information into an information carrier to obtain a target data frame corresponding to the extended information, wherein the information carrier is a low bit rate medium.

[0012] An insertion unit is used to insert the target data frame into the data stream corresponding to the live audio and / or live video to obtain the target data stream;

[0013] The sending unit is used to send the target data stream to the second terminal.

[0014] In some embodiments, the information carrier includes at least one of a still image, a blank frame, a solid color frame, or a low-resolution image.

[0015] In some embodiments, the static image is a solid color image.

[0016] In some embodiments, when the encapsulation unit encapsulates the extended information into the information carrier to obtain the target data frame corresponding to the extended information, it is specifically used for:

[0017] The extended information is encapsulated into a supplemental enhancement information (SEI) frame;

[0018] The SEI frame is encapsulated into a target network abstraction layer data unit (NALU) of a preset type;

[0019] The target NALU is inserted into the encoded data of the information carrier to obtain the target data frame corresponding to the extended information.

[0020] In some embodiments, when the encapsulation unit is used to insert the target NALU into the encoded data of the information carrier, it is specifically used for:

[0021] The target NALU is inserted before the key frame or prediction frame of the encoded data of the information carrier.

[0022] In some embodiments, the information carrier employs a high compression ratio and / or low frame rate encoding method.

[0023] In some embodiments, when the insertion unit inserts the target data frame into the data stream corresponding to the audio and / or video to obtain the target data stream, it is specifically used for:

[0024] The target data frame is inserted into the data stream corresponding to the live audio and / or live video based on a preset transmission frequency to obtain the target data stream.

[0025] In some embodiments, the preset transmission frequency is per frame transmission, and the insertion unit, when used to insert the target data frame into the data stream corresponding to the live audio and / or live video based on the preset transmission frequency to obtain the target data stream, is specifically used for:

[0026] The target data frame is inserted into each frame of the data stream corresponding to the live audio and / or live video to obtain the target data stream.

[0027] In some embodiments, the preset transmission frequency is periodic transmission, and the insertion unit, when used to insert the target data frame into the data stream corresponding to the live audio and / or live video based on the preset transmission frequency to obtain the target data stream, is specifically used for:

[0028] The target data frame is inserted into the data stream corresponding to the live audio and / or live video every N frames, where N is an integer greater than 1.

[0029] In some embodiments, before encapsulating the extended information into the information carrier, the information transmission device is further configured to:

[0030] The extended information is encrypted.

[0031] In some embodiments, the information transmission device is further configured to:

[0032] The insertion position of the target data frame when it is inserted into the data stream corresponding to the live audio and / or live video is marked.

[0033] On the other hand, an embodiment of this application provides a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the information transmission method as described in any of the above embodiments.

[0034] On the other hand, an embodiment of this application provides a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the information transmission method as described in any of the above embodiments by calling the computer program stored in the memory.

[0035] On the other hand, an embodiment of this application provides a computer program product, including computer instructions, which, when executed by a processor, implement the information transmission method as described in any of the above embodiments.

[0036] The scheme provided in this application, which involves obtaining extended information to be transmitted (which is additional information transmitted along with live audio and / or live video), encapsulating the extended information into an information carrier to obtain a target data frame corresponding to the extended information (the information carrier being a low-bitrate medium), inserting the target data frame into the data stream corresponding to the live audio and / or live video to obtain a target data stream, and sending the target data stream to a second terminal, achieves efficient transmission of extended information without significantly increasing bandwidth usage. This also allows all viewers to obtain the extended information within the same relative timeframe, ensuring live stream smoothness and user experience. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the architecture of the information transmission system provided in the embodiments of this application.

[0039] Figure 2 This is a schematic diagram of the audio live streaming interface provided in an embodiment of this application.

[0040] Figure 3 This is a first flowchart illustrating the information transmission method provided in an embodiment of this application.

[0041] Figure 4 This is a schematic diagram illustrating the activation of the extended information transmission function provided in an embodiment of this application.

[0042] Figure 5 This is a schematic diagram illustrating the selection of static images provided in an embodiment of this application.

[0043] Figure 6 This is a data format diagram of the SEI data provided in the embodiments of this application.

[0044] Figure 7 This is a data format diagram of NALU data provided in the embodiments of this application.

[0045] Figure 8 This is a second flowchart illustrating the information transmission method provided in an embodiment of this application.

[0046] Figure 9 This is a schematic diagram of the structure of the information transmission device provided in the embodiments of this application.

[0047] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] This application provides an information transmission method, apparatus, storage medium, device, and program product. Exemplarily, the information transmission method of this application can be executed by a computer device, which can be a terminal or a server, etc. The terminal can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, personal computer (PC), smart vehicle terminal, etc. The terminal may also include a client, which can be a video client, shopping application client, reading application client, browser client, or instant messaging client, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0050] The embodiments of this application can be applied to scenarios such as audio live streaming, video live streaming, interactive live streaming, virtual gifts and bullet comments, online education live streaming, and e-commerce live streaming.

[0051] First, some of the nouns or terms that appear in the description of the embodiments of this application are explained as follows:

[0052] Still images: Still images are a form of image display that does not produce dynamic effects in terms of visual presentation over time. From a technical perspective, it is a two-dimensional image matrix composed of a series of pixels arranged in a specific manner and with specific color values. In this matrix, the color, brightness, and other attributes of each pixel are fixed and do not change over time like dynamic images or videos.

[0053] Extended Information: In the field of audio and video transmission, extended information refers to additional data that is interconnected with and transmitted in tandem with the main audio and video stream data. This information is not the core content of the audio and video stream, but rather content added to enhance its expressiveness, functionality, or provide more auxiliary information. From a transmission perspective, extended information is strictly synchronized with the audio and video stream data on the timeline. Through specific encoding and encapsulation technologies, it can be transmitted along with the audio and video stream, ensuring that the extended information and the corresponding audio and video content are time-synchronized upon arrival at the receiving end. This characteristic enables extended information to achieve functions with extremely high time synchronization requirements. For example, in live streaming scenarios, the broadcaster can send real-time interactive information, text descriptions, special effects commands, real-time data statistics, and other extended information related to the current live content. After receiving this extended information, viewers can perform corresponding operations or displays at the same relative point in time as the content presented by the broadcaster, thereby achieving a richer and more precise interactive experience, such as real-time quizzes, interactive voting, and augmented reality effects triggered at specific times.

[0054] Audio live streaming: Audio live streaming is a form of live broadcasting that transmits information in real time solely through sound signals. During an audio live stream, the broadcaster uses audio capture devices, such as microphones, to capture and convert sound signals into digitally transmittable audio data. This audio data is then encoded and transmitted in real-time to a server via network protocols, from where it is distributed to individual viewers.

[0055] Supplemental Enhancement Information (SEI): SEI is a data unit in video coding standards (such as H.264 / AVC, H.265 / HEVC, etc.) used to carry additional information related to the video stream. An SEI frame is not a standalone video frame, but rather additional metadata, typically embedded in the video stream, used to assist in decoding, display, or other processing.

[0056] Network Abstraction Layer Unit (NALU): NALU is an important concept in video coding, especially in standards such as H.264 / AVC and H.265 / HEVC. NALU is the basic unit for encoding video data, used for encapsulating and decapsulating video data during network transmission or storage.

[0057] The information transmission and other technologies involved in the solutions provided in this application are specifically illustrated through the following embodiments. These will be described in detail below. It should be noted that the order of description of the following embodiments is not intended to limit the priority of the embodiments.

[0058] Audio live streaming in related technologies primarily captures the broadcaster's audio signal, encodes and compresses it, and then transmits it to the audience. In terms of data transmission, it focuses only on the audio signal, neglecting the synchronous transmission of extended information. During the live stream, its system architecture is designed solely around the acquisition, encoding, transmission, and decoding of audio signals, lacking modules and mechanisms for processing extended information. Therefore, in pure audio live streaming, due to the lack of support for transmitting extended information, when the live stream requires the transmission of extended information, it cannot be encoded into the audio and video streams for synchronous transmission. For example, in audio live streaming of a quiz game, the solutions in related technologies cannot transmit the answer results to the audience.

[0059] Furthermore, even when attempting to transmit extended information, the technology cannot guarantee the time synchronization between the extended information and the audio stream. For example, when playing a quiz game in a live audio broadcast, the answers need to be transmitted synchronously to all viewers to ensure that viewers with different delays have the same relative time (relative to the broadcaster's time) to participate fairly. The current technology cannot meet this requirement, resulting in viewers with different delays being unable to participate fairly within the same relative time.

[0060] This technical solution successfully solves this problem by introducing an information carrier to carry extended information, ensuring that the extended information is transmitted synchronously with the audio stream, meeting the needs of different live streaming scenarios for extended information transmission, and at the same time, not significantly increasing the bit rate due to the transmission of extended information, thus ensuring a smooth viewing experience for the live stream.

[0061] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of an information transmission system provided in an embodiment of this application. The system can implement an information transmission method. The information transmission system includes a first terminal 1000, a server 2000, and a second terminal 3000.

[0062] The first terminal 1000 and the second terminal 3000 communicate directly or indirectly with the server 2000 via a wireless network or a wired network.

[0063] The first terminal 1000 has an application installed and running that supports live audio and / or live video streaming. Please refer to [link / reference]. Figure 2 , Figure 2This is a schematic diagram of the audio live streaming interface provided in this application embodiment. During audio live streaming, only sound signals are used for real-time information transmission, and the live streaming interface only displays the live streaming cover image. The first terminal 1000 is used by the live streaming content creator (such as a broadcaster) to create audio live streaming and / or video live streaming. The first terminal 1000 is mainly used to acquire extended information to be transmitted. This extended information, as supplementary information transmitted along with the live audio and / or live video, covers diverse supplementary content in the live streaming scenario, such as product details, interaction rules, and event introductions. The first terminal 1000 encapsulates the acquired extended information into an information carrier of a low bitrate medium, generates a corresponding target data frame, and then inserts the target data frame into the data stream corresponding to the live audio and / or live video. Finally, the formed target data stream is sent to the server 2000.

[0064] Server 2000 includes at least one of the following: a single server, multiple servers, a cloud computing platform, or a virtualization center. Server 2000 provides backend services for applications supporting live audio and / or live video streaming, and possesses powerful data processing and forwarding capabilities. On one hand, it receives target data streams from the first terminal 1000 and performs processing such as caching, verification, and integration on the data; on the other hand, according to requests from the second terminal 3000, it forwards the processed target data streams to the corresponding receiving terminals.

[0065] The second terminal 3000 has an application installed and running that supports receiving live audio and / or live video content. The second terminal 3000 requests live data streams from the server 2000 and receives target data streams containing extended information forwarded by the server. After receiving the target data stream, the second terminal 3000 parses the target data stream, separating the live audio and / or live video, as well as the extended information, and finally presents this content to the user, allowing viewers to obtain the corresponding extended information while watching the live stream.

[0066] Please see Figure 3 , Figure 3 This is a first flowchart illustrating an information transmission method provided in an embodiment of this application. In this embodiment, the information transmission method is described using an example where the method is executed by a first terminal. The method may include the following steps 110 to 140:

[0067] Step 110: Obtain the extended information to be transmitted. The extended information is additional information transmitted along with the live audio and / or live video.

[0068] The first terminal has an extended information transmission function. When the first terminal activates the extended information transmission function, the first terminal obtains the extended information to be transmitted. The extended information is additional information transmitted along with the live audio and / or live video, such as interactive instructions, answer results, live status updates, etc.

[0069] In some embodiments, the first terminal may initiate the extended information transmission function in response to the broadcaster's extended information transmission operation.

[0070] Figure 4 This is a schematic diagram illustrating the activation of the extended information transmission function provided in the embodiments of this application, such as... Figure 4 As shown, in other embodiments, the first terminal may poll in real time whether it needs to transmit extended information, determine whether it needs to transmit extended information, start the extended information transmission function and transmit the extended information if it is determined that extended information needs to be transmitted, and turn off the extended information transmission function (if the extended information transmission function is in the open state) and perform the next polling.

[0071] In some embodiments, the extended information can be manually entered by the broadcaster through a specific information input area on the first terminal. For example, when the broadcaster clicks the "Send Answer Result" button, the first terminal generates the answer result as extended information.

[0072] In other embodiments, the extended information can also be automatically acquired by the first terminal. For example, when interactive elements are involved, such as quiz games, information such as answering results and leaderboards will also be acquired by the first terminal as extended information. The first terminal will poll in real time whether such extended information needs to be transmitted. If the first terminal detects that the results need to be published after the quiz is over, it will immediately acquire the relevant answering result data as extended information.

[0073] Step 120: Encapsulate the extended information into an information carrier to obtain the target data frame corresponding to the extended information. The information carrier is a low bit rate medium.

[0074] The first terminal encapsulates the extended information into an information carrier to obtain the target data frame corresponding to the extended information. The information carrier is a low bit rate medium.

[0075] In some embodiments, the information carrier includes at least one of a still image, a blank frame, a solid color frame, or a low-resolution image.

[0076] A blank frame is a frame that does not contain any image data. It is typically a completely transparent frame or an undefined frame, containing no pixel data or pixel data with default values ​​(e.g., fully transparent). Blank frames are often used as placeholders or padding, for example, inserting blank frames into a video stream to maintain the frame rate. When extremely low bitrates are required, blank frames can serve as information carriers to transmit extended information.

[0077] A solid color frame is a frame composed entirely of a single color (such as pure black, pure white, or pure red). All pixels in a solid color frame have the same color value, resulting in extremely small data size and a very high compression ratio. During decoding, a solid color frame appears as a single-color image. Solid color frames can be used for low-bitrate transmission, such as in live audio broadcasts as information carriers for transmitting extended information.

[0078] These types of information carriers, due to their relatively small data volume, meet the requirements of low bit rate media and can effectively reduce bandwidth consumption during transmission.

[0079] In some embodiments, the static image is a solid color image.

[0080] In some embodiments, static images can also be used as live stream cover images or custom images for the broadcaster.

[0081] like Figure 5 As shown, Figure 5 This illustration shows the selection of static images provided in the embodiments of this application. Specifically, when selecting a static image as an information carrier, if the static image is only used to carry extended information and not for display, a pure black image can be selected as the static image used as the information carrier. This is because a pure black image can achieve a high compression ratio, effectively reducing the bit rate and the amount of data transmitted. For example, in a live broadcast of a poetry recitation consisting solely of audio, selecting a pure black static image as the information carrier will not distract the audience from the audio and can transmit extended information, such as the analysis of the poem and the introduction of the author, at a low bit rate. If the static image needs to be used for display, a custom image selected by the broadcaster can be used as the static image used as the information carrier. For example, in a live music audio broadcast, selecting the album cover of a song as the information carrier of extended information can both display the theme and carry extended information.

[0082] In some embodiments, before encapsulating extended information into an information carrier, the method further includes:

[0083] The extended information is encrypted.

[0084] In some embodiments, encrypting the extended information includes:

[0085] A dynamic key is generated using a preset encryption algorithm. The extended information is then encrypted using the dynamic key to obtain the encrypted extended information. The encrypted extended information is then bound to the index information of the dynamic key. The index information is used by the second terminal to obtain the decryption key.

[0086] In this embodiment, the encrypted extended information cannot be tampered with or stolen by intermediate nodes, ensuring the security of sensitive data (such as payment links and privacy instructions). Furthermore, by optimizing the encryption algorithm, the increase in the amount of encrypted extended information can be ensured to be controllable.

[0087] In some embodiments, encapsulating extended information into an information carrier to obtain a target data frame corresponding to the extended information may include the following steps 1201 to 1203:

[0088] Step 1201: Encapsulate the extended information into a supplementary enhancement information (SEI) frame.

[0089] The first terminal encapsulates the acquired extended information into SEI frames according to the format specifications for Supplemental Enhancement Information (SEI) in video coding standards, such as the widely used H.264 or H.265 standards.

[0090] More specifically, when the first terminal encapsulates the extended information into an SEI frame, it adds the extended information to the custom data of the SEI frame.

[0091] Please see Figure 6 , Figure 6 This diagram illustrates the data format of SEI data provided in this application embodiment. SEI data includes a start code, a Network Abstraction Layer (NAL) type, a Payload type, a Universally Unique Identifier (UUID), a custom data length, custom data, and an end alignment code. The start code identifies the beginning of a NALU, with common values ​​such as 0x000001 or 0x00000001. The NAL type specifies the type of NALU unit. The Payload type specifies the type of specific payload data in the SEI information; in the H.264 standard, the payload type value is 0x05, occupying one byte. The Universally Unique Identifier serves as a service-defined identifier, used to distinguish extended information from different services. The custom data length is calculated based on the custom data, and the custom data's type and format are specified by the service provider. The end alignment code ensures that data is aligned according to specific rules during storage or transmission, guaranteeing clear data boundaries and facilitating parsing and processing; in the SEI data format, 0x80 is typically used as the end alignment code. In this application embodiment, the custom data is the extended information.

[0092] For example, in e-commerce audio live streaming, if the extended information is the product's discount code information, the first terminal will organize the product's discount code information into a specific data format as the custom data of the SEI frame according to the above SEI frame syntax structure, and add the necessary metadata to generate a standard-compliant SEI frame.

[0093] Step 1202: Encapsulate the SEI frame into a target network abstraction layer data unit (NALU) of a preset type.

[0094] The first terminal encapsulates the SEI frame into a target NALU of a preset type.

[0095] Please see Figure 7 , Figure 7 This diagram illustrates the data format of NALU data provided in this embodiment. A NALU includes a start code, a header, and a body. The start code identifies the specific byte sequence at the beginning of the NALU; common start codes are 0x000001 or 0x00000001 in the H.264 standard. The header contains a series of key information, such as the NALU type (identifying the data type carried by the NALU, such as keyframe, prediction frame, supplementary enhancement information, etc.) and a priority identifier (used to determine the importance of the data unit during network transmission, so as to determine the data processing order in case of network congestion, etc.). The body is the part that carries the actual valid data, and the data content depends on the NALU type. For example, when the NALU type represents a keyframe (I-frame), the NALU body contains the complete image encoded data of the keyframe; if it is a SEI type NALU, the NALU body contains the specific information of the SEI, such as custom data, timestamps, and other auxiliary information.

[0096] Specifically, during video encoding, the video is encoded into multiple macroblocks, and then the macroblocks are further divided into multiple NALUs. After the extended information is encapsulated into supplementary enhancement information (SEI) frames, the resulting SEI frames are encapsulated into NALUs of a specific type. The first terminal adds NALU header information to the generated SEI frames according to the corresponding standard requirements, including NALU type identifier (clearly indicating that this NALU carries SEI information), priority identifier, etc., so that the SEI frame becomes a complete NALU unit that can be effectively transmitted in the network, i.e., the target NALU.

[0097] Step 1203: Insert the target NALU into the encoded data of the information carrier to obtain the target data frame corresponding to the extended information.

[0098] The first terminal inserts the target NALU into the encoded data of the information carrier to obtain the target data frame corresponding to the extended information.

[0099] In some embodiments, inserting the target NALU into the encoded data of the information carrier includes inserting the target NALU before a keyframe or prediction frame of the encoded data of the information carrier.

[0100] Taking a still image as an information carrier as an example, the first terminal first acquires the encoded data of the still image and finds the keyframes (for still images, keyframes usually contain the main information of the image) or prediction frames (if the image has a predictive encoded part). For example, if the information carrier is a pure black image encoded with a high compression ratio JPEG, the first terminal inserts the encapsulated target NALU before the keyframes of the encoded data of the pure black image encoded with a high compression ratio JPEG.

[0101] This insertion method ensures that the extended information is closely related to the encoded data of the information carrier, and that it is processed in the correct order during subsequent transmission and parsing.

[0102] In some embodiments, the information carrier employs a high compression ratio and / or low frame rate encoding method.

[0103] Specifically, to reduce the bitrate of live streaming transmission, a high compression ratio and / or low frame rate encoding scheme is selected when encoding the information carrier. For example, an H.264 encoder is used, with a high compression ratio parameter "ultrafast" and a low frame rate such as 1fps.

[0104] Step 130: Insert the target data frame into the data stream corresponding to the live audio and / or live video to obtain the target data stream.

[0105] After the first terminal obtains the target data frame, it inserts the target data frame into the data stream corresponding to the live audio and / or live video to form the target data stream.

[0106] The solution provided in this application enables the extended information to be transmitted synchronously with the live audio and / or live video, ensuring that all viewers receive the extended information within the same relative time frame. This synchronous transmission mechanism is particularly suitable for live streaming scenarios with high time synchronization requirements, such as quiz games, real-time voting, and interactive commands. For example, in an audio-based live quiz game, the host's answers can be transmitted synchronously to all viewers, ensuring that all viewers receive the answers at the same time and avoiding unfair interaction caused by network latency.

[0107] In some embodiments, inserting the target data frame into the data stream corresponding to the audio and / or video to obtain the target data stream includes:

[0108] The target data frame is inserted into the data stream corresponding to the live audio and / or live video based on the preset transmission frequency to obtain the target data stream.

[0109] The specific setting of the preset transmission frequency needs to take into account many factors, such as the nature of the live broadcast content, the network bandwidth, and the urgency and timeliness requirements of the extended information.

[0110] In some embodiments, the preset transmission frequency is per frame transmission. Based on the preset transmission frequency, the target data frame is inserted into the data stream corresponding to the live audio and / or live video to obtain the target data stream, including:

[0111] The target data frame is inserted into each frame of the data stream corresponding to the live audio and / or live video to obtain the target data stream.

[0112] Specifically, in live streaming scenarios where the real-time requirements for extended information are extremely high, such as when the results of an audio-based quiz game are announced, the host needs to convey the latest quiz results, leaderboards, and other extended information to the audience as soon as possible. In this case, when the first terminal generates the live audio data stream, it will insert a target data frame carrying extended information at an appropriate position in each frame of data (e.g., before the relevant data identifier of the audio frame) to ensure that the audience can obtain the latest key information in a timely manner.

[0113] In some embodiments, the preset transmission frequency is periodic transmission. Based on the preset transmission frequency, the target data frame is inserted into the data stream corresponding to the live audio and / or live video to obtain the target data stream, including:

[0114] Insert the target data frame every N frames in the data stream corresponding to the live audio and / or live video, where N is an integer greater than 1.

[0115] In typical audio live streaming scenarios, such as music sharing live streams, periodic transmission can be used for relatively stable extended information with low update frequency, such as the background of a song's creation and the singer's basic information. For example, if N is set to 15, when the first terminal generates the target data stream, it inserts the target data frame before the 16th frame every 15 frames of audio data.

[0116] This method of generating target data streams can ensure that extended information is periodically delivered to the audience, while also reducing the network transmission burden to a certain extent and avoiding excessive bandwidth consumption due to frequent insertion of target data frames, which would affect the smooth transmission of live audio.

[0117] In some embodiments, the preset transmission frequency is determined based on the processing capability of the second terminal, which includes, but is not limited to, device model, CPU clock speed, network bandwidth, etc.

[0118] Specifically, high transmission frequencies (such as per frame) can be used for high-performance devices to ensure real-time information transmission, while low transmission frequencies (such as periodic transmission) can be used for low-performance devices to ensure priority playback of the main audio and video streams.

[0119] In this embodiment, adjusting the transmission frequency of the target data frame according to the performance of the second device can adapt to second terminals with different configurations, reduce the system pressure on the second terminal caused by extended information transmission, and avoid excessive redundancy while ensuring the effectiveness of information transmission, thereby reducing bandwidth and computing resource consumption.

[0120] In some embodiments, the method further includes:

[0121] Mark the insertion position when inserting the target data frame into the corresponding data stream of live audio and / or live video.

[0122] In some embodiments, a preset identifier can be used for labeling, which is used to guide the second terminal to locate the target data frame.

[0123] In this embodiment, the insertion position of the target data frame is marked, so that the second terminal can locate the target data frame without traversing the entire data stream, thereby improving the parsing speed of the second terminal.

[0124] Step 140: Send the target data stream to the second terminal.

[0125] After the first terminal generates the target data stream, it transmits the target data stream to the second terminal through its network communication module based on a preset network protocol.

[0126] After receiving the target data stream, the second terminal uses its decoding and data processing capabilities to parse and reconstruct it. It separates the live audio and / or live video data, SEI information including extended information, and the information carrier. The second terminal decodes and plays the live audio and / or live video data, allowing viewers to hear the live sound and / or see the live picture. Whether to display the information carrier depends on the broadcaster's settings. If the broadcaster chooses not to display the information carrier, the second terminal ignores it; if it is set to display, the second terminal decodes the information carrier, but since the information carrier is usually covered by a blurred cover image, the actual display effect may only be a blurred background. During the decapsulation process, the second terminal extracts extended information from the SEI information and executes corresponding business functions based on the type and format of the extended information. For example, if the extended information is a quiz result, the second terminal displays the result on the live interface to inform viewers of their answer; if the extended information is a product link, the second terminal provides corresponding redirection or display functions to facilitate viewers' purchase.

[0127] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0128] The present application provides a scheme for obtaining extended information to be transmitted, wherein the extended information is additional information transmitted along with live audio and / or live video; encapsulating the extended information into an information carrier to obtain a target data frame corresponding to the extended information, wherein the information carrier is a low bitrate medium; inserting the target data frame into the data stream corresponding to the live audio and / or live video to obtain a target data stream; and sending the target data stream to a second terminal. By encapsulating the extended information transmitted along with the live audio and / or live video into a low bitrate information carrier, and then inserting it into the corresponding data stream and transmitting it, the scheme achieves efficient transmission of extended information without significantly increasing bandwidth usage, thus ensuring live streaming smoothness and user experience.

[0129] The information transmission method provided in the embodiments of this application will be described below. Please refer to [link / reference]. Figure 8 , Figure 8 A second flowchart illustrating the information transmission method provided in this application embodiment includes the following steps 201 to 204:

[0130] Step 201: The broadcaster terminal (i.e., the first terminal) responds to the broadcaster by performing an extended information transmission operation and obtains the extended information input by the broadcaster.

[0131] Step 202: The broadcaster selects a suitable static image as the information carrier to carry extended information.

[0132] Step 203: The broadcaster encapsulates the extended information into the encoded data of the static image to obtain the target data frame, inserts the target data frame into the data stream corresponding to the live audio to obtain the target data stream, and sends the target data stream to the viewer.

[0133] Step 204: The viewer (i.e., the second terminal) parses the target data stream and obtains extended information.

[0134] The specific implementation methods corresponding to each step in the above-described interactive process in this application embodiment can be found in the embodiment content corresponding to the first process diagram above. For the sake of brevity, they will not be repeated here.

[0135] To facilitate better implementation of the information transmission method of this application embodiment, this application embodiment also provides an information transmission device, which is applied to a first terminal. Please refer to... Figure 9 , Figure 9 This is a schematic diagram of the structure of an information transmission device provided in an embodiment of this application. The information transmission device 300 may include:

[0136] The acquisition unit 310 is used to acquire the extended information to be transmitted, which is additional information transmitted together with the live audio and / or live video.

[0137] The encapsulation unit 320 is used to encapsulate extended information into an information carrier to obtain the target data frame corresponding to the extended information. The information carrier is a low bit rate medium.

[0138] The insertion unit 330 is used to insert the target data frame into the data stream corresponding to the live audio and / or live video to obtain the target data stream.

[0139] The sending unit 340 is used to send the target data stream to the second terminal.

[0140] In some embodiments, the information carrier includes at least one of a still image, a blank frame, a solid color frame, or a low-resolution image.

[0141] In some embodiments, the static image is a solid color image.

[0142] In some embodiments, when the encapsulation unit 320 encapsulates extended information into an information carrier to obtain a target data frame corresponding to the extended information, it is specifically used for:

[0143] The extended information is encapsulated into a supplemental enhancement information (SEI) frame;

[0144] Encapsulate the SEI frame into a Target Network Abstraction Layer Data Unit (NALU) of a preset type;

[0145] The target NALU is inserted into the encoded data of the information carrier to obtain the target data frame corresponding to the extended information.

[0146] In some embodiments, when the encapsulation unit 320 is used to insert the target NALU into the encoded data of the information carrier, it is specifically used for:

[0147] Insert the target NALU before the key frame or prediction frame of the encoded data of the information carrier.

[0148] In some embodiments, the information carrier employs a high compression ratio and / or low frame rate encoding method.

[0149] In some embodiments, when the insertion unit 330 inserts the target data frame into the data stream corresponding to the audio and / or video to obtain the target data stream, it is specifically used for:

[0150] The target data frame is inserted into the data stream corresponding to the live audio and / or live video based on the preset transmission frequency to obtain the target data stream.

[0151] In some embodiments, the preset transmission frequency is transmission per frame. When the insertion unit 330 inserts the target data frame into the data stream corresponding to the live audio and / or live video based on the preset transmission frequency to obtain the target data stream, it is specifically used for:

[0152] The target data frame is inserted into each frame of the data stream corresponding to the live audio and / or live video to obtain the target data stream.

[0153] In some embodiments, the preset transmission frequency is periodic transmission. When the insertion unit 330 inserts the target data frame into the data stream corresponding to the live audio and / or live video based on the preset transmission frequency to obtain the target data stream, it is specifically used for:

[0154] Insert the target data frame every N frames in the data stream corresponding to the live audio and / or live video, where N is an integer greater than 1.

[0155] In some embodiments, before encapsulating extended information into an information carrier, the information transmission device is further configured to:

[0156] The extended information is encrypted.

[0157] In some embodiments, the information transmission device is further configured to:

[0158] Mark the insertion position when inserting the target data frame into the corresponding data stream of live audio and / or live video.

[0159] It should be noted that the functions of each module in the information transmission device 300 in this application embodiment can be referred to the specific implementation of any embodiment in the above method embodiments, and will not be repeated here.

[0160] Each unit in the above-described device can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0161] For example, the information transmission device 300 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the information transmission device 300 can be the terminal or server.

[0162] In some embodiments, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0163] Figure 10 A schematic structural diagram of the computer device provided in the embodiments of this application, such as Figure 10 As shown, the computer device 400 may include: a communication interface 401, a memory 402, a processor 403, and a communication bus 404. The communication interface 401, memory 402, and processor 403 communicate with each other via the communication bus 404. The communication interface 401 is used for data communication between the device 400 and external devices. The memory 402 can be used to store software programs and modules, and the processor 403 runs the software programs and modules stored in the memory 402, such as the software programs for the corresponding operations in the foregoing method embodiments.

[0164] In some embodiments, the processor 403 may invoke software programs and modules stored in the memory 402 to perform the following operations: acquiring extended information to be transmitted, the extended information being additional information transmitted along with live audio and / or live video; encapsulating the extended information into an information carrier to obtain a target data frame corresponding to the extended information, the information carrier being a low bitrate medium; inserting the target data frame into the data stream corresponding to the live audio and / or live video to obtain a target data stream; and sending the target data stream to the second terminal.

[0165] In some embodiments, the computer device 400 may be integrated into a terminal or server that has storage and a processor, thus possessing computing capabilities; or the computer device 400 may be the terminal or server. The terminal may be a smartphone, tablet, laptop, smart TV, smart speaker, wearable smart device, personal computer, or other similar device. The server may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0166] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the methods described above in the embodiments of this application; for brevity, these will not be elaborated further here.

[0167] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0168] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0169] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0170] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0171] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0172] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0174] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0177] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0178] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An information transmission method, characterized in that, Applied to a first terminal, the method includes: Obtain the extended information to be transmitted, which is additional information transmitted along with the live audio and / or live video; The extended information is encapsulated into an information carrier to obtain the target data frame corresponding to the extended information, wherein the information carrier is a low bit rate medium. The target data frame is inserted into the data stream corresponding to the live audio and / or live video to obtain the target data stream; The target data stream is sent to the second terminal.

2. The information transmission method as described in claim 1, characterized in that, The information carrier includes at least one of the following: a static image, a blank frame, a solid color frame, or a low-resolution image.

3. The information transmission method as described in claim 2, characterized in that, The static image is a solid color image.

4. The information transmission method as described in claim 1, characterized in that, The step of encapsulating the extended information into the information carrier to obtain the target data frame corresponding to the extended information includes: The extended information is encapsulated into a supplemental enhancement information (SEI) frame; The SEI frame is encapsulated into a target network abstraction layer data unit (NALU) of a preset type; The target NALU is inserted into the encoded data of the information carrier to obtain the target data frame corresponding to the extended information.

5. The information transmission method as described in claim 4, characterized in that, The step of inserting the target NALU into the encoded data of the information carrier includes: The target NALU is inserted before the key frame or prediction frame of the encoded data of the information carrier.

6. The information transmission method as described in claim 4, characterized in that, The information carrier employs a high compression ratio and / or low frame rate encoding method.

7. The information transmission method as described in claim 1, characterized in that, Inserting the target data frame into the data stream corresponding to the audio and / or video to obtain the target data stream includes: The target data frame is inserted into the data stream corresponding to the live audio and / or live video based on a preset transmission frequency to obtain the target data stream.

8. The information transmission method as described in claim 7, characterized in that, The preset transmission frequency is per frame. Based on the preset transmission frequency, the target data frame is inserted into the data stream corresponding to the live audio and / or live video to obtain the target data stream, including: The target data frame is inserted into each frame of the data stream corresponding to the live audio and / or live video to obtain the target data stream.

9. The information transmission method as described in claim 7, characterized in that, The preset transmission frequency is for periodic transmission. Based on the preset transmission frequency, the target data frame is inserted into the data stream corresponding to the live audio and / or live video to obtain the target data stream, including: The target data frame is inserted into the data stream corresponding to the live audio and / or live video every N frames, where N is an integer greater than 1.

10. The information transmission method as described in claim 1, characterized in that, Before encapsulating the extended information into the information carrier, the method further includes: The extended information is encrypted.

11. The information transmission method as described in claim 1, characterized in that, The method further includes: The insertion position of the target data frame when it is inserted into the data stream corresponding to the live audio and / or live video is marked.

12. An information transmission device, characterized in that, Applied to a first terminal, the device includes: An acquisition unit is used to acquire extended information to be transmitted, wherein the extended information is additional information transmitted together with live audio and / or live video; An encapsulation unit is used to encapsulate the extended information into an information carrier to obtain a target data frame corresponding to the extended information, wherein the information carrier is a low bit rate medium. An insertion unit is used to insert the target data frame into the data stream corresponding to the live audio and / or live video to obtain the target data stream; The sending unit is used to send the target data stream to the second terminal.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the information transmission method as described in any one of claims 1-11.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the information transmission method as described in any one of claims 1-11 by calling the computer program stored in the memory.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the information transmission method according to any one of claims 1-11.

Images