Multimedia stream processing method, apparatus, device, storage medium, and program product

Abstract

The application provides a multimedia stream processing method, device, equipment, storage medium and program product, which is applied to various multimedia stream transmission scenes such as cloud technology, multimedia, communication and video; the multimedia stream processing method comprises the following steps: acquiring a first target stream segment identifier corresponding to each file body block of a plurality of file body blocks of a first multimedia stream, the first target stream segment identifier being used for determining a first target media stream of a target media type, and the first multimedia stream comprising at least two first media streams belonging to the same media type; determining a second target stream segment identifier corresponding to the first target stream segment identifier based on an identifier correspondence relationship between the first stream segment identifier and a second stream segment identifier; and converting the first multimedia stream into a second multimedia stream based on a plurality of second target stream segment identifiers corresponding to the plurality of file body blocks, and the second multimedia stream being used for transmission to a consumer end device. Through the application, resource transmission consumption can be reduced.

Description

Technical Field

[0001] This application relates to multimedia technology in the field of computer applications, and more particularly to a multimedia stream processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In the process of transmitting multimedia streams, media data, including multiple different tracks of the same type, is often split into multiple streams to be transmitted, thereby achieving multi-track transmission through multiplexing. However, multi-track transmission through multiplexing increases the consumption of transmission resources. Summary of the Invention

[0003] This application provides a multimedia stream processing method, apparatus, device, storage medium, and program product that can reduce resource transmission consumption.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a multimedia stream processing method, the method comprising:

[0006] For multiple file blocks of a first multimedia stream, a first target stream segment identifier corresponding to each file block is obtained, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams belonging to the same media type, and the file block is used to store stream segments of the first media stream;

[0007] Based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, a second target stream segment identifier corresponding to the first target stream segment identifier is determined, wherein the second target stream segment identifier includes a second target media stream used to determine the target message type;

[0008] Based on multiple second target stream segment identifiers corresponding to multiple file blocks, the first multimedia stream is converted into a second multimedia stream, wherein the second multimedia stream is used for transmission to a consumer device.

[0009] This application provides another multimedia stream processing method, the method comprising:

[0010] From the second multimedia stream, obtain multiple second target stream segment identifiers, wherein the second target stream segment identifiers are used to determine the second target media stream of the target message type, and the second multimedia stream is transmitted by the production end device;

[0011] Based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, a first target stream segment identifier is determined for each second target stream segment identifier, wherein the first target stream segment identifier is used to determine the first target media stream of the target media type;

[0012] Generate a file block including the identifier of the first target stream segment, and obtain a plurality of file blocks corresponding to a plurality of second target stream segment identifiers, wherein the file blocks are used to store stream segments of the first media stream;

[0013] A first multimedia stream comprising multiple file blocks is generated, wherein the first multimedia stream includes at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

[0014] This application provides a first multimedia stream processing device, the first multimedia stream processing device comprising:

[0015] The first identifier acquisition module is used to acquire a first target stream segment identifier corresponding to each of the multiple file blocks of the first multimedia stream, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams belonging to the same media type, and the file block is used to store stream segments of the first media stream.

[0016] The first identifier acquisition module is further configured to determine a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, wherein the second target stream segment identifier includes a second target media stream used to determine the target message type;

[0017] The first conversion module is used to convert the first multimedia stream into a second multimedia stream based on multiple second target stream segment identifiers corresponding to multiple file blocks, wherein the second multimedia stream is used to transmit to a consumer device.

[0018] In this embodiment of the application, the first multimedia stream processing device further includes a relationship acquisition module, configured to determine at least one media type to be expanded from at least one media type corresponding to the initial correspondence relationship, and perform the following processing on each of the media types to be expanded: expanding the media type to be expanded to obtain at least two expanded first stream identifiers, a media type expansion identifier, at least two expanded second stream identifiers, and a message type expansion identifier; combining one of the expanded first stream identifiers with the initial media type identifier in the initial correspondence relationship, and combining at least one of the expanded first stream identifiers with the media type expansion identifier respectively to obtain at least two first stream segment identifiers; combining one of the expanded second stream identifiers with the initial message type identifier in the initial correspondence relationship, and combining at least one of the expanded second stream identifiers with the message type expansion identifier respectively to obtain at least two second stream segment identifiers; and obtaining the identifier correspondence relationship based on the correspondence relationship between the at least two first stream segment identifiers and the at least two second stream segment identifiers.

[0019] In this application embodiment, the media type includes at least one of the following: audio type, video type, and subtitle type.

[0020] In this embodiment of the application, at least two first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bitrate, resolution, encoding format and language type.

[0021] In this embodiment of the application, the first multimedia stream processing device further includes a data parsing module, configured to terminate the processing of the file block when the target media type identifier in the first target stream segment identifier fails to match in the initial correspondence, wherein the target media type identifier is used to determine the target media type; when the target media type identifier matches successfully in the initial correspondence, the media type initial identifier that matches the target media type identifier in the initial correspondence is determined as the target media type initial identifier, and the first multimedia stream is converted into the second multimedia stream based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence.

[0022] In this embodiment of the application, the container format used for the first multimedia stream is a streaming media format, and the container format used for the second multimedia stream is supported by a real-time messaging protocol.

[0023] In this embodiment of the application, the first multimedia stream processing device further includes a data acquisition module, configured to, for at least one first media stream corresponding to each media type of the multimedia to be transmitted, determine the first target stream segment identifier of each first media stream from the identifier correspondence included in the first stream segment identifiers, wherein the multimedia to be transmitted corresponds to at least one media type; generate a file block including the first target stream segment identifier for each stream segment of each first media stream; and generate a first multimedia stream corresponding to the multimedia to be transmitted from each file block.

[0024] This application provides a second multimedia stream processing device, the second multimedia stream processing device comprising:

[0025] The second identifier acquisition module is used to acquire multiple second target stream segment identifiers from the second multimedia stream, wherein the second target stream segment identifier is used to determine the second target media stream of the target message type, and the second multimedia stream is transmitted by the production end device;

[0026] The second identifier acquisition module is further configured to determine a first target stream segment identifier corresponding to each second target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type;

[0027] The second conversion module is used to generate a file block including the identifier of the first target stream segment, and obtain a plurality of file blocks corresponding to a plurality of the second target stream segment identifiers, wherein the file blocks are used to store stream segments of the first media stream;

[0028] The second conversion module is further configured to generate a first multimedia stream comprising multiple file blocks, wherein the first multimedia stream comprises at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

[0029] In this embodiment, the second multimedia stream processing device further includes a data consumption module, configured to acquire the stream segment corresponding to each file block of the first multimedia stream, thereby obtaining a plurality of stream segments corresponding to the plurality of file blocks; based on the plurality of stream segments, obtain a parsing result of the first multimedia stream; in response to a multimedia consumption request, obtain a data identifier to be consumed; acquire the data to be consumed corresponding to the data identifier to be consumed from the parsing result; and perform data consumption based on the data to be consumed, wherein the data consumption includes at least one of the following: distribution, playback, media processing, and storage.

[0030] This application embodiment provides a production-end device for multimedia stream processing, the production-end device comprising:

[0031] The first memory is used to store computer-executable instructions or computer programs;

[0032] The first processor, when executing computer-executable instructions or computer programs stored in the first memory, implements the multimedia stream processing method for production equipment provided in the embodiments of this application.

[0033] This application provides a consumer device for multimedia stream processing, the consumer device comprising:

[0034] Secondary memory is used to store computer-executable instructions or computer programs;

[0035] The second processor, when executing computer-executable instructions or computer programs stored in the second memory, implements the multimedia stream processing method for consumer devices provided in the embodiments of this application.

[0036] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When executed by a first processor, the computer-executable instructions or computer programs implement the multimedia stream processing method for production devices provided in this application; or, when executed by a second processor, the computer-executable instructions or computer programs implement the multimedia stream processing method for consumer devices provided in this application.

[0037] This application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a first processor, they implement the multimedia stream processing method for consumer devices provided in this application. Alternatively, when the computer-executable instructions or the computer program are executed by a second processor, they implement the multimedia stream processing method for consumer devices provided in this application.

[0038] The embodiments of this application have at least the following beneficial effects: Since each file block used to represent the first multimedia stream includes a first target stream segment identifier, and the first target stream segment identifier is used to determine the first target media stream of the target media type, the first multimedia stream can include at least two first media streams belonging to the same media type, which is a type of multi-track multimedia content; then, based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, the first multimedia stream is converted into a second multimedia stream for transmission; in this way, the transmission of a type of multi-track multimedia stream can be realized through one transmission, which can reduce resource transmission consumption. Attached Figure Description

[0039] Figure 1 This is an exemplary diagram illustrating the splitting of a multimedia stream;

[0040] Figure 2 This is a schematic diagram of the architecture of the multimedia stream processing system provided in the embodiments of this application;

[0041] Figure 3 This is one of the embodiments provided in this application. Figure 2 A schematic diagram of the server structure in the diagram;

[0042] Figure 4 This is one of the embodiments provided in this application. Figure 2 A schematic diagram of the terminal structure in the diagram;

[0043] Figure 5 This is a flowchart illustrating the multimedia stream processing method provided in the embodiments of this application. Figure 1 ;

[0044] Figure 6 This is an exemplary flowchart of obtaining a correspondence provided in an embodiment of this application;

[0045] Figure 7 This is a flowchart illustrating the multimedia stream processing method provided in the embodiments of this application. Figure 2 ;

[0046] Figure 8 This is a flowchart illustrating the multimedia stream processing method provided in the embodiments of this application. Figure 3 ;

[0047] Figure 9 This is an exemplary schematic diagram of multi-track transmission provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] In the following description, the terms "first" and "second" are used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0051] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0052] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0053] 1) Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. The multimedia stream processing method provided in this application embodiment can be applied to the field of cloud technology, such as pull streaming and push streaming processing in the cloud technology field.

[0054] 2) Real-Time Messaging Protocol (RTMP) is a media transport protocol used for real-time data communication. RTMP supports both video-on-demand and live streaming. For example, it can be used for streaming between media content producers and media servers, forwarding between media server clusters, and content distribution from media servers to clients. RTMP has low latency (below the latency threshold) and low deployment cost (below the cost threshold), and can achieve compatibility.

[0055] 3) Streaming media format (Flash Video, FLV) uses a container format for streaming processing; RTMP uses FLV as its container format; where container format refers to the format in which encoded data is packaged.

[0056] 4) Multitrack transmission refers to the transmission of multitrack data including at least two tracks of the same media type, such as the transmission of a multimedia stream including video A, audio B1, and audio B2. Single-track transmission, on the other hand, refers to the transmission of a multimedia stream where each media type includes one track of data, such as the transmission of a multimedia stream including video A and audio B1.

[0057] It should be noted that during multimedia stream transmission, such as when using RTMP and FLV for multimedia information transmission, RTMP and FLV are used for single-track transmission. To achieve multi-track transmission, RTMP is usually replaced with other transport protocols (such as User Datagram Protocol, UDP), and the streaming media format is replaced with other container formats (such as Transport Stream, TS). However, for already deployed services based on RTMP transmission, migrating to other transport protocols involves issues such as cost and service stability, affecting the complexity and efficiency of multi-track transmission.

[0058] Additionally, to achieve multitrack transmission, media data comprising multiple different tracks of the same type can be split into multiple streams to be transmitted, thus enabling multitrack transmission through multi-channel transmission; for example, a multimedia stream can be split into multiple RTMP streams, where each RTMP stream is a single-track transmission; see [link to relevant documentation]. Figure 1 , Figure 1 This is an exemplary diagram illustrating the splitting of a multimedia stream; such as... Figure 1 As shown, the multimedia stream is multi-track media content 1-1, exemplarily illustrating video 1-11, audio 1-12, audio 1-13, and so on. Based on the number of audio tracks, the multi-track media content 1-1 is split into multiple RTMP transmissions, namely video 1-11 and audio 1-12, video 1-11 and audio 1-13, and so on. After transmission 1-2, media processing, secondary distribution, and playback occur at the consumer end 1-3. However, the multi-RTMP transmission process, including the splitting, management, and aggregation of the multimedia stream, as well as the determination of the transmission of each RTMP, is implemented using a scheduling center, increasing the complexity and coupling of the architecture and reducing flexibility. Furthermore, multi-RTMP transmission increases the probability of transmission anomalies, and the fault tolerance processing performed when one RTMP transmission fails while others are normal also increases resource consumption.

[0059] Based on this, embodiments of this application provide a multimedia stream processing method, apparatus, device, computer-readable storage medium, and computer program product, which can reduce resource transmission consumption. The exemplary applications of the production-end device and consumer-end device provided in the embodiments of this application are described below. The production-end device and consumer-end device provided in the embodiments of this application can be implemented as various types of terminals such as smartphones, smartwatches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart in-vehicle devices, portable music players, personal digital assistants, dedicated messaging devices, smart voice interaction devices, portable gaming devices, and smart speakers. They can also be implemented as servers, or a combination of both. The exemplary application when the production-end device is implemented as a server and the consumer-end device is implemented as a terminal will be described below.

[0060] See Figure 2 , Figure 2 This is a schematic diagram of the architecture of the multimedia stream processing system provided in the embodiments of this application; as shown... Figure 2 As shown, to support a multimedia streaming application, in the multimedia streaming system 100, terminals 400 (terminals 400-1 and 400-2 are shown as examples) connect to server 200 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the multimedia streaming system 100 includes a database 500 for providing data support to server 200; and... Figure 2 The example shown illustrates a scenario where the database 500 is independent of the server 200. However, the database 500 can also be integrated into the server 200, and this embodiment does not limit this to any particular case.

[0061] Server 200 is configured to: obtain a first target stream segment identifier corresponding to each file block of the first multimedia stream; determine a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier; convert the first multimedia stream into a second multimedia stream based on the multiple second target stream segment identifiers corresponding to the multiple file blocks; and transmit the second multimedia stream to terminal 400 via network 300.

[0062] Terminal 400 is configured to receive a second multimedia stream sent by server 200 via network 300; obtain multiple second target stream segment identifiers from the second multimedia stream; determine a first target stream segment identifier corresponding to each second target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier; generate a file block including the first target stream segment identifier, thereby obtaining multiple file blocks corresponding to the multiple second target stream segment identifiers; and generate a first multimedia stream including multiple file blocks (exemplarily shown through graphical interfaces 410-1 and 410-2, which include a video stream and multiple audio streams), the first multimedia stream being used for data consumption on terminal 400.

[0063] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0064] See Figure 3 , Figure 3 This is one of the embodiments provided in this application. Figure 2 A schematic diagram of the server structure in the diagram; such as Figure 3 As shown, server 200 includes at least one first processor 210, a first memory 250, and at least one first network interface 220. The various components of server 200 are coupled together via a first bus system 240. It is understood that the first bus system 240 is used to implement communication between these components. In addition to a data bus, bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general designated all buses as the first bus system 240.

[0065] The first processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0066] The first memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The first memory 250 may optionally include one or more storage devices physically located remote from the first processor 210.

[0067] The first memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The first memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0068] In some embodiments, the first memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0069] The first operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0070] The first network communication module 252 is used to reach other electronic devices via one or more (wired or wireless) first network interfaces 220, exemplary first network interfaces 220 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc.

[0071] In some embodiments, the first multimedia stream processing apparatus provided in this application can be implemented in software. Figure 3 A first multimedia stream processing device 255 stored in a first memory 250 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a first identifier acquisition module 2551, a first conversion module 2552, a relationship acquisition module 2553, a data parsing module 2554, and a data acquisition module 2555. These modules are logically linked and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0072] See Figure 4 , Figure 4 This is one of the embodiments provided in this application. Figure 2 A schematic diagram of the terminal structure in the diagram; such as Figure 4As shown, terminal 400 includes at least one second processor 410, a second memory 450, at least one second network interface 420, and a user interface 430. The various components in terminal 400 are coupled together via a second bus system 440. It is understood that the second bus system 440 is used to implement communication between these components. In addition to a data bus, the second bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The various buses are all labeled as the second bus system 440.

[0073] The second processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0075] The second memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The second memory 450 may optionally include one or more storage devices physically located remote from the second processor 410.

[0076] The second memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory, and the volatile memory may be random access memory. The second memory 450 described in the embodiments of this application is intended to include any suitable type of memory.

[0077] In some embodiments, the second memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0078] The second operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0079] The second network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) second network interfaces 420, exemplary second network interfaces 420 including: Bluetooth, wireless compatibility authentication, and Universal Serial Bus, etc.

[0080] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0081] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0082] In some embodiments, the second multimedia stream processing apparatus provided in this application can be implemented in software. Figure 4 A second multimedia stream processing device 455 stored in a second memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a second identifier acquisition module 4551, a second conversion module 4552, and a data consumption module 4553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0083] In some embodiments, the first multimedia stream processing device and the second multimedia stream processing device provided in this application can be implemented in hardware. As an example, the first multimedia stream processing device and the second multimedia stream processing device provided in this application can be processors in the form of hardware decoding processors, which are programmed to execute the multimedia stream processing method provided in this application. For example, the processor in the form of hardware decoding processors can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0084] In some embodiments, the terminal or server can implement the multimedia stream processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as live streaming APPs or instant messaging APPs; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0085] The multimedia stream processing method provided in this application will be described below with reference to exemplary applications and implementations of the production-end equipment and consumer-end equipment provided in the embodiments of this application. Furthermore, the multimedia stream processing method provided in this application is applicable to various multimedia stream transmission scenarios such as cloud technology, multimedia, communication, and video.

[0086] See Figure 5 , Figure 5 This is a flowchart illustrating the multimedia stream processing method provided in the embodiments of this application. Figure 1 The following will combine Figure 5 The steps shown are explained.

[0087] Step 101: For the multiple file blocks of the first multimedia stream, the production-end device obtains the first target stream segment identifier corresponding to each file block.

[0088] In this embodiment, after obtaining the first multimedia stream, the production device transmits it to the consumer device. Since the message format supported by the transmission protocol used during transmission differs from the container format of the first multimedia stream, the first multimedia stream is converted to a message format supported by the transmission protocol. The production device performs the conversion on a file block basis, and for each file block, the first media stream to which the stored stream segment belongs is determined by the first target stream segment identifier in that file block. Thus, the production device first obtains the first target stream segment identifier in each file block and performs the conversion based on the first target stream segment identifier in the file block. The first multimedia stream includes multiple first media streams, and among the multiple first media streams are at least two first media streams belonging to the same media type but on different tracks.

[0089] It should be noted that the first multimedia stream comprises multiple file blocks, which are a sequence of file blocks. Each file block stores a segment of the first media stream, that is, it stores a track content segment under a certain media type. The first media stream is a track content under a certain media type. In addition, the first multimedia stream includes at least two first media streams belonging to the same media type. Thus, the multiple file blocks include at least two stream segments belonging to the same media type. Therefore, the first multimedia stream is multi-track data, that is, at least one of a multimedia stream with multiple audio tracks, a multimedia stream with multiple subtitle tracks, and a multimedia stream with multiple video tracks. Here, the first target stream segment identifier includes a first target stream identifier and a target media type identifier. The first target stream identifier is used to determine the first target media stream (e.g., audio 1, audio 2, video 2, etc.) in combination with the target media type identifier. The target media type identifier is used to determine the target media type (e.g., audio type, video type, subtitle type, etc.). Each file block uses the first target stream identifier and the target media type identifier to jointly determine the first media stream (called track content) under the target media type. In addition, the production-end equipment refers to the electronic device that obtains the first multimedia stream. It can be a multimedia content creator, such as a multimedia content creation client; it can also be a multimedia content processor, such as a multimedia server; etc. The embodiments of this application do not limit this.

[0090] In this embodiment, the media type includes one of the following: audio type, video type, and subtitle type; wherein, audio type indicates that the data carried is audio, video type indicates that the data carried is video, and subtitle type indicates that the data carried is subtitle. Thus, at least one of multiple audio tracks, multiple video tracks, and multiple subtitle tracks can be implemented, wherein multiple audio tracks refer to multiple audio tracks corresponding to the first multimedia stream (i.e., including multiple audio streams), multiple video tracks refer to multiple video tracks corresponding to the first multimedia stream (i.e., including multiple video streams), and multiple subtitle tracks refer to multiple subtitle tracks corresponding to the first multimedia stream (i.e., including multiple subtitle streams).

[0091] In the embodiments of this application, at least two first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bitrate, resolution, encoding format and language type.

[0092] Understandably, when the media attribute is bitrate, multiple audio or video tracks with different bitrates can be implemented; when the media attribute is resolution, multiple audio or video tracks with different resolutions can be implemented; when the media attribute is encoding format, multiple audio or video tracks with different encoding formats can be implemented; when the media attribute is language type, multiple subtitle tracks with different language types can be implemented; and when the media attribute is any combination of bitrate, resolution, encoding format, and language type, multiple tracks with the corresponding media attribute can be implemented. Furthermore, for media attributes that are any combination of bitrate, resolution, and encoding format, the first multimedia stream can provide multimedia content of different qualities to suit different multimedia consumption needs.

[0093] In this application, the container format used for the first multimedia stream is a streaming media format, such as the FLV container format. In this case, the first multimedia stream is an FLV stream.

[0094] Step 102: Based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, the production-end equipment determines the second target stream segment identifier corresponding to the first target stream segment identifier.

[0095] In this embodiment, the production device can obtain an identifier mapping relationship, which represents the correspondence between a first stream segment identifier and a second stream segment identifier. The first stream segment identifier includes a first stream identifier and a media type identifier, and the second stream segment identifier includes a second stream identifier and a message type identifier. The first stream identifier is the stream identifier in the container format corresponding to the first multimedia stream, used to determine the track under the media type in combination with the media type identifier. The second stream identifier is the stream identifier in the message format of the transport protocol, used to determine the track under the message type in combination with the message type identifier. Furthermore, one first stream segment identifier corresponds to one second stream segment identifier. Here, the production device matches the first target stream segment identifier of each file block with each first stream segment identifier in the identifier mapping relationship, and determines the second target stream identifier corresponding to the matched first stream segment identifier as the second target stream segment identifier corresponding to the first target stream segment identifier.

[0096] It should be noted that the second target stream segment identifier includes a second target stream identifier and a target message type identifier. The second target stream identifier is used in conjunction with the target message type identifier to determine the second target media stream of the target message type (e.g., audio 1, audio 2, video 2, etc.). The target message type identifier is used to determine the target message type (e.g., audio type, video type, subtitle type, etc.). Furthermore, in the identifier correspondence, the first stream identifiers for the same media type are different, while the first stream identifiers for different media types can be the same or different; similarly, the second stream identifiers for the same message type are different, while the second stream identifiers for different message types can be the same or different.

[0097] Step 103: The production-end device converts the first multimedia stream into a second multimedia stream based on the identifiers of multiple second target stream segments corresponding to multiple file blocks.

[0098] In this embodiment of the application, the production device obtains multiple second target stream segment identifiers corresponding to multiple file blocks from the second target stream segment identifier corresponding to each file block; then, the production device combines the multiple second target stream segment identifiers and multiple stream segments in the multiple file blocks into a second multimedia stream according to the message format adopted by the transmission protocol; at this time, the conversion from the first multimedia stream to the second multimedia stream is completed.

[0099] It should be noted that the message format of the second multimedia stream is the message format supported by the transmission protocol used to transmit the first multimedia stream; the second multimedia stream is used to transmit to a consumer device; wherein, the consumer device is an electronic device that receives the second multimedia stream, which can be a multimedia content processor, such as a multimedia server; it can also be a multimedia content playback end, such as a multimedia content playback client; etc., and this application embodiment does not limit this. Here, when the production end device is a multimedia content creator, the consumer end device can be a multimedia content processor; when the production end device is a multimedia content processor, the consumer end device can be a multimedia content playback end.

[0100] In this embodiment of the application, when the container format used by the first multimedia stream is a streaming media format, the second multimedia stream uses a container format supported by a real-time messaging protocol, such as the RTMP container format. In this case, the second multimedia stream is an RTMP stream.

[0101] It is understood that the multimedia stream processing method provided in this application, for the first multimedia stream, determines the first target media stream of the target media type through the first target stream segment identifier, and for the second multimedia stream, determines the second target media stream of the target message type through the second target stream segment identifier; and, through the preset identifier correspondence between the first stream segment identifier and the second stream segment identifier, realizes the conversion of the first multimedia stream to the second multimedia stream, and then realizes multi-track transmission by transmitting the second multimedia stream, thereby improving the transmission flexibility of multimedia content.

[0102] Step 104: The production-end equipment transmits the second multimedia stream to the consumer-end equipment.

[0103] In this application, since the second multimedia stream is used to transmit to the consumer device, after the production device obtains the second multimedia stream, it transmits the second multimedia stream to the consumer device, and at this time, the consumer device receives the second multimedia stream.

[0104] Step 105: The consumer device obtains multiple second target stream segment identifiers from the second multimedia stream.

[0105] In this embodiment, the consumer device consumes data from the second multimedia stream. Since the second multimedia stream is multimedia content in a message format corresponding to the transmission protocol, it is converted back to the first multimedia stream before data consumption. It is readily understood that this second multimedia stream is transmitted by the production device. Here, since the second multimedia stream is obtained based on multiple second target stream segment identifiers, the consumer device can obtain multiple second target stream segment identifiers from the second multimedia stream.

[0106] Step 106: The consumer device determines the first target stream segment identifier corresponding to each second target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier.

[0107] In this embodiment, the consumer device is able to obtain an identifier mapping relationship. Here, since each second target stream segment identifier obtained by the consumer device is used to determine the second target media stream of the target message type in the second multimedia stream, and the first target media stream used to determine the target media type in the first multimedia stream is a first target stream segment identifier; therefore, in order to convert the second multimedia stream back to the first multimedia stream, the consumer device matches each second target stream segment identifier with each second stream segment identifier in the identifier mapping relationship, and determines the first stream segment identifier corresponding to the matched second stream segment identifier as the first target stream segment identifier; thereby, it is possible to obtain multiple first target stream segment identifiers corresponding to multiple second target stream segment identifiers.

[0108] Step 107: The consumer device generates a file block including the first target stream segment identifier, and obtains multiple file blocks corresponding to multiple second target stream segment identifiers.

[0109] In this embodiment, the consumer device combines each first target stream segment identifier and the stream segment in the second media stream corresponding to the corresponding second target stream segment identifier to form a file block, wherein the corresponding second target stream segment corresponds to the first target stream segment identifier; thus, multiple file blocks can be obtained for multiple first target stream segment identifiers; that is, the consumer device can obtain one file block for one second target stream segment identifier, thereby obtaining multiple file blocks corresponding to multiple second target stream segment identifiers.

[0110] Step 108: The consumer device generates a first multimedia stream comprising multiple file blocks.

[0111] In this embodiment, after the consumer device obtains multiple file blocks, it combines the multiple file blocks into a first multimedia stream according to the combination method of the container format of the first multimedia stream. It is easy to see that the first multimedia stream includes multiple file blocks.

[0112] It should be noted that the first multimedia stream is used for data consumption on consumer devices.

[0113] It is understandable that the production-end equipment converts the first multimedia stream of multiple tracks into a second multimedia stream through a preset identifier mapping relationship to achieve transmission to the consumer-end equipment; then, the consumer-end equipment converts the second multimedia stream back into the first multimedia stream. Here, since the first stream segment identifier in the identifier mapping relationship is used to determine the first media stream of the media type, that is, to determine the track under the media type; and the second stream segment identifier is used to determine the second media stream of the message type, that is, to determine the track under the message type; in this way, the first multimedia stream and the second multimedia stream include media streams of different tracks of the same type (media type or message type), thereby enabling the transmission of multi-track multimedia content through a single data transmission, reducing the resource consumption of multi-track multimedia content transmission, and improving the transmission flexibility of multi-track multimedia content.

[0114] See Figure 6 , Figure 6 This is an exemplary flowchart of obtaining a correspondence provided in an embodiment of this application. The executing entity of this step can be a production-end device or a consumer-end device; here, the example is given with the executing entity being a production-end device. Figure 6 As shown, the multimedia stream processing method also includes a process of obtaining a correspondence, which includes steps 109 to 113. Each step is described below.

[0115] Step 109: Determine at least one media type to be expanded from at least one media type corresponding to the initial correspondence.

[0116] In this embodiment, the initial correspondence between the media type identifier and the message type identifier obtained by the production device is an initial correspondence. This initial correspondence corresponds to at least one media type and at least one message type corresponding to the at least one media type. Each media type is determined by the initial media type identifier, and each media type includes one track content, which corresponds to one first media stream. Therefore, based on the initial media type identifier, a unique first media stream can be determined. Similarly, each message type is determined by the initial message type identifier, and each message type includes one track content, which corresponds to one second media stream. Therefore, based on the initial message type identifier, a unique second media stream can be determined. The at least one media type to be expanded can be all media types among the at least one media type, or it can be some media types among the at least one media type. This embodiment does not limit this. Here, the production device performs the following processing (steps 110 to 112) for each media type to be expanded.

[0117] Step 110: Extend the media type to be extended to obtain at least two first-level identifiers, media type extension identifiers, at least two second-level identifiers, and message type extension identifiers.

[0118] In this embodiment, the production-end device extends the first-stream identifier for each media type to be extended, obtaining at least two extended first-stream identifiers. For each media type to be extended, the media type identifier is extended to obtain a media type extended identifier; wherein, the initial media type identifier for each media type to be extended is called the media type initial identifier. Here, both the media type extended identifier and the media type initial identifier are used to determine the media type to be extended. For each message type corresponding to each media type to be extended, the second-stream identifier is extended to obtain at least two extended second-stream identifiers. For each message type corresponding to each media type to be extended, the message type identifier is extended to obtain a message type extended identifier; wherein, the initial message type identifier for each message type corresponding to each media type to be extended is called the message type initial identifier. Here, both the message type extended identifier and the message type initial identifier are used to determine the message type corresponding to the media type to be extended.

[0119] Step 111: Combine one of the extended first-stream identifiers with the initial media type identifier in the initial correspondence, and combine at least one of the extended first-stream identifiers with the media type extended identifiers respectively to obtain at least two first-stream segment identifiers.

[0120] In this embodiment, the production device selects one of the extended first-stream identifiers from the at least two extended first-stream identifiers, and combines the extended first-stream identifier with the media type initial identifier in the initial correspondence to obtain a first-stream segment identifier; and selects at least one of the remaining extended first-stream identifiers from the at least two extended first-stream identifiers, and combines the at least one extended first-stream identifier with the media type extended identifier to obtain at least one first-stream segment identifier; in summary, at least two first-stream segment identifiers can be obtained.

[0121] Step 112: Combine one of the extended second-stream identifiers with the initial message type identifier in the initial correspondence, and combine at least one of the extended second-stream identifiers with the message type extended identifiers respectively to obtain at least two second-stream fragment identifiers.

[0122] In this embodiment, the production device selects one extended second stream identifier from the at least two extended second stream identifiers and combines the extended second stream identifier with the initial message type identifier in the initial correspondence to obtain a second stream fragment identifier; and selects at least one of the remaining extended second stream identifiers from the at least two extended second stream identifiers, and combines the at least one extended second stream identifier with the message type extended identifier to obtain at least one second stream fragment identifier; in summary, at least two second stream fragment identifiers can be obtained.

[0123] It should be noted that steps 111 and 112 are not in any particular order of execution. Figure 6 This is an example of an execution order.

[0124] Step 113: Obtain the identifier correspondence based on the correspondence between at least two first-stream segment identifiers and at least two second-stream segment identifiers.

[0125] In this embodiment of the application, the production device establishes a one-to-one correspondence between at least two first-stream segment identifiers and at least two second-stream segment identifiers based on the correspondence between media types and message types indicated by the initial correspondence. Here, the production device combines the established correspondences with at least one media type to be expanded into an identifier correspondence.

[0126] See Figure 7 , Figure 7 This is a flowchart illustrating the multimedia stream processing method provided in the embodiments of this application. Figure 2 ;like Figure 7As shown in the embodiment of this application, step 101 is followed by step 114A or step 114B; that is, after obtaining the first target stream segment identifier corresponding to each file block of the first multimedia stream, the multimedia stream processing method further includes step 114A or step 114B. Each step will be described below.

[0127] Step 114A: When the target media type identifier in the first target stream segment identifier fails to match in the initial correspondence, the generating end device ends the processing of the file block.

[0128] It should be noted that the identifier mapping relationship is derived from the initial mapping relationship. When the production-end device obtains the initial mapping relationship, it parses the first multimedia stream based on this initial mapping relationship. During the parsing process, since the target media type identifier in the first target stream segment identifier may be either the initial media type identifier or the extended media type identifier, and when the target media type identifier is the extended media type identifier, the production-end device cannot find a matching initial media type identifier in the initial mapping relationship, resulting in a matching failure. At this point, the production-end device ends the processing of the file block; thus, the processing of the file block can be skipped. The target media type identifier is used to determine the target media type.

[0129] Step 114B: When the target media type identifier is successfully matched in the initial correspondence, the generating device determines the initial media type identifier that matches the target media type identifier in the initial correspondence as the target media type initial identifier, and converts the first multimedia stream into the second multimedia stream based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence.

[0130] It should be noted that during the process of parsing the first multimedia stream based on the initial correspondence, when the target media type identifier is the initial media type identifier, the production end device can match the initial media type identifier that matches the target media type identifier in the initial correspondence, thus achieving a successful match; at this time, the production end device can correctly parse the file block and achieve the correct parsing of a track content.

[0131] In this embodiment, the process by which the consumer device parses the second media stream based on the initial correspondence is similar to the process described in steps 114A and 114B, and will not be repeated here.

[0132] It is understandable that, since the initial correspondence represents the correspondence between the initial media type identifier and the initial message type identifier, by defining each media type to be extended and the corresponding message type extension identifier, when processing the first multimedia stream based on the initial correspondence, the processing of the file block can be terminated when the target media type identifier in the first target stream segment identifier is a media type extension identifier; and when processing the second multimedia stream based on the initial correspondence, the processing of the second target stream segment identifier can be terminated when the target message type identifier in the second target stream segment identifier is a message type extension identifier; thus, correct parsing and compatibility can be achieved.

[0133] In this embodiment of the application, after step 110, the production-end device extends the identifier correspondence from the initial correspondence. Therefore, the media stream processing method may further include: the production-end device extends the media type to be extended to obtain at least two extended first-stream identifiers and at least two extended second-stream identifiers; the two extended first-stream identifiers are combined with the initial media type identifier in the initial correspondence to obtain at least two first-stream segment identifiers; the two extended second-stream identifiers are combined with the initial message type identifier in the initial correspondence to obtain at least two second-stream segment identifiers; and finally, the identifier correspondence is obtained based on the correspondence between the at least two first-stream segment identifiers and the at least two second-stream segment identifiers.

[0134] See Figure 8 , Figure 8 This is a flowchart illustrating the multimedia stream processing method provided in the embodiments of this application. Figure 3 ;like Figure 8 As shown in the embodiment of this application, after step 108, there is also a process of consuming the first multimedia stream; that is, after the consumer device generates the first multimedia stream including multiple file blocks, the multimedia stream processing method further includes steps 115 to 119, and each step will be described below.

[0135] Step 115: The consumer device obtains the stream segment corresponding to each file block of the first multimedia stream, thus obtaining multiple stream segments corresponding to multiple file blocks.

[0136] It should be noted that since each file block is used to store a stream segment of the first media stream of the target media type, the consumer device can obtain the corresponding stream segment from each file block of the first multimedia stream; thus, multiple stream segments can be obtained for multiple file blocks of the first multimedia stream.

[0137] Step 116: The consumer device obtains the parsing result of the first multimedia stream based on multiple stream segments.

[0138] It should be noted that consumer devices can directly identify multiple stream segments as the parsing result of the first multimedia stream. Consumer devices can also concatenate the same first multimedia stream of the same media type based on attributes such as timestamps in the file body, and identify the concatenated full first multimedia stream as the parsing result of the first multimedia stream.

[0139] Step 117: The consumer device responds to the multimedia consumption request and obtains the identifier of the data to be consumed.

[0140] In this embodiment of the application, the multimedia consumption request is used to request the parsing result of consuming the first multimedia stream; wherein, the data to be consumed identifier in the multimedia consumption request is used to select data from the parsing result, which may be to select all the data in the parsing result, or to select part of the data in the parsing result (e.g., one or more first media streams, a combination of first media streams of different media types, etc.), etc., and this embodiment of the application does not limit this.

[0141] Step 118: The consumer device obtains the data to be consumed corresponding to the data identifier to be consumed from the parsing results.

[0142] It should be noted that the data to be consumed obtained by the consumer device can be the full data from the parsing result, or it can be a portion of the data from the parsing result, and so on. The data to be consumed is used for data consumption.

[0143] Step 119: The consumer device consumes data based on the data to be consumed.

[0144] It should be noted that data consumption includes at least one of the following: distribution, playback, media processing, and storage. Distribution can refer to distributing data to be consumed to other clients or servers; playback refers to playing the data to be consumed on the consumer device; media processing refers to data processing on the consumer device, such as transcoding and resolution modification; and storage refers to storing data on the consumer device.

[0145] In this embodiment, the production-end device and the consumer-end device can be changed according to the actual scenario. For example, when the consumer-end device distributes data to be consumed, it changes to a production-end device.

[0146] The following describes an exemplary application of the embodiments of this application in a practical application scenario. This exemplary application describes the process of implementing multi-track multimedia stream processing based on RTMP / FLV.

[0147] It should be noted that the FLV container format includes an FLV file header and an FLV file body, the format of which is shown in Table 1.

[0148] Table 1

[0149]

[0150]

[0151] It should be noted that each tag (called a file block) is used to store a stream segment of a media stream of a certain media type in a multimedia stream; and the FLV tag format is shown in Table 2.

[0152] Table 2

[0153]

[0154] As can be seen from the fields in Tables 1 and 2, the current FLV container format includes one track (one audio track, one video track, and one subtitle track) for data of the same media type.

[0155] It should be noted that in multimedia content applications, there are scenarios where multiple audio tracks are carried for the same media type. For example, different audio tracks can use different encoding formats and parameters to provide differentiated audio playback experiences for different devices; different audio tracks can also be in different languages, enhancing the diversity of multimedia content choices. In addition to multiple audio tracks, scenarios such as multiple video tracks and multiple subtitle tracks can also be implemented.

[0156] Here, multi-track support is achieved by modifying the media type field and track identifier field in the FLV tag; the modified FLV tag format is shown in Table 3.

[0157] Table 3

[0158]

[0159] It should be noted that in the modified FLV tag format, track identifiers for the same media type (e.g., the media type corresponding to audio type identifier 8 and extended audio type identifier 24) are different, while track identifiers for different media types (e.g., the media type corresponding to audio type identifier 8 and extended video type identifier 25) can be the same. For example, when multimedia content includes video 1, audio 1, and audio 2, tag 1 can store a video stream segment of video 1, tag 2 can store an audio stream segment of audio 1, tag 3 can store an audio stream segment of audio 2, tag 4 can store the next video stream segment of video 1, and so on, interleaved. Tag 2 and tag 3 are different in their track identifiers, thus enabling multi-track audio transmission.

[0160] It should also be noted that audio type identifier 8, video type identifier 9, and script type identifier 18 are all referred to as the above-mentioned initial media type identifiers; extended audio type identifier 24, extended video type identifier 25, extended subtitle identifier, and script type identifier 26 are all referred to as the above-mentioned extended media type identifiers.

[0161] In addition, a default track can be selected from multiple tracks (for example, the default audio track is the audio track corresponding to audio type identifier 8, the default video track is the video track corresponding to video type identifier 9, and the default audio track is the subtitle track corresponding to subtitle type identifier 18), thus achieving backward compatibility.

[0162] Understandably, older electronic devices that did not implement FLV tag format modification did not process the track identifier field. Therefore, modifying the track identifier field also modifies the media type field, allowing older electronic devices to automatically skip processing data for non-default media types, ignore extended media types, and process default media type data normally, ensuring correct parsing.

[0163] It should also be noted that the transmission protocol for multimedia streams in FLV container format is RTMP, and the RTMP message format is shown in Table 4.

[0164] Table 4

[0165]

[0166] Accordingly, based on the modified FLV tag format, the RTMP message format is modified as shown in Table 5.

[0167] Table 5

[0168]

[0169]

[0170] It should be noted that the message type field in the modified RTMP message format corresponds to the first byte (including the reserved field, filter identifier field, and media type field) of the modified FLV tag format. The newly added message types in the message type field of the modified RTMP message format correspond to the newly added media types in the media type field of the modified FLV tag format. Furthermore, in the RTMP message format, audio type, video type, and subtitle type share a fixed value, while in the modified RTMP message format, the track identifiers for the same message type (e.g., the message types corresponding to audio type identifier 8 and extended audio type identifier 24) are different, while the track identifiers for different message types (e.g., the message types corresponding to audio type identifier 8 and extended video type identifier 25) can be the same.

[0171] It should also be noted that audio type identifier 8, video type identifier 9, and script type identifier 18 are all referred to as the initial identifiers of the aforementioned message types; extended audio type identifier 24, extended video type identifier 25, extended subtitle identifier, and script type identifier 26 are all referred to as the extended identifiers of the aforementioned message types.

[0172] In this embodiment, when a multimedia production end (referred to as the production end device) sends a multimedia stream to a multimedia consumption end (referred to as the consumption end device), the multimedia production end maps the media type identifier and track identifier (collectively referred to as the first stream segment identifier) ​​in the FLV multimedia stream (referred to as the first multimedia stream) to the message type identifier and track identifier (collectively referred to as the second stream segment identifier) ​​in the RTMP multimedia stream (referred to as the second multimedia stream) to achieve the conversion from FLV multimedia stream to RTMP multimedia stream. Then, based on the RTMP multimedia stream, the multimedia stream is transmitted between the multimedia production end and the multimedia consumption end. Furthermore, after receiving the RTMP multimedia stream from the multimedia production end, the multimedia consumption end maps the message type and track identifier in the RTMP multimedia stream to the media type identifier and track identifier in the FLV multimedia stream to achieve the conversion from RTMP multimedia stream to FLV multimedia stream. Then, by parsing the FLV multimedia stream, media data for different tracks can be obtained.

[0173] It should be noted that the same type of multitrack transmission provided in this application embodiment is applied to live streaming between media content production and media servers, forwarding between media server clusters, and content distribution from media servers to the user side.

[0174] For example, see Figure 9 , Figure 9 This is an exemplary schematic diagram of multi-track transmission provided in an embodiment of this application; as shown... Figure 9As shown, the media content production end 9-1 pushes the stream to the cloud device 9-2, and the cloud device 9-2 processes the pushed RTMP multimedia stream to respond to the playback end 9-3 pulling the stream. The cloud device 9-2 includes an access layer 9-21 for receiving the RTMP multimedia stream pushed by the media content production end 9-1; a media processing device 9-22 for converting the RTMP multimedia stream to an FLV multimedia stream and performing media processing on the FLV multimedia stream, such as transcoding and mixing; and a media distribution device 9-23 for distributing the pulled RTMP multimedia stream, converted from the FLV multimedia stream, to the playback end 9-3. Here, the pushing and pulling processes use RTMP; during the pushing process, the media content production end 9-1 is the multimedia production end, and the cloud device 9-2 is the multimedia consumption end; during the pulling process, the cloud device 9-2 is the multimedia production end, and the playback end 9-3 is the multimedia consumption end.

[0175] It is understood that the embodiments of this application implement a backward-compatible multimedia stream processing method for the transmission, distribution, storage, and playback of multiple audio tracks (or multiple video tracks, multiple subtitle tracks) in RTMP / FLV. This method achieves compatibility, simplifies the transmission structure, reduces the consumption of transmission resources, and ensures service stability. Furthermore, the backward-compatible RTMP / FLV multi-audio track multimedia content processing procedure provided by the embodiments of this application supports multiple audio tracks (or multiple video tracks, multiple subtitle tracks) in a single RTMP transmission, simplifies the central scheduling, management, and maintenance architecture, reduces the coupling of the system structure, improves the flexibility of service processing, and enhances the accuracy of transmission. Additionally, the backward-compatible RTMP / FLV multi-audio track multimedia content processing procedure provided by the embodiments of this application reduces the space occupied by additional fields by modifying the original fields. Moreover, the backward-compatible RTMP / FLV multi-track multimedia content processing procedure provided by the embodiments of this application can improve the flexibility of multimedia content applications, such as providing audio and video content of different qualities, providing multilingual internationalization capabilities, and so on.

[0176] The following continues to describe the exemplary structure of the first multimedia stream processing device 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software modules stored in the first multimedia stream processing device 255 in the first memory 250 may include:

[0177] The first identifier acquisition module 2551 is used to acquire a first target stream segment identifier corresponding to each of the multiple file blocks of the first multimedia stream. The first target stream segment identifier is used to determine a first target media stream of a target media type. The first multimedia stream includes at least two first media streams belonging to the same media type. The file block is used to store stream segments of the first media stream.

[0178] The first identifier acquisition module 2551 is further configured to determine a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, wherein the second target stream segment identifier includes a second target media stream used to determine the target message type;

[0179] The first conversion module 2552 is used to convert the first multimedia stream into a second multimedia stream based on multiple second target stream segment identifiers corresponding to multiple file blocks, wherein the second multimedia stream is used to transmit to a consumer device.

[0180] In this embodiment, the first multimedia stream processing device 255 further includes a relationship acquisition module 2553, configured to determine at least one media type to be expanded from at least one media type corresponding to the initial correspondence relationship, and perform the following processing on each media type to be expanded: expand the media type to be expanded to obtain at least two expanded first stream identifiers, a media type expansion identifier, at least two expanded second stream identifiers, and a message type expansion identifier; combine one of the expanded first stream identifiers with the initial media type identifier in the initial correspondence relationship, and combine at least one of the expanded first stream identifiers with the media type expansion identifier respectively to obtain at least two first stream segment identifiers; combine one of the expanded second stream identifiers with the initial message type identifier in the initial correspondence relationship, and combine at least one of the expanded second stream identifiers with the message type expansion identifier respectively to obtain at least two second stream segment identifiers; and obtain the identifier correspondence relationship based on the correspondence relationship between the at least two first stream segment identifiers and the at least two second stream segment identifiers.

[0181] In this application embodiment, the media type includes at least one of the following: audio type, video type, and subtitle type.

[0182] In this embodiment of the application, at least two first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bitrate, resolution, encoding format and language type.

[0183] In this embodiment, the first multimedia stream processing device 255 further includes a data parsing module 2554, configured to terminate the processing of the file block when the target media type identifier in the first target stream segment identifier fails to match in the initial correspondence, wherein the target media type identifier is used to determine the target media type; when the target media type identifier matches successfully in the initial correspondence, the media type initial identifier that matches the target media type identifier in the initial correspondence is determined as the target media type initial identifier, and the first multimedia stream is converted into the second multimedia stream based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence.

[0184] In this embodiment of the application, the container format used for the first multimedia stream is a streaming media format, and the container format used for the second multimedia stream is supported by a real-time messaging protocol.

[0185] In this embodiment of the application, the first multimedia stream processing device 255 further includes a data acquisition module 2555, configured to, for at least one first media stream corresponding to each media type of the multimedia to be transmitted, determine the first target stream segment identifier of each first media stream from the identifier correspondence included in the first stream segment identifiers, wherein the multimedia to be transmitted corresponds to at least one media type; generate a file block including the first target stream segment identifier for each stream segment of each first media stream; and generate a first multimedia stream corresponding to the multimedia to be transmitted from each file block.

[0186] The following description continues to illustrate the exemplary structure of the second multimedia stream processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 4 As shown, the software modules stored in the second multimedia stream processing device 455 in the second memory 450 may include:

[0187] The second identifier acquisition module 4551 is used to acquire multiple second target stream segment identifiers from the second multimedia stream, wherein the second target stream segment identifier is used to determine the second target media stream of the target message type, and the second multimedia stream is transmitted by the production end device;

[0188] The second identifier acquisition module 4551 is further configured to determine a first target stream segment identifier corresponding to each second target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, wherein the first target stream segment identifier is used to determine a first target media stream of the target media type;

[0189] The second conversion module 4552 is used to generate a file block including the first target stream segment identifier, and obtain a plurality of file blocks corresponding to a plurality of second target stream segment identifiers, wherein the file block is used to store stream segments of the first media stream;

[0190] The second conversion module 4552 is further configured to generate a first multimedia stream comprising a plurality of the file blocks, wherein the first multimedia stream comprises at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

[0191] In this embodiment, the second multimedia stream processing device 455 further includes a data consumption module 4553, configured to acquire the stream segment corresponding to each file block of the first multimedia stream, thereby obtaining a plurality of stream segments corresponding to the plurality of file blocks; based on the plurality of stream segments, obtain the parsing result of the first multimedia stream; in response to a multimedia consumption request, obtain a data identifier to be consumed; acquire the data to be consumed corresponding to the data identifier to be consumed from the parsing result; and perform data consumption based on the data to be consumed, wherein the data consumption includes at least one of the following: distribution, playback, media processing, and storage.

[0192] This application provides a computer program product comprising computer-executable instructions or a computer program stored in a computer-readable storage medium. A first processor of a production device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the production device to perform the multimedia stream processing method for a production device described in this application. A second processor of a consumer device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the consumer device to perform the multimedia stream processing method for a consumer device described in this application.

[0193] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a first processor, the first processor will execute the multimedia stream processing method for production devices provided in this application; or, when the computer-executable instructions or the computer program are executed by the first processor, a second processor will execute the multimedia stream processing method for consumer devices provided in this application. For example, Figure 5 The multimedia stream processing method is shown.

[0194] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0195] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0196] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0197] As an example, computer-executable instructions can be deployed to execute on multiple electronic devices located in one location (in this case, the multiple electronic devices located in one location are production-end devices and consumer-end devices), or to execute on multiple electronic devices distributed across multiple locations and interconnected via a communication network (in this case, the multiple electronic devices distributed across multiple locations and interconnected via a communication network are production-end devices and consumer-end devices).

[0198] It is understood that, in the embodiments of this application, data related to multimedia content is involved. When these embodiments are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. Furthermore, in the implementation of the data scraping technology solutions involved in these embodiments, when applied to specific products or technologies, the data collection, use, and processing processes should comply with national laws and regulations, adhere to the principles of legality, legitimacy, and necessity, and should not involve obtaining data types prohibited or restricted by laws and regulations, nor hinder the normal operation of the target website.

[0199] In summary, in this embodiment, since each file block representing the first multimedia stream includes a first target stream segment identifier, and this first target stream segment identifier is used to determine the first target media stream of the target media type, the first multimedia stream can include at least two first media streams belonging to the same media type, which is a type of multi-track multimedia content. Then, based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, the first multimedia stream is converted into a second multimedia stream for transmission. In this way, the transmission of a type of multi-track multimedia stream can be realized through a single transmission, which can reduce resource transmission consumption, improve transmission flexibility, and also achieve service compatibility and improve service stability.

[0200] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A multimedia stream processing method, characterized in that, The method includes: For multiple file blocks of a first multimedia stream, a first target stream segment identifier corresponding to each file block is obtained, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams belonging to the same media type, and the file block is used to store stream segments of the first media stream; Based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, a second target stream segment identifier corresponding to the first target stream segment identifier is determined, wherein the second target stream segment identifier includes a second target media stream used to determine the target message type; Based on multiple second target stream segment identifiers corresponding to multiple file blocks, the first multimedia stream is converted into a second multimedia stream, wherein the second multimedia stream is used for transmission to a consumer device.

2. The method according to claim 1, characterized in that, The method further includes: From at least one media type corresponding to the initial correspondence, at least one media type to be expanded is determined, and the following processing is performed on each media type to be expanded: The media type to be extended is extended to obtain at least two extended first-stream identifiers, a media type extension identifier, at least two extended second-stream identifiers, and a message type extension identifier; The extended first stream identifier is combined with the media type initial identifier in the initial correspondence, and at least one extended first stream identifier is combined with the media type extended identifier to obtain at least two first stream segment identifiers; Combine one of the extended second stream identifiers with the initial message type identifier in the initial correspondence, and combine at least one of the extended second stream identifiers with the message type extended identifier respectively to obtain at least two second stream fragment identifiers; The identifier correspondence is obtained based on the correspondence between at least two first stream segment identifiers and at least two second stream segment identifiers.

3. The method according to claim 1 or 2, characterized in that, The media type includes at least one of the following: audio type, video type, and subtitle type.

4. The method according to claim 1 or 2, characterized in that, At least two of the first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bitrate, resolution, encoding format, and language type.

5. The method according to claim 1 or 2, characterized in that, After obtaining the first target stream segment identifier corresponding to each of the multiple file blocks of the first multimedia stream, the method further includes: When the target media type identifier in the first target stream segment identifier fails to match in the initial correspondence, the processing of the file block ends, wherein the target media type identifier is used to determine the target media type; When a match is successfully found in the initial correspondence based on the target media type identifier, the initial media type identifier that matches the target media type identifier in the initial correspondence is determined as the target media type initial identifier. Based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence, the first multimedia stream is converted into the second multimedia stream.

6. The method according to claim 1 or 2, characterized in that, The first multimedia stream uses a streaming media format as its container, while the second multimedia stream uses a container format supported by a real-time messaging protocol.

7. The method according to claim 1 or 2, characterized in that, Before obtaining the first target stream segment identifier corresponding to each of the multiple file blocks of the first multimedia stream, the method further includes: For at least one first media stream corresponding to each media type of the multimedia to be transmitted, the first target stream segment identifier of each first media stream is determined from the first stream segment identifiers included in the identifier correspondence relationship, wherein the multimedia to be transmitted corresponds to at least one media type; For each stream segment of the first media stream, generate the file body block including the identifier of the first target stream segment; A first multimedia stream corresponding to the multimedia to be transmitted is generated from each of the file blocks.

8. A multimedia stream processing method, characterized in that, The method includes: From the second multimedia stream, obtain multiple second target stream segment identifiers, wherein the second target stream segment identifiers are used to determine the second target media stream of the target message type, and the second multimedia stream is transmitted by the production end device; Based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, a first target stream segment identifier is determined for each second target stream segment identifier, wherein the first target stream segment identifier is used to determine the first target media stream of the target media type; Generate a file block including the identifier of the first target stream segment, and obtain a plurality of file blocks corresponding to a plurality of second target stream segment identifiers, wherein the file blocks are used to store stream segments of the first media stream; A first multimedia stream comprising multiple file blocks is generated, wherein the first multimedia stream includes at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

9. The method according to claim 8, characterized in that, After generating the first multimedia stream comprising multiple file blocks, the method further includes: Obtain the stream segment corresponding to each file block of the first multimedia stream to obtain a plurality of stream segments corresponding to a plurality of file blocks; Based on multiple stream segments, the parsing result of the first multimedia stream is obtained; In response to a multimedia consumption request, obtain the identifier of the data to be consumed; Obtain the data to be consumed corresponding to the data to be consumed identifier from the parsing results; Data consumption is performed based on the data to be consumed, wherein the data consumption includes at least one of the following: distribution, playback, media processing, and storage.

10. A first multimedia stream processing device, characterized in that, The first multimedia stream processing device includes: The first identifier acquisition module is used to acquire a first target stream segment identifier corresponding to each of the multiple file blocks of the first multimedia stream, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams belonging to the same media type, and the file block is used to store stream segments of the first media stream. The first identifier acquisition module is further configured to determine a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, wherein the second target stream segment identifier includes a second target media stream used to determine the target message type; The first conversion module is used to convert the first multimedia stream into a second multimedia stream based on multiple second target stream segment identifiers corresponding to multiple file blocks, wherein the second multimedia stream is used to transmit to a consumer device.

11. A second multimedia stream processing device, characterized in that, The second multimedia stream processing device includes: The second identifier acquisition module is used to acquire multiple second target stream segment identifiers from the second multimedia stream, wherein the second target stream segment identifier is used to determine the second target media stream of the target message type, and the second multimedia stream is transmitted by the production end device; The second identifier acquisition module is further configured to determine a first target stream segment identifier corresponding to each second target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type; The second conversion module is used to generate a file block including the identifier of the first target stream segment, and obtain a plurality of file blocks corresponding to a plurality of the second target stream segment identifiers, wherein the file blocks are used to store stream segments of the first media stream; The second conversion module is further configured to generate a first multimedia stream comprising multiple file blocks, wherein the first multimedia stream comprises at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

12. A production-end device for multimedia stream processing, characterized in that, The production-end equipment includes: The first memory is used to store computer-executable instructions or computer programs; The first processor, when executing computer-executable instructions or computer programs stored in the first memory, implements the multimedia stream processing method according to any one of claims 1 to 7.

13. A consumer-end device for multimedia stream processing, characterized in that, The consumer-end device includes: Secondary memory is used to store computer-executable instructions or computer programs; The second processor, when executing computer-executable instructions or computer programs stored in the second memory, implements the multimedia stream processing method of claim 8 or 9.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the first processor, they implement the multimedia stream processing method according to any one of claims 1 to 7; or, when the computer-executable instructions or computer program are executed by the second processor, they implement the multimedia stream processing method according to claim 8 or 9.

15. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the first processor, they implement the multimedia stream processing method according to any one of claims 1 to 7; or, when the computer-executable instructions or computer program are executed by the second processor, they implement the multimedia stream processing method according to claim 8 or 9.

Images