Multimedia stream processing method and device, equipment, storage medium and program product

Abstract

The invention provides a multimedia stream processing method, device and equipment, a storage medium and a program product, which are applied to various multimedia stream transmission scenes such as cloud technology, multimedia, communication and video. The multimedia stream processing method comprises: for a plurality of file blocks of a first multimedia stream, obtaining a first target stream segment identifier corresponding to each file block, the first target stream segment identifier being used for determining a first target media stream of a target media type, and the first multimedia stream comprising at least two first media streams belonging to the same media type; determining a second target stream segment identifier corresponding to the first target stream segment identifier based on an identifier corresponding relationship between the first stream segment identifier and the second stream segment identifier; and converting the first multimedia stream into a second multimedia stream based on a plurality of second target stream fragment identifiers corresponding to the plurality of file blocks, the second multimedia stream being used for being transmitted to the consumer equipment. According to the invention, resource transmission consumption can be reduced.

Description

Technical Field

[0001] The present application relates to multimedia technology in the field of computer applications, and in particular to a multimedia stream processing method, device, equipment, storage medium and program product. Background Art

[0002] In the process of transmitting multimedia streams, media data including multiple different tracks of the same type are often split into multiple channels of data to be transmitted, so as to realize multi-track transmission through multiple channels. However, when realizing multi-track transmission through multiple channels, the consumption of resource transmission is increased. Summary of the invention

[0003] The embodiments of the present application provide a multimedia stream processing method, apparatus, device, storage medium and program product, which can reduce resource transmission consumption.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a multimedia stream processing method, the method comprising:

[0006] For a plurality of file blocks of a first multimedia stream, obtaining a first target stream segment identifier corresponding to each of the file blocks, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams belonging to the same media type, and the file block is used to store stream segments of the first media stream;

[0007] Determine, based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, a second target stream segment identifier corresponding to the first target stream segment identifier, wherein the second target stream segment identifier includes a second target media stream for determining a target message type;

[0008] Based on a plurality of the second target stream segment identifiers corresponding to a plurality of the file body blocks, the first multimedia stream is converted into a second multimedia stream, wherein the second multimedia stream is used for transmission to a consumer device.

[0009] The present application provides another multimedia stream processing method, the method comprising:

[0010] Acquire a plurality of second target stream segment identifiers from a second multimedia stream, wherein the second target stream segment identifiers are used to determine a second target media stream of a target message type, and the second multimedia stream is transmitted by a production end device;

[0011] Determine, based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, a first target stream segment identifier corresponding to each of the second target stream segment identifiers, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type;

[0012] Generate a file body block including the first target stream segment identifier, and obtain a plurality of the file body blocks corresponding to the plurality of the second target stream segment identifiers, wherein the file body block is used to store the stream segment of the first media stream;

[0013] A first multimedia stream including a plurality of the file blocks is generated, wherein the first multimedia stream includes at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

[0014] The embodiment of the present application provides a first multimedia stream processing device, the first multimedia stream processing device comprising:

[0015] A first identifier acquisition module is used to acquire, for each of the plurality of file body blocks of the first multimedia stream, a first target stream segment identifier corresponding to the file body block, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams of the same media type, and the file body block is used to store stream segments of the first media stream;

[0016] The first identifier acquisition module is further used to determine a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, wherein the second target stream segment identifier includes a second target media stream used to determine a target message type;

[0017] The first conversion module is used to convert the first multimedia stream into a second multimedia stream based on a plurality of second target stream segment identifiers corresponding to a plurality of the file body blocks, wherein the second multimedia stream is used to be transmitted to a consumer device.

[0018] In an embodiment of the present application, the first multimedia stream processing device also includes a relationship acquisition module, which is used to determine at least one media type to be extended from at least one of the media types corresponding to the initial corresponding relationship, and perform the following processing on each of the media types to be extended: extend the media type to be extended to obtain at least two extended first stream identifiers, a media type extension identifier, at least two extended second stream identifiers and a message type extension identifier; combine one of the extended first stream identifiers with the media type initial identifier in the initial corresponding relationship, and combine at least one of the extended first stream identifiers with the media type extension identifier respectively to obtain at least two first stream segment identifiers; combine one of the extended second stream identifiers with the message type initial identifier in the initial corresponding relationship, and combine at least one of the extended second stream identifiers with the message type extension identifier respectively to obtain at least two second stream segment identifiers; obtain the identifier correspondence based on the correspondence between the at least two first stream segment identifiers and the at least two second stream segment identifiers.

[0019] In an embodiment of the present application, the media type includes at least one of the following: audio type, video type, and subtitle type.

[0020] In an embodiment of the present application, at least two of the first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bit rate, resolution, encoding format, and language type.

[0021] In an embodiment of the present application, the first multimedia stream processing device also includes a data parsing module, which is used to end the processing of the file body block when the target media type identifier in the first target stream segment identifier fails to match in the initial correspondence relationship, wherein the target media type identifier is used to determine the target media type; when the target media type identifier successfully matches in the initial correspondence relationship, the media type initial identifier that matches the target media type identifier in the initial correspondence relationship is determined as the target media type initial identifier, and the first multimedia stream is converted into the second multimedia stream based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence relationship.

[0022] In the embodiment of the present application, the container format used by the first multimedia stream is a streaming media format, and the second multimedia stream uses a container format supported by a real-time messaging protocol.

[0023] In an embodiment of the present application, the first multimedia stream processing device also includes a data acquisition module, which is used to determine the first target stream segment identifier of each first media stream from each first stream segment identifier included in the identifier correspondence relationship for at least one first media stream corresponding to each media type of the multimedia to be transmitted, wherein the multimedia to be transmitted corresponds to at least one media type; for each stream segment of each first media stream, generate the file body block including the first target stream segment identifier; and generate the first multimedia stream corresponding to the multimedia to be transmitted from each file body block.

[0024] The embodiment of the present application provides a second multimedia stream processing device, the second multimedia stream processing device comprising:

[0025] A second identifier acquisition module, used to acquire a plurality of second target stream segment identifiers from a second multimedia stream, wherein the second target stream segment identifiers are used to determine a second target media stream of a target message type, and the second multimedia stream is transmitted by a production end device;

[0026] The second identifier acquisition module is further used to determine a first target stream segment identifier corresponding to each second target stream segment identifier based on an identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type;

[0027] A second conversion module is used to generate a file body block including the first target stream segment identifier, and obtain a plurality of the file body blocks corresponding to the plurality of the second target stream segment identifiers, wherein the file body block is used to store the stream segment of the first media stream;

[0028] The second conversion module is further used to generate a first multimedia stream including multiple file blocks, wherein the first multimedia stream includes at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

[0029] In an embodiment of the present application, the second multimedia stream processing device also includes a data consumption module, which is used to obtain the stream segment corresponding to each file block of the first multimedia stream, and obtain multiple stream segments corresponding to multiple file blocks; based on the multiple stream segments, obtain the parsing result of the first multimedia stream; in response to a multimedia consumption request, obtain a data identifier to be consumed; obtain the data to be consumed corresponding to the data identifier to be consumed from the parsing result; and perform data consumption based on the data to be consumed, wherein the data consumption includes at least one of the following: distribution, playback, media processing and storage.

[0030] The embodiment of the present application provides a production end device for multimedia stream processing, the production end device comprising:

[0031] A first memory, used to store computer executable instructions or computer programs;

[0032] The first processor is used to implement the multimedia stream processing method applied to the production end device provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the first memory.

[0033] The embodiment of the present application provides a consumer-end device for multimedia stream processing, the consumer-end device comprising:

[0034] A second memory, used to store computer executable instructions or computer programs;

[0035] The second processor is used to implement the multimedia stream processing method applied to the consumer-end device provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the second memory.

[0036] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a first processor, the multimedia stream processing method applied to a production-end device provided in the embodiment of the present application is implemented; or, when the computer-executable instructions or the computer program are executed by a second processor, the multimedia stream processing method applied to a consumption-end device provided in the embodiment of the present application is implemented.

[0037] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a first processor, the multimedia stream processing method applied to a consumer-end device provided in the embodiment of the present application is implemented; or, when the computer executable instructions or the computer program are executed by a second processor, the multimedia stream processing method applied to a consumer-end device provided in the embodiment of the present application is implemented.

[0038] The embodiments of the present application have at least the following beneficial effects: since each file body block used to represent the first multimedia stream includes a first target stream segment identifier, and the first target stream segment identifier is used to determine the first target media stream of the target media type, the first multimedia stream can include at least two first media streams belonging to the same media type, which is a one-type multi-track multimedia content; then, based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, the first multimedia stream is converted into a one-way second multimedia stream for transmission; in this way, the transmission of a one-type multi-track multimedia stream can be achieved through one-way transmission, which can reduce resource transmission consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of an exemplary splitting of multimedia streams;

[0040] Figure 2 It is a schematic diagram of the architecture of a multimedia stream processing system provided in an embodiment of the present application;

[0041] Figure 3 This is a method provided by the embodiment of the present application. Figure 2 A schematic diagram of the structure of the server in FIG.

[0042] Figure 4 This is a method provided by the embodiment of the present application. Figure 2 A schematic diagram of the structure of the terminal in FIG.

[0043] Figure 5 This is a flowchart of the multimedia stream processing method provided by the embodiment of the present application. Figure 1 ;

[0044] Figure 6 is a flowchart of an exemplary method of obtaining a corresponding relationship provided in an embodiment of the present application;

[0045] Figure 7 This is a flowchart of the multimedia stream processing method provided by the embodiment of the present application. Figure 2 ;

[0046] Figure 8 This is a flowchart of the multimedia stream processing method provided by the embodiment of the present application. Figure 3 ;

[0047] Fig. 9 It is a schematic diagram of an exemplary implementation of multi-track transmission provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0049] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] In the following description, the terms "first\second" are used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0051] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0052] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0053] 1) Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or a local area network to realize data computing, storage, processing and sharing. The multimedia stream processing method provided in the embodiment of the present application can be applied to the field of cloud technology, such as stream pulling and stream pushing in the cloud technology field.

[0054] 2) Real-Time Messaging Protocol (RTMP), a media transmission protocol, is a network protocol for real-time data communication. RTMP supports both on-demand and live broadcasting. For example, it can be applied to streaming from media content producers to media servers, forwarding between media server clusters, and content distribution from media servers to clients. RTMP has low latency (below the latency threshold) and low deployment cost (below the cost threshold), and can achieve compatibility.

[0055] 3) Streaming media format (Flash Video, FLV), using a container format for streaming; the container format used by RTMP is FLV; wherein the container format refers to the format in which the data in the encoding format is packaged.

[0056] 4) Multi-track transmission refers to the transmission of multi-track data including at least two tracks of the same media type, such as the transmission of a multimedia stream including video A, audio B1 and audio B2. In addition, single-track transmission refers to the transmission of a multimedia stream in which each media type includes one track of data, such as the transmission of a multimedia stream including video A and audio B1.

[0057] It should be noted that in the process of transmitting multimedia streams, for example, in the process of using RTMP and FLV to transmit multimedia information, since RTMP and FLV are used for single-track transmission; in order to achieve multi-track transmission, RTMP is usually replaced with other transmission protocols (such as User Datagram Protocol (UDP)), and the streaming media format is replaced with other container formats (such as Transport Stream (TS)). However, for deployed services based on RTMP transmission, migration to other transmission protocols involves issues such as cost expenditure and business stability, which affects the complexity and efficiency of multi-track transmission.

[0058] In addition, in order to realize multi-track transmission, the media data including multiple different tracks of the same type can also be split into multiple channels of data to be transmitted, so as to realize multi-track transmission through multiple transmission; for example, the multimedia stream can be split into multiple RTMP transmissions, wherein each RTMP transmission is a single-track transmission; see Figure 1 , Figure 1 is a schematic diagram of an exemplary splitting of multimedia streams; Figure 1 As shown, the multimedia stream is a multi-track media content 1-1, which exemplarily shows video 1-11, audio 1-12, audio 1-13, etc.; the multi-track media content 1-1 is split based on the number of audios to obtain information to be transmitted through multiple RTMP channels, which are video 1-11 and audio 1-12, video 1-11 and audio 1-13, etc.; after transmission 1-2, media processing, secondary distribution, and playback are performed at the consumer end 1-3. However, in the process of multi-channel RTMP transmission, the splitting, management, and aggregation of multimedia streams, as well as the determination of the transmission of each RTMP channel are implemented by the scheduling center, which increases the complexity and coupling of the architecture and reduces flexibility; in addition, multi-channel RTMP transmission also increases the probability of transmission anomalies, and when one RTMP channel is abnormal and the other RTMP channels are normal, the fault-tolerant processing performed also increases the consumption of resource transmission.

[0059] Based on this, the embodiments of the present application provide a multimedia stream processing method, apparatus, device, computer-readable storage medium and computer program product, which can reduce resource transmission consumption. The following describes the exemplary applications of the production-end device and consumer-end device provided by the embodiments of the present application. The production-end device and consumer-end device provided by the embodiments of the present application can be implemented as various types of terminals such as smart phones, smart watches, laptops, tablet computers, desktop computers, smart home appliances, set-top boxes, smart car devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices and smart speakers. They can also be implemented as servers, or a combination of the two. Below, an exemplary application will be described when the production-end device is implemented as a server and the consumer-end device is implemented as a terminal.

[0060] See also Figure 2 , Figure 2 Schematic diagram of the architecture of the multimedia stream processing system provided in the embodiment of the present application; Figure 2 As shown, to support a multimedia stream processing application, in the multimedia stream processing system 100, the terminal 400 (exemplarily showing the terminal 400-1 and the terminal 400-2) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. In addition, the multimedia stream processing system 100 also includes a database 500 for providing data support to the server 200; and Figure 2 What is shown in the figure is a situation where the database 500 is independent of the server 200. In addition, the database 500 can also be integrated in the server 200, which is not limited in the embodiment of the present application.

[0061] The server 200 is used to obtain a first target stream segment identifier corresponding to each file body block of a first multimedia stream; determine a second target stream segment identifier corresponding to the first target stream segment identifier based on an identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier; convert the first multimedia stream into a second multimedia stream based on a plurality of second target stream segment identifiers corresponding to the plurality of file body blocks, and transmit the second multimedia stream to the terminal 400 via the network 300.

[0062] Terminal 400 is used to receive a second multimedia stream sent by server 200 through network 300; obtain multiple second target stream segment identifiers from the second multimedia stream; determine the first target stream segment identifier corresponding to each second target stream segment identifier based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier; generate a file body block including the first target stream segment identifier, and obtain multiple file body blocks corresponding to the multiple second target stream segment identifiers; generate a first multimedia stream including multiple file body blocks (exemplarily shown by graphical interface 410-1 and graphical interface 410-2 as a first multimedia stream including a video stream and multiple audio streams), the first multimedia stream is used for data consumption on terminal 400.

[0063] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0064] See also Figure 3 , Figure 3 This is a method provided by the embodiment of the present application. Figure 2 The structural diagram of the server in Figure 3 As shown, the server 200 includes: at least one first processor 210, a first memory 250 and at least one first network interface 220. The various components in the server 200 are coupled together through a first bus system 240. It can be understood that the first bus system 240 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 3 In the figure, various buses are labeled as a first bus system 240 .

[0065] The first processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0066] The first memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The first memory 250 may optionally include one or more storage devices physically located away from the first processor 210.

[0067] The first memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The first memory 250 described in the embodiment of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, the first memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0069] The first operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0070] A first network communication module 252, used to reach other electronic devices via one or more (wired or wireless) first network interfaces 220, exemplary first network interfaces 220 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), etc.;

[0071] In some embodiments, the first multimedia stream processing device provided in the embodiments of the present application may be implemented in software. Figure 3 The first multimedia stream processing device 255 stored in the first memory 250 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a first identification acquisition module 2551, a first conversion module 2552, a relationship acquisition module 2553, a data parsing module 2554 and a data acquisition module 2555. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0072] See also Figure 4 , Figure 4 This is a method provided by the embodiment of the present application. Figure 2 The schematic diagram of the terminal structure in FIG. Figure 4As shown, the terminal 400 includes: at least one second processor 410, a second memory 450, at least one second network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a second bus system 440. It can be understood that the second bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 4 In the figure, various buses are labeled as a second bus system 440 .

[0073] The second processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0075] The second memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The second memory 450 may optionally include one or more storage devices physically located away from the second processor 410.

[0076] The second memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory, and the volatile memory may be a random access memory. The second memory 450 described in the embodiment of the present application is intended to include any suitable type of memory.

[0077] In some embodiments, the second memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0078] The second operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0079] A second network communication module 452, for reaching other electronic devices via one or more (wired or wireless) second network interfaces 420, exemplary second network interfaces 420 include: Bluetooth, wireless compatibility authentication, and universal serial bus, etc.;

[0080] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0081] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0082] In some embodiments, the second multimedia stream processing device provided in the embodiments of the present application may be implemented in software. Figure 4 The second multimedia stream processing device 455 stored in the second memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a second identification acquisition module 4551, a second conversion module 4552 and a data consumption module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0083] In some embodiments, the first multimedia stream processing device and the second multimedia stream processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the first multimedia stream processing device and the second multimedia stream processing device provided in the embodiments of the present application can be processors in the form of hardware decoding processors, which are programmed to execute the multimedia stream processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0084] In some embodiments, the terminal or server can implement the multimedia stream processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0085] Below, the multimedia stream processing method provided by the embodiment of the present application will be described in combination with the exemplary application and implementation of the production end device and the consumer end device provided by the embodiment of the present application. In addition, the multimedia stream processing method provided by the embodiment of the present application is applied to various multimedia stream transmission scenarios such as cloud technology, multimedia, communication and video.

[0086] See also Figure 5 , Figure 5 This is a flowchart of the multimedia stream processing method provided by the embodiment of the present application. Figure 1 ; The following will be combined Figure 5 The steps shown are explained.

[0087] Step 101: A production end device obtains a first target stream segment identifier corresponding to each file block of a first multimedia stream.

[0088] In an embodiment of the present application, after obtaining the first multimedia stream, the production-end device is used to transmit the first multimedia stream to the consumer-end device; and the message format supported by the transmission protocol used during transmission is different from the container format of the first multimedia stream, so the first multimedia stream is converted here to be converted into the message format supported by the adopted transmission protocol. The production-end device performs conversion in units of file blocks, and the first media stream to which the stream segment stored in each file block belongs is determined by the first target stream segment identifier in the file block, so that the production-end device first obtains the first target stream segment identifier in each file block to realize conversion based on the first target stream segment identifier in the file block. Among them, the first multi-media stream includes multiple first media streams, and the multiple first media streams include at least two first media streams belonging to different tracks of the same media type.

[0089] It should be noted that the first multimedia stream includes multiple file blocks, which are a sequence of multiple file blocks; wherein each file block is used to store a stream segment of the first media stream, that is, to store a track content segment under a media type, and the first media stream is a track content under a media type; in addition, the first multimedia stream includes at least two first media streams belonging to the same media type, so that the multiple file blocks include at least two stream segments belonging to the same media type, so that the first multimedia stream is multi-track data, that is, at least one of a multimedia stream of multiple audio tracks, a multimedia stream of multiple subtitle tracks, and a multimedia stream of multiple video tracks. Here, the first target stream segment identifier includes a first target stream identifier and a target media type identifier, the first target stream identifier is used to determine the first target media stream (e.g., audio 1, audio 2, video 2, etc.) in combination with the target media type identifier, the target media type identifier is used to determine the target media type (e.g., audio type, video type, subtitle type, etc.), and each file block determines the first media stream (called track content) under the target media type through the first target stream identifier and the target media type identifier. In addition, the production-end device refers to an electronic device that obtains the first multimedia stream, which can be a multimedia content producer, such as a multimedia content production client; it can also be a multimedia content processor, such as a multimedia server; and so on, which is not limited in the embodiments of the present application.

[0090] In an embodiment of the present application, the media type includes one of the following: audio type, video type, subtitle type; wherein the audio type is used to indicate that the data carried is audio, the video type is used to indicate that the data carried is video, and the subtitle type is used to indicate that the data carried is subtitle. In this way, at least one of multiple audio tracks, multiple video tracks, and multiple subtitle tracks can be implemented, wherein multiple audio tracks refer to multiple audio tracks corresponding to the first multimedia stream (i.e., including multiple audio streams), multiple video tracks refer to multiple video tracks corresponding to the first multimedia stream (i.e., including multiple video streams), and multiple subtitle tracks refer to multiple subtitle tracks corresponding to the first multimedia stream (i.e., including multiple subtitle streams).

[0091] In an embodiment of the present application, at least two first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bit rate, resolution, encoding format, and language type.

[0092] It can be understood that when the media attribute is bit rate, multiple audio tracks or multiple video tracks with different bit rates can be realized; when the media attribute is resolution, multiple audio tracks or multiple video tracks with different resolutions can be realized; when the media attribute is encoding format, multiple audio tracks or multiple video tracks with different encoding formats can be realized; when the media attribute is language type, multiple subtitle tracks with different language types can be realized; when the media attribute is any combination of bit rate, resolution, encoding format and language type, multiple tracks with corresponding different media attributes can be realized. In addition, when the media attribute is any combination of bit rate, resolution and encoding format, the first multimedia stream can provide multimedia content of different qualities to adapt to different multimedia consumption.

[0093] In the implementation of the present application, the container format used by the first multimedia stream is a streaming media format, such as an FLV container format. In this case, the first multimedia stream is an FLV stream.

[0094] Step 102: The production-end device determines a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier.

[0095] In an embodiment of the present application, a production-end device can obtain an identifier correspondence relationship, which represents a correspondence relationship between a first stream segment identifier and a second stream segment identifier; wherein the first stream segment identifier includes a first stream identifier and a media type identifier, and the second stream segment identifier includes a second stream identifier and a message type identifier; the first stream identifier is a stream identifier in a container format corresponding to the first multimedia stream, which is used to determine the track under the media type in combination with the media type identifier; the second stream identifier is a stream identifier in a message format of a transmission protocol, which is used to determine the track under the message type in combination with the message type identifier; and one first stream segment identifier corresponds to one second stream segment identifier. Here, the production-end device matches the first target stream segment identifier of each file block with each first stream segment identifier in the identifier correspondence relationship, and determines the second target stream identifier corresponding to the matched first stream segment identifier as the second target stream segment identifier corresponding to the first target stream segment identifier.

[0096] It should be noted that the second target stream segment identifier includes a second target stream identifier and a target message type identifier. The second target stream identifier is used to determine the second target media stream of the target message type (e.g., audio 1, audio 2, video 2, etc.) in combination with the target message type identifier. The target message type identifier is used to determine the target message type (e.g., audio type, video type, subtitle type, etc.). In addition, in the identifier correspondence, the first stream identifiers of the same media type are different, and the first stream identifiers of different media types can be the same or different; the second stream identifiers of the same message type are different, and the second stream identifiers of different message types can be the same or different.

[0097] Step 103: The production-end device converts the first multimedia stream into a second multimedia stream based on a plurality of second target stream segment identifiers corresponding to a plurality of file blocks.

[0098] In an embodiment of the present application, the production-end device obtains multiple second target stream segment identifiers corresponding to multiple file body blocks by the second target stream segment identifier corresponding to each file body block; then, the production-end device combines the multiple second target stream segment identifiers and the multiple stream segments in the multiple file bodies into a second multimedia stream according to the message format adopted by the transmission protocol; at this time, the conversion from the first multimedia stream to the second multimedia stream is completed.

[0099] It should be noted that the message format of the second multimedia stream is a message format supported by the transmission protocol used to transmit the first multimedia stream; the second multimedia stream is used to be transmitted to the consumer-end device; wherein the consumer-end device is an electronic device that receives the second multimedia stream, which can be a processor of multimedia content, such as a multimedia server; or a player of multimedia content, such as a multimedia content player client; etc., which is not limited in the embodiments of the present application. Here, when the production-end device is a multimedia content producer, the consumer-end device can be a processor of multimedia content; when the production-end device is a processor of multimedia content, the consumer-end device can be a player of multimedia content.

[0100] In an embodiment of the present application, when the container format used by the first multimedia stream is a streaming media format, the second multimedia stream uses a container format supported by the real-time messaging protocol, such as the RTMP container format. In this case, the second multimedia stream is an RTMP stream.

[0101] It can be understood that the multimedia stream processing method provided in the embodiment of the present application, for the first multimedia stream, determines the first target media stream of the target media type through the first target stream segment identifier, and for the second multimedia stream, determines the second target media stream of the target message type through the second target stream segment identifier; and, through the identifier correspondence relationship between the preset first stream segment identifier and the second stream segment identifier, the conversion of the first multimedia stream to the second multimedia stream is realized, and then multi-track transmission is realized by transmitting the second multimedia stream, thereby improving the transmission flexibility of the multimedia content.

[0102] Step 104: The production-end device transmits the second multimedia stream to the consumption-end device.

[0103] In the implementation of this application, since the second multimedia stream is used to be transmitted to the consumer device, after the production device obtains the second multimedia stream, it transmits the second multimedia stream to the consumer device. At this time, the consumer device also receives the second multimedia stream.

[0104] Step 105: The consumer device obtains multiple second target stream segment identifiers from the second multimedia stream.

[0105] In the embodiment of the present application, the consumer-end device is used to consume data of the second multimedia stream. Since the second multimedia stream is multimedia content in a message format corresponding to the transmission protocol, the second multimedia stream is converted back to the first multimedia stream before consuming data of the second multimedia stream. It is easy to know that the second multimedia stream is transmitted by the production-end device. Here, since the second multimedia stream is obtained based on multiple second target stream segment identifiers, the consumer-end device can obtain multiple second target stream segment identifiers from the second multimedia stream.

[0106] Step 106: The consumer device determines a first target stream segment identifier corresponding to each second target stream segment identifier based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier.

[0107] In an embodiment of the present application, the consumer-end device can obtain the identifier correspondence. Here, since each second target stream segment identifier obtained by the consumer-end device is used to determine the second target media stream of the target message type in the second multimedia stream, and the first target media stream used to determine the target media type in the first multimedia stream is the first target stream segment identifier; therefore, in order to convert the second multimedia stream back to the first multimedia stream, the consumer-end device matches each second target stream segment identifier with each second stream segment identifier in the identifier correspondence, and determines the first stream segment identifier corresponding to the matched second stream segment identifier as the first target stream segment identifier; thereby, multiple first target stream segment identifiers corresponding to multiple second target stream segment identifiers can be obtained.

[0108] Step 107: The consumer-end device generates a file body block including the first target stream segment identifier, and obtains a plurality of file body blocks corresponding to the plurality of second target stream segment identifiers.

[0109] In an embodiment of the present application, the consumer-end device combines each first target stream segment identifier and the stream segment corresponding to the corresponding second target stream segment identifier in the second media stream to assemble a file body block, wherein the corresponding second target stream segment corresponds to the first target stream segment identifier; thereby, multiple file body blocks can be obtained for multiple first target stream segment identifiers; that is, the consumer-end device can obtain a file body block for one second target stream segment identifier, thereby, multiple file body blocks corresponding to multiple second target stream segment identifiers can be obtained.

[0110] Step 108: The consumer device generates a first multimedia stream including a plurality of file blocks.

[0111] In the embodiment of the present application, after the consumer device obtains multiple file blocks, the multiple file blocks are combined into a first multimedia stream according to the combination mode of the container format of the first multimedia stream. It is easy to know that the first multimedia stream includes multiple file blocks.

[0112] It should be noted that the first multimedia stream is used for data consumption on a consumer device.

[0113] It can be understood that the production-end device converts the multi-track first multimedia stream into the second multimedia stream through a preset identification correspondence relationship to achieve transmission to the consumer-end device; then the conversion of the second multimedia stream to the first multimedia stream is achieved on the consumer-end device side; wherein, since the first stream segment identifier in the identification correspondence relationship is used to determine the first media stream of the media type, that is, the track under the media type; and the second stream segment identifier is used to determine the second media stream of the message type, that is, the track under the message type; in this way, the first multimedia stream and the second multimedia stream include media streams of different tracks of the same type (media type or message type), thereby being able to transmit multi-track multimedia content through one-way data, reducing the resource consumption of multi-track multimedia content transmission and improving the transmission flexibility of multi-track multimedia content.

[0114] See also Figure 6 , Figure 6 is an exemplary flowchart of obtaining a corresponding relationship provided in an embodiment of the present application, wherein the execution subject of this step can be a production-end device or a consumption-end device. Here, the execution subject is an example of a production-end device; Figure 6 As shown, the multimedia stream processing method also includes a process of obtaining a corresponding relationship, and the process of obtaining a corresponding relationship includes steps 109 to 113, and each step is described below.

[0115] Step 109: Determine at least one media type to be extended from the at least one media type corresponding to the initial corresponding relationship.

[0116] In an embodiment of the present application, the correspondence between the media type identifier and the message type identifier initially obtained by the production end device is an initial correspondence, and the initial correspondence corresponds to at least one media type and at least one message type corresponding to the at least one media type; and each media type is determined by the media type initial identifier, and each media type includes a track content, that is, corresponds to a first media stream, so that, based on the media type initial identifier, a unique first media stream can be determined; similarly, each message type is determined by the message type initial identifier, and each message type includes a track content, that is, corresponds to a second media stream, so that, based on the message type initial identifier, a unique second media stream can be determined. Among them, at least one media type to be extended can be all media types in at least one media type, or it can be part of the media types in at least one media type, and the embodiment of the present application does not limit this. Here, the production end device performs the following processing (steps 110 to 112) for each media type to be extended.

[0117] Step 110: Extend the media type to be extended to obtain at least two extended first flow identifiers, a media type extension identifier, at least two extended second flow identifiers, and a message type extension identifier.

[0118] In an embodiment of the present application, the production-end device extends the first stream identifier for each media type to be extended, and obtains at least two extended first stream identifiers. The media type identifier is extended for each media type to be extended, and a media type extension identifier is obtained; wherein, the initial media type identifier of each media type to be extended is called the media type initial identifier, and here, the media type extension identifier and the media type initial identifier of each media type to be extended are both used to determine the media type to be extended. The second stream identifier is extended for each message type corresponding to the media type to be extended, and at least two extended second stream identifiers are obtained. The message type identifier is extended for each message type corresponding to the media type to be extended, and a message type extension identifier is obtained; wherein, the initial message type identifier of the message type corresponding to each media type to be extended is called the message type initial identifier, and here, the message type extension identifier and the message type initial identifier of the message type corresponding to each media type to be extended are both used to determine the message type corresponding to the media type to be extended.

[0119] Step 111: Combine an extended first stream identifier with the media type initial identifier in the initial corresponding relationship, and respectively combine at least one extended first stream identifier with the media type extended identifier to obtain at least two first stream segment identifiers.

[0120] In an embodiment of the present application, a production-end device selects an extended first stream identifier from at least two extended first stream identifiers, and combines the extended first stream identifier with the media type initial identifier in the initial corresponding relationship to obtain a first stream fragment identifier; and selects at least one remaining extended first stream identifier from the at least two extended first stream identifiers, and combines the at least one extended first stream identifier with the media type extended identifier, respectively, to obtain at least one first stream fragment identifier; in summary, at least two first stream fragment identifiers can be obtained.

[0121] Step 112: Combine an extended second flow identifier with the message type initial identifier in the initial corresponding relationship, and respectively combine at least one extended second flow identifier with the message type extended identifier to obtain at least two second flow segment identifiers.

[0122] In an embodiment of the present application, a production-end device selects an extended second flow identifier from at least two extended second flow identifiers, and combines the extended second flow identifier with the message type initial identifier in the initial corresponding relationship to obtain a second flow fragment identifier; and selects at least one remaining extended second flow identifier from the at least two extended second flow identifiers, and combines the at least one extended second flow identifier with the message type extended identifier, respectively, to obtain at least one second flow fragment identifier; in summary, at least two second flow fragment identifiers can be obtained.

[0123] It should be noted that step 111 and step 112 are executed in no particular order. Figure 6 is an exemplary execution order.

[0124] Step 113: Obtain an identifier corresponding relationship based on a corresponding relationship between at least two first stream segment identifiers and at least two second stream segment identifiers.

[0125] In an embodiment of the present application, the production-end device establishes a one-to-one correspondence between at least two first stream segment identifiers and at least two second stream segment identifiers based on the correspondence between the media type and the message type indicated by the initial correspondence; here, the production-end device combines at least one correspondence established corresponding to at least one media type to be extended into an identifier correspondence.

[0126] See also Figure 7 , Figure 7 This is a flowchart of the multimedia stream processing method provided by the embodiment of the present application. Figure 2 ;like Figure 7As shown, in the embodiment of the present application, step 101 also includes step 114A or step 114B; that is, for multiple file blocks of the first multimedia stream, after obtaining the first target stream segment identifier corresponding to each file block, the multimedia stream processing method also includes step 114A or step 114B, and each step is described below.

[0127] Step 114A: When the target media type identifier in the first target stream segment identifier fails to match in the initial corresponding relationship, the generating end device ends the processing of the file body block.

[0128] It should be noted that the identification correspondence is obtained by expanding the initial correspondence. When the production-end device obtains the initial correspondence, the first multimedia stream is parsed based on the initial correspondence. During the parsing process, since the target media type identifier in the first target stream segment identifier may be a media type initial identifier or a media type extension identifier; and when the target media type identifier is a media type extension identifier, the production-end device cannot match the media type initial identifier that matches the target media type identifier in the initial correspondence, and the match fails; at this time, the production-end device ends the processing of the file body block; in this way, the processing of the file body block can be skipped. Among them, the target media type identifier is used to determine the target media type.

[0129] Step 114B, when the target media type identifier is successfully matched in the initial correspondence, the generating end device determines the media type initial identifier that matches the target media type identifier in the initial correspondence as the target media type initial identifier, and converts the first multimedia stream into the second multimedia stream based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence.

[0130] It should be noted that, in the process of the production-end device parsing the first multimedia stream based on the initial correspondence, when the target media type identifier is the media type initial identifier, the production-end device can match the media type initial identifier that matches the target media type identifier in the initial correspondence, thereby successfully matching; at this time, the production-end device can correctly parse the file body block and achieve correct parsing of a track content.

[0131] In the embodiment of the present application, the process of the consumer device parsing the second media stream based on the initial correspondence is similar to the process described in step 114A and step 114B, and the embodiment of the present application will not be repeated here.

[0132] It can be understood that, since the initial correspondence represents the correspondence between the media type initial identifier and the message type initial identifier; therefore, by using each media type to be extended and the message type extension identifier corresponding to the media type to be extended, when the first multimedia stream is processed based on the initial correspondence, when the target media type identifier in the first target stream segment identifier is the media type extension identifier, the processing of the file body block can be terminated; and when the second multimedia stream is processed based on the initial correspondence, when the target message type identifier in the second target stream segment identifier is the message type extension identifier, the processing of the second target stream segment identifier can be terminated; thereby, correct parsing can be achieved and compatibility can be achieved.

[0133] In an embodiment of the present application, after step 110, the production-end device expands the initial correspondence to obtain an identification correspondence, so that the media stream processing method may also include: the production-end device expands the media type to be extended to obtain at least two first stream identifiers and at least two second stream identifiers; the two first stream identifiers are respectively combined with the media type initial identifier in the initial correspondence to obtain at least two first stream segment identifiers; the two second stream identifiers are combined with the message type initial identifier in the initial correspondence to obtain at least two second stream segment identifiers; finally, based on the correspondence between the at least two first stream segment identifiers and the at least two second stream segment identifiers, an identification correspondence is obtained.

[0134] See also Figure 8 , Figure 8 This is a flowchart of a multimedia stream processing method provided by an embodiment of the present application. Figure 3 ;like Figure 8 As shown, in the embodiment of the present application, after step 108, the process of consuming the first multimedia stream is also included; that is, after the consumer-end device generates the first multimedia stream including multiple file blocks, the multimedia stream processing method also includes steps 115 to 119, and each step is explained below.

[0135] Step 115: The consumer device obtains a stream segment corresponding to each file block of the first multimedia stream, and obtains a plurality of stream segments corresponding to the plurality of file blocks.

[0136] It should be noted that, since each file block is used to store a stream segment of the first media stream of the target media type, the consumer-end device can obtain a corresponding stream segment from each file block of the first multimedia stream; thus, multiple stream segments can be obtained for multiple file blocks of the first multimedia stream.

[0137] Step 116: The consumer device obtains a parsing result of the first multimedia stream based on the multiple stream segments.

[0138] It should be noted that the consumer device can directly determine the multiple stream segments as the parsing result of the first multimedia stream. The consumer device can also splice the same first multimedia stream of the same media type based on the attributes of the combination order such as the timestamp in the file body block, and determine the spliced ​​full first multimedia stream as the parsing result of the first multimedia stream.

[0139] Step 117: The consumer device obtains an identifier of the data to be consumed in response to the multimedia consumption request.

[0140] In an embodiment of the present application, a multimedia consumption request is used to request consumption of a parsing result of a first multimedia stream; wherein, the identifier of data to be consumed in the multimedia consumption request is used to select data from the parsing result, which may be the full amount of data in the parsing result, or partial data in the parsing result (for example, one or more first media streams, a combination of first media streams of different media types, etc.), etc., which is not limited to this in the embodiment of the present application.

[0141] Step 118: The consumer-end device obtains the data to be consumed corresponding to the identifier of the data to be consumed from the parsing result.

[0142] It should be noted that the data to be consumed obtained by the consumer-end device may be the full amount of data in the parsing result, or may be part of the data in the parsing result, etc. The data to be consumed is used for data consumption.

[0143] Step 119: The consumer device consumes data based on the data to be consumed.

[0144] It should be noted that data consumption includes at least one of the following: distribution, playback, media processing and storage. Distribution can be the distribution of the data to be consumed to other clients or servers; playback refers to the playback of the data to be consumed on the consumer-end device; media processing refers to the data processing such as transcoding and resolution modification on the consumer-end device; and storage refers to storage on the consumer-end device.

[0145] In the embodiment of the present application, the production-end device and the consumption-end device can be changed according to the actual scenario. For example, when the consumption-end device distributes the data to be consumed, it is changed to the production-end device.

[0146] The following will describe an exemplary application of the embodiment of the present application in a practical application scenario. The exemplary application describes a process of implementing multi-track multimedia stream processing based on RTMP / FLV.

[0147] It should be noted that the FLV container format includes an FLV file header and an FLV file body, wherein the FLV file body format is shown in Table 1.

[0148] Table 1

[0149]

[0150]

[0151] It should be noted that each tag (called a file body block) is used to store a stream segment of a media stream of a media type in a multimedia stream; and the FLV tag format is shown in Table 2.

[0152] Table 2

[0153]

[0154] It can be seen from the fields in Table 1 and Table 2 that the current FLV container format includes one track (one audio track, one video track, and one subtitle track) for the same media type data.

[0155] It should be noted that in the application scenarios of multimedia content, there are scenarios where the same media type carries multiple audio tracks; for example, different audio tracks can use different encoding formats and encoding parameters to provide differentiated audio playback experiences for different devices; different audio tracks can be in different languages ​​to increase the diversity of multimedia content choices. In addition, in addition to multiple audio tracks, there can also be multiple video tracks, multiple subtitle tracks, and other scenarios.

[0156] Here, the support for multiple tracks is achieved by modifying the media type field and the track identification field in the FLV tag; the modified FLV tag format is shown in Table 3.

[0157] Table 3

[0158]

[0159] It should be noted that in the modified FLV tag format, the track identifiers of the same media type (for example, the media type corresponding to the audio type identifier 8 and the extended audio type identifier 24) are different, while the track identifiers of different media types (for example, the media type corresponding to the audio type identifier 8 and the extended video type identifier 25) can be the same. For example, when the multimedia content includes video 1, audio 1, and audio 2, tag 1 can store the video stream segment of video 1, tag 2 can store the audio stream segment of audio 1, tag 3 can store the audio stream segment of audio 2, and tag 4 can store the next video stream segment of video 1... in this staggered arrangement. Among them, tag 2 and tag 3 are different in track identifiers, so that multi-audio track transmission is achieved.

[0160] It should also be noted that the audio type identifier 8, the video type identifier 9, and the script type identifier 18 are all referred to as the above-mentioned media type initial identifiers; the extended audio type identifier 24, the extended video type identifier 25, the extended subtitle identifier and other script type identifiers 26 are all referred to as the above-mentioned media type extended identifiers.

[0161] In addition, you can also select a default track from multiple tracks (for example, the default audio track is the audio track corresponding to the audio type identifier 8, the default video track is the video track corresponding to the video type identifier 9, and the default audio track is the subtitle track corresponding to the subtitle type identifier 18) to achieve backward compatibility.

[0162] It is understandable that, since the track identification field is not processed in old electronic devices that have not implemented the FLV tag format modification, the media type field is also modified accordingly when the track identification field is modified, so that the processing of non-default media type data can be automatically skipped in the old electronic devices, the extended media type is ignored, and the default media type data is processed normally to ensure the correctness of the parsing.

[0163] It should also be noted that the transmission protocol of the multimedia stream in the FLV container format is RTMP, wherein the RTMP message format is shown in Table 4.

[0164] Table 4

[0165]

[0166] Accordingly, based on the modified FLV tag format, the RTMP message format is modified as shown in Table 5

[0167] Table 5

[0168]

[0169]

[0170] It should be noted that the message type field in the modified RTMP message format corresponds to the first byte of the modified FLV tag format (including the reserved field, the filter identifier field and the media type field). The newly added message type in the message type field of the modified RTMP message format corresponds to the newly added media type in the media type field of the modified FLV tag format. In addition, in the RTMP message format, the audio type, video type and subtitle type share a fixed value, while in the modified RTMP message format, the track identifiers of the same message type (for example, the message type corresponding to the audio type identifier 8 and the extended audio type identifier 24) are different, while the track identifiers of different message types (for example, the message type corresponding to the audio type identifier 8 and the extended video type identifier 25) can be the same.

[0171] It should also be noted that the audio type identifier 8, the video type identifier 9, and the script type identifier 18 are all referred to as the above-mentioned message type initial identifiers; the extended audio type identifier 24, the extended video type identifier 25, the extended subtitle identifier and other script type identifiers 26 are all referred to as the above-mentioned message type extended identifiers.

[0172] In an embodiment of the present application, when a multimedia production end (referred to as a production-end device) sends a multimedia stream to a multimedia consumption end (referred to as a consumption-end device), the multimedia production end maps the media type identifier and track identifier (collectively referred to as the first stream segment identifier) ​​in the FLV multimedia stream (referred to as the first multimedia stream) to the message type identifier and track identifier (collectively referred to as the second stream segment identifier) ​​in the RTMP multimedia stream (referred to as the second multimedia stream) to achieve conversion of the FLV multimedia stream to the RTMP multimedia stream, and then achieves transmission of the multimedia stream between the multimedia production end and the multimedia consumption end based on the RTMP multimedia stream. In addition, after the multimedia consumption end receives the RTMP multimedia stream from the multimedia production end, it maps the message type and track identifier in the RTMP multimedia stream to the media type identifier and track identifier in the FLV multimedia stream to achieve conversion of the RTMP multimedia stream to the FLV multimedia stream, and then parses the FLV multimedia stream to obtain media data of different tracks.

[0173] It should be noted that the same type of multi-track transmission provided in the embodiments of the present application is applied to live streaming between media content production and media servers, forwarding between media server clusters, and content distribution from media servers to user sides.

[0174] For example, see Fig. 9 , Fig. 9 is a schematic diagram of an exemplary implementation of multi-track transmission provided in an embodiment of the present application; Fig. 9As shown, the media content production end 9-1 pushes the stream to the cloud device 9-2, and the cloud device 9-2 processes the pushed RTMP multimedia stream to respond to the pull stream of the playback end 9-3. Among them, the cloud device 9-2 includes an access layer 9-21, which is used to receive the RTMP multimedia stream pushed by the media content production end 9-1; the cloud device 9-2 also includes a media processing device 9-22, which is used to convert the RTMP multimedia stream into an FLV multimedia stream and perform media processing on the FLV multimedia stream, such as transcoding, mixing, etc.; the cloud device 9-2 also includes a media distribution device 9-23, which is used to distribute the pulled RTMP multimedia stream converted from the FLV multimedia stream to the playback end 9-3. Here, the process of pushing and pulling streams adopts RTMP; in addition, in the process of pushing streams, the media content production end 9-1 is a multimedia production end, and the cloud device 9-2 is a multimedia consumption end; in the process of pulling streams, the cloud device 9-2 is a multimedia production end, and the playback end 9-3 is a multimedia consumption end.

[0175] It can be understood that the embodiment of the present application implements a multimedia stream processing method for transmission, distribution, storage and playback of RTMP / FLV multi-track (or multi-video track, multi-subtitle track) that is backward compatible, which can achieve compatibility, simplify the transmission structure, reduce the consumption of transmission resources, and ensure business stability; in addition, the processing process of the multimedia content of the backward compatible RTMP / FLV multi-track provided by the embodiment of the present application supports multiple audio tracks (or multiple video tracks, multiple subtitle tracks) in one RTMP transmission, simplifies the central scheduling management and maintenance architecture, reduces the coupling degree of the system structure, improves the flexibility of business processing, and improves the accuracy of transmission; and the processing process of the multimedia content of the backward compatible RTMP / FLV multi-track provided by the embodiment of the present application is implemented by modifying the original field, which reduces the space occupied by the additional field; in addition, the processing process of the multimedia content of the backward compatible RTMP / FLV multi-track provided by the embodiment of the present application can improve the flexibility of multimedia content application, for example, provide audio and video content of different qualities, provide multi-language internationalization capabilities, and so on.

[0176] The following is a description of an exemplary structure of the first multimedia stream processing device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 3 As shown, the software modules stored in the first multimedia stream processing device 255 of the first memory 250 may include:

[0177] A first identifier acquisition module 2551 is configured to acquire, for each of a plurality of file body blocks of a first multimedia stream, a first target stream segment identifier corresponding to each of the file body blocks, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams of the same media type, and the file body block is used to store stream segments of the first media stream;

[0178] The first identifier acquisition module 2551 is further used to determine a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, wherein the second target stream segment identifier includes a second target media stream for determining a target message type;

[0179] The first conversion module 2552 is used to convert the first multimedia stream into a second multimedia stream based on a plurality of second target stream segment identifiers corresponding to a plurality of the file body blocks, wherein the second multimedia stream is used to be transmitted to a consumer device.

[0180] In the embodiment of the present application, the first multimedia stream processing device 255 also includes a relationship acquisition module 2553, which is used to determine at least one media type to be extended from at least one of the media types corresponding to the initial corresponding relationship, and perform the following processing on each of the media types to be extended: expand the media type to be extended to obtain at least two extended first stream identifiers, a media type extension identifier, at least two extended second stream identifiers and a message type extension identifier; combine one of the extended first stream identifiers with the media type initial identifier in the initial corresponding relationship, and combine at least one of the extended first stream identifiers with the media type extension identifier respectively to obtain at least two first stream segment identifiers; combine one of the extended second stream identifiers with the message type initial identifier in the initial corresponding relationship, and combine at least one of the extended second stream identifiers with the message type extension identifier respectively to obtain at least two second stream segment identifiers; obtain the identifier correspondence based on the correspondence between the at least two first stream segment identifiers and the at least two second stream segment identifiers.

[0181] In an embodiment of the present application, the media type includes at least one of the following: audio type, video type, and subtitle type.

[0182] In an embodiment of the present application, at least two of the first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bit rate, resolution, encoding format, and language type.

[0183] In the embodiment of the present application, the first multimedia stream processing device 255 also includes a data parsing module 2554, which is used to end the processing of the file body block when the target media type identifier in the first target stream segment identifier fails to match in the initial correspondence relationship, wherein the target media type identifier is used to determine the target media type; when the target media type identifier matches successfully in the initial correspondence relationship, the media type initial identifier that matches the target media type identifier in the initial correspondence is determined as the target media type initial identifier, and the first multimedia stream is converted into the second multimedia stream based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence.

[0184] In the embodiment of the present application, the container format used by the first multimedia stream is a streaming media format, and the second multimedia stream uses a container format supported by a real-time messaging protocol.

[0185] In an embodiment of the present application, the first multimedia stream processing device 255 also includes a data acquisition module 2555, which is used to determine the first target stream segment identifier of each first media stream from each first stream segment identifier included in the identifier correspondence relationship for at least one first media stream corresponding to each media type of the multimedia to be transmitted, wherein the multimedia to be transmitted corresponds to at least one media type; for each stream segment of each first media stream, generate the file body block including the first target stream segment identifier; and generate the first multimedia stream corresponding to the multimedia to be transmitted from each file body block.

[0186] The following is a description of an exemplary structure of the second multimedia stream processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 4 As shown, the software modules stored in the second multimedia stream processing device 455 of the second memory 450 may include:

[0187] A second identifier acquisition module 4551 is used to acquire a plurality of second target stream segment identifiers from a second multimedia stream, wherein the second target stream segment identifiers are used to determine a second target media stream of a target message type, and the second multimedia stream is transmitted by a production end device;

[0188] The second identifier acquisition module 4551 is further used to determine a first target stream segment identifier corresponding to each second target stream segment identifier based on an identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type;

[0189] A second conversion module 4552 is configured to generate a file body block including the first target stream segment identifier, and obtain a plurality of the file body blocks corresponding to the plurality of the second target stream segment identifiers, wherein the file body block is used to store the stream segment of the first media stream;

[0190] The second conversion module 4552 is further used to generate a first multimedia stream including multiple file blocks, wherein the first multimedia stream includes at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

[0191] In the embodiment of the present application, the second multimedia stream processing device 455 also includes a data consumption module 4553, which is used to obtain the stream segment corresponding to each file block of the first multimedia stream, and obtain multiple stream segments corresponding to multiple file blocks; based on the multiple stream segments, obtain the parsing result of the first multimedia stream; in response to a multimedia consumption request, obtain a data identifier to be consumed; obtain the data to be consumed corresponding to the data identifier to be consumed from the parsing result; and perform data consumption based on the data to be consumed, wherein the data consumption includes at least one of the following: distribution, playback, media processing and storage.

[0192] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer-readable storage medium. The first processor of the production-end device reads the computer executable instructions or computer programs from the computer-readable storage medium, and the first processor executes the computer executable instructions or computer programs, so that the production-end device executes the multimedia stream processing method applied to the production-end device mentioned in the embodiment of the present application. The second processor of the consumer-end device reads the computer executable instructions or computer programs from the computer-readable storage medium, and the second processor executes the computer executable instructions or computer programs, so that the consumer-end device executes the multimedia stream processing method applied to the consumer-end device mentioned in the embodiment of the present application.

[0193] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a first processor, the first processor will be caused to execute the multimedia stream processing method applied to the production end device provided by the present application embodiment; or, when the computer executable instructions or computer programs are executed by the first processor, the second processor will be caused to execute the multimedia stream processing method applied to the consumption end device provided by the present application embodiment; for example, Figure 5 The multimedia stream processing method is shown.

[0194] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0195] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0196] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0197] As an example, computer-executable instructions may be deployed to be executed on multiple electronic devices located at one location (in this case, the multiple electronic devices located at one location are production-end devices and consumer-end devices), or, to be executed on multiple electronic devices distributed at multiple locations and interconnected by a communication network (in this case, the multiple electronic devices distributed at multiple locations and interconnected by a communication network are production-end devices and consumer-end devices).

[0198] It is understandable that in the embodiments of the present application, when data related to multimedia content is involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. In addition, in the embodiments of the present application, the implementation of the data capture technical solution involved, when the above embodiments of the present application are applied to specific products or technologies, the collection, use and processing of relevant data should comply with the requirements of national laws and regulations, comply with the principles of legality, legitimacy and necessity, do not involve the acquisition of data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0199] To summarize, in the embodiment of the present application, since each file body block used to represent the first multimedia stream includes a first target stream segment identifier, and the first target stream segment identifier is used to determine the first target media stream of the target media type, the first multimedia stream can include at least two first media streams belonging to the same media type, which is a one-type multi-track multimedia content; then, based on the identifier correspondence between the first stream segment identifier and the second stream segment identifier, the first multimedia stream is converted into a second multimedia stream for transmission; in this way, the transmission of a one-type multi-track multimedia stream can be achieved through one transmission, which can reduce resource transmission consumption, improve transmission flexibility, and achieve business compatibility and improve business stability.

[0200] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A multimedia stream processing method, characterized in that: The method comprises: For a plurality of file blocks of a first multimedia stream, obtaining a first target stream segment identifier corresponding to each of the file blocks, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams belonging to the same media type, and the file block is used to store stream segments of the first media stream; Determine, based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, a second target stream segment identifier corresponding to the first target stream segment identifier, wherein the second target stream segment identifier includes a second target media stream for determining a target message type; Based on a plurality of the second target stream segment identifiers corresponding to a plurality of the file body blocks, the first multimedia stream is converted into a second multimedia stream, wherein the second multimedia stream is used for transmission to a consumer device.

2. The method according to claim 1, characterized in that The method further comprises: Determine at least one media type to be extended from at least one media type corresponding to the initial corresponding relationship, and perform the following processing on each media type to be extended: Extending the media type to be extended to obtain at least two extended first stream identifiers, a media type extension identifier, at least two extended second stream identifiers, and a message type extension identifier; Combining an extended first stream identifier with the media type initial identifier in the initial corresponding relationship, and combining at least one extended first stream identifier with the media type extended identifier respectively, to obtain at least two first stream segment identifiers; Combining an extended second stream identifier with the message type initial identifier in the initial corresponding relationship, and combining at least one extended second stream identifier with the message type extended identifier respectively, to obtain at least two second stream segment identifiers; The identifier corresponding relationship is obtained based on the corresponding relationship between at least two of the first stream segment identifiers and at least two of the second stream segment identifiers.

3. The method according to claim 1 or 2, characterized in that: The media type includes at least one of the following: audio type, video type, and subtitle type.

4. The method according to claim 1 or 2, characterized in that: At least two of the first media streams belonging to the same media type correspond to different media attributes, wherein the media attributes include at least one of the following: bit rate, resolution, encoding format and language type.

5. The method according to claim 1 or 2, characterized in that: After obtaining the first target stream segment identifier corresponding to each of the plurality of file blocks of the first multimedia stream, the method further comprises: When the target media type identifier in the first target stream segment identifier fails to match in the initial corresponding relationship, the processing of the file body block is terminated, wherein the target media type identifier is used to determine the target media type; When the target media type identifier is successfully matched in the initial correspondence, the media type initial identifier in the initial correspondence that matches the target media type identifier is determined as the target media type initial identifier, and the first multimedia stream is converted into the second multimedia stream based on the message media type initial identifier corresponding to the target media type initial identifier in the initial correspondence.

6. The method according to claim 1 or 2, characterized in that: The container format adopted by the first multimedia stream is a streaming media format, and the second multimedia stream adopts a container format supported by a real-time messaging protocol.

7. The method according to claim 1 or 2, characterized in that: Before obtaining the first target stream segment identifier corresponding to each of the plurality of file blocks of the first multimedia stream, the method further comprises: For at least one of the first media streams corresponding to each of the media types of the multimedia to be transmitted, determining the first target stream segment identifier of each of the first media streams from the first stream segment identifiers included in the identifier correspondence, wherein the multimedia to be transmitted corresponds to at least one of the media types; For each of the stream segments of each of the first media streams, generating the file body block including the first target stream segment identifier; The first multimedia stream corresponding to the multimedia to be transmitted is generated from each of the file blocks.

8. A multimedia stream processing method, characterized in that: The method comprises: Acquire a plurality of second target stream segment identifiers from a second multimedia stream, wherein the second target stream segment identifiers are used to determine a second target media stream of a target message type, and the second multimedia stream is transmitted by a production end device; Determine, based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, a first target stream segment identifier corresponding to each of the second target stream segment identifiers, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type; Generate a file body block including the first target stream segment identifier, and obtain a plurality of the file body blocks corresponding to the plurality of the second target stream segment identifiers, wherein the file body block is used to store the stream segment of the first media stream; A first multimedia stream including a plurality of the file blocks is generated, wherein the first multimedia stream includes at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

9. The method according to claim 8, characterized in that After generating the first multimedia stream including the plurality of file blocks, the method further comprises: Acquire the stream segment corresponding to each file body block of the first multimedia stream, and obtain a plurality of stream segments corresponding to a plurality of file body blocks; Obtaining a parsing result of the first multimedia stream based on the plurality of stream segments; In response to the multimedia consumption request, obtaining an identifier of data to be consumed; Acquire the to-be-consumed data corresponding to the to-be-consumed data identifier from the parsing result; Data consumption is performed based on the data to be consumed, wherein the data consumption includes at least one of the following: distribution, playback, media processing, and storage.

10. A first multimedia stream processing device, characterized in that: The first multimedia stream processing device comprises: A first identifier acquisition module is used to acquire, for each of the plurality of file body blocks of the first multimedia stream, a first target stream segment identifier corresponding to the file body block, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type, the first multimedia stream includes at least two first media streams of the same media type, and the file body block is used to store stream segments of the first media stream; The first identifier acquisition module is further used to determine a second target stream segment identifier corresponding to the first target stream segment identifier based on the identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, wherein the second target stream segment identifier includes a second target media stream used to determine a target message type; The first conversion module is used to convert the first multimedia stream into a second multimedia stream based on a plurality of second target stream segment identifiers corresponding to a plurality of the file body blocks, wherein the second multimedia stream is used to be transmitted to a consumer device.

11. A second multimedia stream processing device, characterized in that: The second multimedia stream processing device comprises: A second identifier acquisition module, used to acquire a plurality of second target stream segment identifiers from a second multimedia stream, wherein the second target stream segment identifiers are used to determine a second target media stream of a target message type, and the second multimedia stream is transmitted by a production end device; The second identifier acquisition module is further used to determine a first target stream segment identifier corresponding to each of the second target stream segment identifiers based on an identifier correspondence relationship between the first stream segment identifier and the second stream segment identifier, wherein the first target stream segment identifier is used to determine a first target media stream of a target media type; A second conversion module is used to generate a file body block including the first target stream segment identifier, and obtain a plurality of the file body blocks corresponding to the plurality of the second target stream segment identifiers, wherein the file body block is used to store the stream segment of the first media stream; The second conversion module is further used to generate a first multimedia stream including multiple file blocks, wherein the first multimedia stream includes at least two first media streams belonging to the same media type, and the first multimedia stream is used for data consumption on a consumer device.

12. A production-end device for multimedia stream processing, characterized in that: The production end equipment includes: A first memory, used to store computer executable instructions or computer programs; The first processor is configured to implement the multimedia stream processing method according to any one of claims 1 to 7 when executing the computer executable instructions or computer programs stored in the first memory.

13. A consumer device for multimedia stream processing, characterized in that: The consumer device comprises: A second memory, used to store computer executable instructions or computer programs; The second processor is used to implement the multimedia stream processing method according to claim 8 or 9 when executing the computer executable instructions or computer program stored in the second memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by the first processor, the multimedia stream processing method described in any one of claims 1 to 7 is implemented; or, when the computer executable instructions or computer program are executed by the second processor, the multimedia stream processing method described in claim 8 or 9 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by the first processor, the multimedia stream processing method described in any one of claims 1 to 7 is implemented; or, when the computer executable instructions or computer program are executed by the second processor, the multimedia stream processing method described in claim 8 or 9 is implemented.

Images