A multimedia stream processing method, device, storage medium and program product

Abstract

The present application provides a multimedia stream processing method, device, storage medium and program product, which are applied to various scenarios such as cloud technology and artificial intelligence; the multimedia stream processing method includes: receiving a multimedia stream to be decoded sent by a server device, and obtaining at least one event trigger moment corresponding to the multimedia stream to be decoded, where the at least one event trigger moment is obtained by adjusting at least one event marking moment based on stream time change information, and the stream time change information refers to the time change before and after the encoding of the multimedia stream, and each event marking moment refers to an event moment in the multimedia stream before encoding; decoding the multimedia stream to be decoded and playing the multimedia stream to be played; when the multimedia stream to be played reaches a target event trigger moment, performing a specified process corresponding to the target event trigger moment, where the target event trigger moment is any one of the at least one event trigger moment. Through the present application, the accuracy of triggering the specified process can be improved.

Description

Technical Field

[0001] This application relates to information processing technology in the field of computer applications, and in particular, to a multimedia stream processing method, device, storage medium, and program product. Background Art

[0002] In the process of processing multimedia information such as audio and video, it is usually necessary to mark the event time in the multimedia information on the server device and transmit the marked event time to the playback device, so that the playback device can trigger specified processing when the multimedia information is played to the event time; for example, mark the "pentakill" time on the server device and transmit the marked "pentakill" time to the playback device, so that when the game live broadcast reaches the "pentakill" time, the playback device can trigger the presentation of virtual red envelopes.

[0003] Generally speaking, in order for the playback device to obtain the event time, the server device usually directly sends the marked event time to the playback device. However, since the multimedia information received by the playback device is the multimedia information encoded by the server device, and the encoding of the multimedia information will change the time stamp corresponding to the frame information of the multimedia information, therefore, the time corresponding to the event time obtained by the playback device in the encoded multimedia information will no longer be the time when the event occurs, resulting in the inability to accurately trigger specified processing at the time when the event of the multimedia information occurs, so the accuracy of triggering specified processing is relatively low. Summary of the Invention

[0004] Embodiments of this application provide a multimedia stream processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of triggering specified processing.

[0005] The technical solution of the embodiments of this application is implemented as follows:

[0006] Embodiments of this application provide a multimedia stream processing method, including:

[0007] Receiving a multimedia stream to be decoded sent by a server device, and obtaining at least one event trigger time corresponding to the multimedia stream to be decoded, where at least one of the event trigger times is obtained by adjusting at least one event marking time based on stream time change information, and the stream time change information refers to the time change before and after the encoding of the multimedia stream, and each event marking time refers to the event time in the multimedia stream before encoding;

[0008] Playing the multimedia stream to be played by decoding the multimedia stream to be decoded;

[0009] When the multimedia stream to be played reaches the target event trigger moment, perform the specified processing corresponding to the target event trigger moment, where the target event trigger moment is any one of at least one of the event trigger moments.

[0010] An embodiment of the present application further provides a multimedia stream processing method, including:

[0011] Perform event marking on the multimedia stream to be encoded to obtain at least one event marking moment;

[0012] Encode the multimedia stream to be encoded to obtain a multimedia stream to be decoded;

[0013] Obtain the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded;

[0014] Based on at least one of the event marking moments and the stream time change information, send the multimedia stream to be decoded to the playback device, so that the playback device plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and perform the specified processing corresponding to the target event trigger moment when the multimedia stream to be played reaches the target event trigger moment. The target event trigger moment is any one of at least one event trigger moment, and at least one of the event trigger moments is obtained by adjusting at least one of the event marking moments based on the stream time change information.

[0015] An embodiment of the present application provides a first multimedia stream processing device, including:

[0016] An information acquisition module, configured to receive the multimedia stream to be decoded sent by the server device, and obtain at least one event trigger moment corresponding to the multimedia stream to be decoded, where at least one of the event trigger moments is obtained by adjusting at least one event marking moment based on the stream time change information, and the stream time change information refers to the time change before and after multimedia stream encoding, and each of the event marking moments refers to the event moment in the multimedia stream before encoding;

[0017] An information playback module, configured to play the multimedia stream to be played by decoding the multimedia stream to be decoded;

[0018] A processing trigger module, configured to perform the specified processing corresponding to the target event trigger moment when the multimedia stream to be played reaches the target event trigger moment, where the target event trigger moment is any one of at least one of the event trigger moments.

[0019] In an embodiment of the present application, the information playback module is further configured to receive the stream time change information and at least one of the event marking moments sent by the server device; and adjust at least one of the event marking moments based on the stream time change information to obtain at least one event trigger moment.

[0020] In an embodiment of the present application, the stream time change information includes the timestamp to be encoded and the encoded timestamp of the target frame information; the information playback module is further configured to obtain the time change amount between the encoded timestamp and the timestamp to be encoded; and fuse the time change amount with each of the event marking moments to obtain at least one event trigger moment corresponding to at least one of the event marking moments.

[0021] In an embodiment of the present application, the information playback module is further configured to receive at least one of the event trigger moments corresponding to the multimedia stream to be decoded sent by the server device, where at least one of the event trigger moments is obtained by the server device adjusting at least one of the event marking moments based on the stream time change information.

[0022] In an embodiment of the present application, the specified processing includes at least one of presenting virtual resource information, presenting specified special effects, inserting multimedia information, presenting Q&A information, and presenting recommended information.

[0023] An embodiment of the present application provides a second multimedia stream processing device, including:

[0024] An event marking module, configured to perform event marking on the multimedia stream to be encoded to obtain at least one event marking moment;

[0025] An information encoding module, configured to encode the multimedia stream to be encoded to obtain a multimedia stream to be decoded;

[0026] A time acquisition module, configured to acquire the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded;

[0027] An information sending module, configured to send the multimedia stream to be decoded to a playback device based on at least one of the event marking moments and the stream time change information, so that the playback device plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and performs specified processing corresponding to the target event trigger moment when the multimedia stream to be played is played to the target event trigger moment, where the target event trigger moment is any one of at least one event trigger moment, and at least one of the event trigger moments is obtained by adjusting at least one of the event marking moments based on the stream time change information.

[0028] In an embodiment of the present application, the information sending module is further configured to send at least one of the event marking moments, the stream time change information, and the multimedia stream to be decoded to the playback device, so that the playback device adjusts at least one of the event marking moments based on the stream time change information to obtain at least one of the event triggering moments.

[0029] In an embodiment of the present application, the information sending module is further configured to adjust at least one of the event marking moments based on the stream time change information to obtain at least one of the event triggering moments; and send at least one of the event triggering moments and the multimedia stream to be decoded to the playback device.

[0030] In an embodiment of the present application, the event marking is implemented by at least one of a marking operation, a neural network model, and marking information, where the marking operation refers to an operation for determining the event marking moment, the neural network model is used to determine the event marking moment by obtaining the content of the multimedia stream to be played, and the neural network model is trained based on multimedia stream samples and annotated event moments, and the marking information is used to represent the event marking moment.

[0031] In an embodiment of the present application, the time acquisition module is further configured to acquire the timestamp to be encoded corresponding to the target frame information of the multimedia stream to be encoded; acquire the encoding method of the multimedia stream to be decoded; determine the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded based on the encoding method; and determine the timestamp to be encoded and the encoding timestamp as the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded.

[0032] In an embodiment of the present application, the time acquisition module is further configured to, when the encoding method is the first specified encoding method, determine the timestamp to be encoded as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the first specified encoding method is a method that does not change the playback timestamp during the encoding process; when the encoding method is the second specified encoding method, acquire a synchronization timestamp and determine the synchronization timestamp as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the second specified encoding method is a method that determines the playback timestamp based on synchronized time information during the encoding process; when the encoding method is the third specified encoding method, determine the specified timestamp as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the third specified encoding method is a method that determines the playback timestamp based on specified time information during the encoding process.

[0033] An embodiment of the present application provides a playback device, including:

[0034] A first memory for storing executable instructions;

[0035] A first processor, when executing the executable instructions stored in the first memory, implements the multimedia stream processing method applied to the playback device provided by the embodiments of the present application.

[0036] The embodiments of the present application provide a server device, including:

[0037] A second memory for storing executable instructions;

[0038] A second processor, when executing the executable instructions stored in the second memory, implements the multimedia stream processing method applied to the server device provided by the embodiments of the present application.

[0039] The embodiments of the present application provide a computer-readable storage medium storing executable instructions, where the executable instructions, when executed by the first processor, implement the multimedia stream processing method applied to the playback device provided by the embodiments of the present application; or, the executable instructions, when executed by the second processor, implement the multimedia stream processing method applied to the server device provided by the embodiments of the present application.

[0040] The embodiments of the present application provide a computer program product including a computer program or instructions, where the computer program or instructions, when executed by the first processor, implement the multimedia stream processing method applied to the playback device provided by the embodiments of the present application; or, the computer program or instructions, when executed by the second processor, implement the multimedia stream processing method applied to the server device provided by the embodiments of the present application.

[0041] The embodiments of the present application have at least the following beneficial effects: Since among the at least one event trigger moment obtained by the playback device, each event trigger moment is obtained by adjusting at least one event marking moment based on the stream time change information, and the stream time change information refers to the time change before and after multimedia stream encoding; thus, each event trigger moment obtained by the playback device is the occurrence moment of the event in the to-be-decoded live stream. In this way, during the playback of the to-be-played multimedia stream obtained by decoding the to-be-decoded multimedia stream, the playback content is matched with the event trigger moment; thereby, when the to-be-played multimedia stream is played to any event trigger moment, it is exactly the moment to play the event content, and at this time, triggering the specified processing can improve the accuracy of triggering the specified processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is the architecture schematic diagram of the multimedia stream processing system provided by the embodiments of the present application Figure 1 ;

[0043] Figure 2It is a schematic diagram of the architecture of the multimedia stream processing system provided by the embodiments of the present application Figure 2 ;

[0044] Figure 3 It is provided by the embodiments of the present application Figure 2 A schematic diagram of the composition structure of one of the terminals in

[0045] Figure 4 It is provided by the embodiments of the present application Figure 2 A schematic diagram of the composition structure of one of the servers in

[0046] Figure 5a It is a schematic diagram of the process of the multimedia stream processing method provided by the embodiments of the present application Figure 1 ;

[0047] Figure 5b A schematic diagram of the process of an exemplary multimedia stream processing method provided by the embodiments of the present application;

[0048] Figure 6 It is a schematic diagram of the process of the multimedia stream processing method provided by the embodiments of the present application Figure 2 ;

[0049] Figure 7 It is a schematic diagram of the process of the multimedia stream processing method provided by the embodiments of the present application Figure 3 ;

[0050] Figure 8 It is an exemplary schematic diagram of an event screen provided by the embodiments of the present application;

[0051] Figure 9 It is an exemplary schematic diagram of the change of a timestamp provided by the embodiments of the present application;

[0052] Figure 10 It is an exemplary schematic diagram of the architecture of a live stream system provided by the embodiments of the present application;

[0053] Figure 11 It is an exemplary schematic diagram of the encoding result of a live stream provided by the embodiments of the present application;

[0054] Figure 12 It is an exemplary schematic diagram of an application scenario for obtaining different live streams provided by the embodiments of the present application;

[0055] Figure 13 It is an exemplary schematic diagram of live stream interaction provided by the embodiments of the present application. Detailed implementation manners

[0056] To make the objectives, technical solutions and advantages of this application clearer, the following will further describe this application in detail with reference to the accompanying drawings. The described embodiments should not be construed as limitations on this application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0057] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0058] In the following description, the terms "first / second / third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0059] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used in the embodiments of this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are explained. The nouns and terms involved in the embodiments of this application are subject to the following explanations.

[0061] 1) Client: An application program running on a device that provides various services; for example, an instant messaging client, a video client, a live broadcast client, etc. A playback device is a device on which a client for implementing multimedia stream playback runs; a server device is a back-end server that interacts with the playback device and provides computing services to the playback device.

[0062] 2) Cloud Computing: A computing model in which computing tasks are distributed across a resource pool composed of a large number of computing devices, enabling various application systems to obtain computing power, storage space, and information services as needed; among them, the network that provides resources for the resource pool is called the "cloud". The resources in the "cloud" seem to be infinitely expandable to users, and can be obtained at any time, used on demand, expanded at any time, and paid according to usage; in the embodiments of this application, the server device can be a cloud device.

[0063] 3) Artificial Intelligence (AI) is a theory, method, technology, and application system that uses machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In the embodiments of the present application, event marking can be performed in combination with artificial intelligence. For example, based on artificial intelligence, a specified processing flow can be determined, and so on.

[0064] It should be noted that during the processing of multimedia information such as audio and video (also known as multimedia streams), it is usually necessary to mark the event time in the multimedia information on the server device and transmit the marked event time to the playback device, so that the playback device can trigger specified processing when the multimedia information is played to the event time.

[0065] Generally speaking, in order for the playback device to obtain the event time, the server device usually carries the event time in the supplemental enhancement information (SEI) of the frame information in the multimedia information and encodes the SEI, so that the playback device can obtain the event time by decoding the SEI. However, in the process of the playback device obtaining the event time above, it is required that the server device supports the encoding of SEI and the playback device supports the decoding of SEI; thus, it is only applicable to devices that support SEI encoding and decoding, and the applicability is poor.

[0066] In addition, the server device carrying the event time in the supplemental enhancement information of the multimedia information increases the size of the multimedia information. Thus, the bit rate of the multimedia information will be increased, and the requirement for the transmission network is relatively high. And after the playback device obtains the event time by decoding the SEI, it needs to send the event time to the application layer through a callback function, resulting in a large resource consumption and a low smoothness of the multimedia information playback. Moreover, when the playback device decodes the SEI, there is also a situation where the callback function cannot be called (such as when there is no interface for setting the callback function). Thus, the application layer will not be able to obtain the event time, and thus cannot trigger specified processing based on the event time; there is also a situation of decoding error, resulting in abnormal playback of the multimedia information.

[0067] In addition, if the server device directly sends the marked event time to the playback device, since the encoding of the multimedia information by the server device will change the playback timestamp corresponding to the frame information of the multimedia information, the time corresponding to the received event time in the decoded multimedia information will no longer be the time when the event occurred, resulting in the invalidation of the marked event time.

[0068] Based on this, embodiments of the present application provide a multimedia stream processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy and generality of triggering a specified event. The following describes an exemplary application of the device provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as various types of terminals such as smart phones, smart watches, laptop computers, tablet computers, desktop computers, smart home appliances, set-top boxes, in-vehicle intelligent devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable game devices, and smart speakers, or can be implemented as a server. Next, an exemplary application will be described when the playback device is implemented as a terminal and the server device is implemented as a server.

[0069] See Figure 1 , Figure 1 is a schematic architecture of the multimedia stream processing system provided by the embodiments of the present application Figure 1 ; as Figure 1 shown, to support a multimedia stream processing application, in the multimedia stream processing system 100, the terminal 400 (referred to as the playback device, and the terminal 400-1 and the terminal 400-2 are exemplarily shown) is connected to the server 200 (referred to as the server device) through the network 300. The network 300 can be a wide area network, a local area network, or a combination of the two. In addition, the multimedia stream processing system 100 further includes a database 500 for providing data support to the server 200; and, Figure 1 shown in

[0070] is a case where the database 500 is independent of the server 200. In addition, the database 500 can also be integrated in the server 200. The embodiments of the present application do not make any limitations in this regard.

[0071] A server 200 is configured to perform event marking on a multimedia stream to be encoded, obtaining at least one event marking moment; encode the multimedia stream to be encoded to obtain a multimedia stream to be decoded; acquire the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded; send the at least one event marking moment, the stream time change information, and the multimedia stream to be decoded to a terminal 400 via a network 300, so that the terminal 400 adjusts the at least one event marking moment based on the stream time change information to obtain at least one event triggering moment, plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and performs a specified process corresponding to the target event triggering moment when the multimedia stream to be played reaches the target event triggering moment.

[0072] Based on Figure 1 , see Figure 2 , Figure 2 is a schematic architecture of a multimedia stream processing system provided by an embodiment of the present application Figure 2 ; as Figure 2 shown, different from Figure 1 is the interaction process between the terminal 400 and the server 200.

[0073] A terminal 400 is configured to receive, via the network 300, the multimedia stream to be decoded sent by the server 200 and at least one event triggering moment corresponding to the multimedia stream to be decoded, where the at least one event triggering moment is obtained by the server 200 adjusting at least one event marking moment based on the stream time change information, the stream time change information refers to the time change before and after encoding of the multimedia stream, and each event marking moment refers to an event moment in the multimedia stream before encoding; play the multimedia stream to be played by decoding the multimedia stream to be decoded; and perform a specified process corresponding to the target event triggering moment when the multimedia stream to be played reaches the target event triggering moment, where the target event triggering moment is any one of the at least one event triggering moments.

[0074] A server 200 is configured to perform event marking on a multimedia stream to be encoded, obtaining at least one event marking moment; encode the multimedia stream to be encoded to obtain a multimedia stream to be decoded; acquire the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded; adjust at least one event marking moment based on the stream time change information to obtain at least one event triggering moment; and send the at least one event triggering moment and the multimedia stream to be decoded to the terminal 400 via the network 300, so that the terminal 400 plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and performs a specified process corresponding to the target event triggering moment when the multimedia stream to be played reaches the target event triggering moment, where the target event triggering moment is any one of the at least one event triggering moments.

[0075] In some embodiments, the server 200 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a smart watch, a laptop computer, a tablet computer, a desktop computer, a smart TV, a set-top box, a smart vehicle-mounted device, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, and a smart speaker, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected through wired or wireless communication methods, which are not limited in the embodiments of the present application.

[0076] See Figure 3 , Figure 3 is a schematic diagram of the composition structure of a terminal provided in the embodiments of the present application Figure 2 in one of them. Figure 3 The terminal 400 shown includes: at least one first processor 410, a first memory 450, at least one first network interface 420, and a first user interface 430. Each component in the terminal 400 is coupled together through a first bus system 440. It can be understood that the first bus system 440 is used to implement the connection and communication between these components. In addition to including a data bus, the first bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 3 all kinds of buses are labeled as the first bus system 440.

[0077] The first processor 410 may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0078] The first user interface 430 includes one or more first output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The first user interface 430 also includes one or more first input devices 432, including user interface components that assist user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons, and controls.

[0079] The first memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memories, hard disk drives, optical disc drives, etc. The first memory 450 optionally includes one or more storage devices that are physically remote from the first processor 410.

[0080] The first memory 450 includes volatile memory or non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The first memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0081] In some embodiments, the first memory 450 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which are described below by way of example.

[0082] The first operating system 451, including system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and handling hardware-based tasks;

[0083] The first network communication module 452 is used to reach other computer devices via one or more (wired or wireless) first network interfaces 420. Exemplary first network interfaces 420 include: Bluetooth, Wi-Fi (Wireless Fidelity), and USB (Universal Serial Bus), etc.;

[0084] The first presentation module 453 is used to enable the presentation of information (such as a user interface for operating peripheral devices and displaying content and information) via one or more first output devices 431 associated with the first user interface 430 (such as a display screen, a speaker, etc.);

[0085] The first input processing module 454 is used to detect and translate one or more user inputs or interactions from one of one or more first input devices 432.

[0086] In some embodiments, a first multimedia stream processing device provided by the embodiments of the present application may be implemented in software. Figure 3The first multimedia stream processing device 455 stored in the first memory 450 is shown. It can be software in the form of programs and plugins, etc., and includes the following software modules: an information acquisition module 4551, an information playback module 4552, and a processing trigger module 4553. These modules are logical, so they can be combined arbitrarily or further split according to the implemented functions. The functions of each module will be described below.

[0087] See Figure 4 , Figure 4 is the composition structure schematic diagram of a server provided by an embodiment of the present application Figure 2 in one of them. Figure 4 The server 200 shown includes: at least one second processor 210, a second memory 250, at least one second network interface 220, and a second user interface 230. Each component in the server 200 is coupled together through a second bus system 240. It can be understood that the second bus system 240 is used to realize the connection and communication between these components. In addition to including a data bus, the second bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 4 all kinds of buses are labeled as the second bus system 240.

[0088] The second processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0089] The second user interface 230 includes one or more second output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The second user interface 230 also includes one or more second input devices 232, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons, and controls.

[0090] The second memory 250 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disc drives, etc. The second memory 250 optionally includes one or more storage devices that are physically located away from the second processor 210.

[0091] The second memory 250 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory, and the volatile memory can be random access memory. The second memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0092] In some embodiments, the second memory 250 can store data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, which will be exemplarily described below.

[0093] The second operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0094] The second network communication module 252 is used to reach other computer devices via one or more (wired or wireless) second network interfaces 220. Exemplary second network interfaces 220 include: Bluetooth, Wi-Fi, and Universal Serial Bus, etc.;

[0095] The second presentation module 253 is used to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more second output devices 231 associated with the second user interface 230 (e.g., a display screen, a speaker, etc.);

[0096] The second input processing module 254 is used to detect and translate one or more user inputs or interactions from one of one or more second input devices 232.

[0097] In some embodiments, a second multimedia stream processing device provided by an embodiment of the present application can be implemented in software. Figure 4 Shown is the second multimedia stream processing device 255 stored in the second memory 250, which can be software in the form of programs and plugins, etc., including the following software modules: an event marking module 2551, an information encoding module 2552, a time acquisition module 2553, and an information sending module 2554. These modules are logical, so they can be combined arbitrarily or further split according to the functions to be implemented. The functions of each module will be described below.

[0098] In some embodiments, the first multimedia stream processing and the second multimedia stream processing devices provided in the embodiments of the present application may be implemented in a hardware manner. As an example, the first multimedia stream processing and the second multimedia stream processing devices provided in the embodiments of the present application may be processors in the form of a hardware decoding processor, which is programmed to execute the multimedia stream processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may employ one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0099] In some embodiments, a terminal or a server may implement the multimedia stream processing method provided in the embodiments of the present application by running a computer program. For example, the computer program may be a native program or a software module in an operating system; it may be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or an instant messaging APP; it may also be a small program, that is, a program that only needs to be downloaded into a browser environment to run; it may also be a small program that can be embedded into any APP. In short, the above computer program may be any form of application program, module, or plug-in.

[0100] Next, the multimedia stream processing method provided in the embodiments of the present application will be described in conjunction with the exemplary applications and implementations of the playback device and the server device provided in the embodiments of the present application. In addition, the multimedia stream processing method provided in the embodiments of the present application is applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and vehicle-mounted.

[0101] See Figure 5a , Figure 5a is a flowchart of the multimedia stream processing method provided in the embodiments of the present application Figure 1 , and will be described in conjunction with Figure 5a the steps shown.

[0102] S501. The server device performs event marking on the multimedia stream to be encoded to obtain at least one event marking moment.

[0103] In an embodiment of the present application, the server device includes a function for marking events; thus, in order to trigger specified processing based on events in the multimedia stream in the playback device, the server device marks the events in the multimedia stream to be encoded before encoding, so as to mark at least one event marking moment corresponding to at least one event in the multimedia stream to be encoded.

[0104] It should be noted that the multimedia stream to be encoded is a type of multimedia information, including at least one of an audio stream and a video stream, and can be a live stream or a VOD stream. The embodiments of the present application do not limit this. Each event in the at least one event is an event in the multimedia stream to be encoded, and the at least one events can be events of the same type (for example, events corresponding to highlight moments such as consecutive kill events in a game scene, goal events, etc., where the consecutive kill event is, for example, a "penta kill" event in a game scene), or can be events of different types (for example, kill events and events of controlling a hostile virtual object, etc.). The embodiments of the present application do not limit this. The at least one event corresponds to the at least one event marking moment one by one. Thus, each event marking moment is the occurrence moment of the event in the multimedia stream to be encoded, and the time axis of the at least one event marking moment is the same as the time axis of the multimedia stream to be encoded.

[0105] Here, when the server device includes an event marking device, the server device can implement event marking processing through the event marking device.

[0106] S502. The server device encodes the multimedia stream to be encoded to obtain a multimedia stream to be decoded.

[0107] In an embodiment of the present application, the server device further includes a function for encoding; thus, before sending the multimedia stream to be encoded to the playback device, the server device first encodes the multimedia stream to be encoded to improve the sending efficiency; here, the obtained encoding result is the multimedia stream to be decoded; that is to say, the multimedia stream to be decoded is the encoded multimedia stream to be encoded. Among them, the multimedia stream to be decoded corresponds to the multimedia information to be encoded and is also a type of multimedia information, including at least one of an audio stream and a video stream.

[0108] Here, when the server device further includes an encoding device, the server device can implement encoding processing through the encoding device.

[0109] It should be noted that the encoding processing and the event marking processing are not in a sequential order. The encoding processing can be executed first and then the event marking processing, or the event marking processing can be executed first and then the encoding processing, or the encoding processing and the event marking processing can be executed in parallel, etc. The embodiments of the present application do not limit this. Figure 5aThe execution order shown is only an exemplary one. Exemplarily, in a live broadcast scenario, event marking processing can be performed first and then encoding processing.

[0110] In an embodiment of the present application, when the server device may further include a multimedia stream distribution device, the server device may send the multimedia stream to be encoded to the encoding device and the event marking device respectively through the multimedia stream distribution device, so that the encoding device can implement encoding processing and the event marking device can implement event marking processing. Among them, S502 may be triggered by the following processing: the server device receives a play request sent by the playback device, and in response to the play request, S502 is executed.

[0111] S503. The server device obtains the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded.

[0112] In an embodiment of the present application, during the process of the server device obtaining the multimedia stream to be decoded, it also records the time change situation of the multimedia stream to be decoded relative to the multimedia stream to be encoded, and determines the time change situation of the multimedia stream to be decoded relative to the multimedia stream to be encoded that is recorded as the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded.

[0113] It should be noted that the stream time change information may be the time change amount between the multimedia stream to be encoded and the multimedia stream to be decoded, and may also be the moment of the target frame information in the multimedia stream to be encoded and the moment in the multimedia stream to be decoded, etc. The embodiments of the present application do not limit this; among them, the target frame information may be the frame with the same content in the multimedia stream to be encoded and the multimedia stream to be decoded, for example, the first frame, the end frame, the first frame or the end frame of a specified segment, etc.

[0114] Here, the server device may obtain the stream time change information through the encoding device.

[0115] S504. The server device sends the multimedia stream to be decoded to the playback device based on at least one event marking moment and the stream time change information.

[0116] In an embodiment of the present application, the server device may send at least one event marking moment, the stream time change information, and the multimedia stream to be decoded to the playback device, so that the playback device determines at least one event trigger moment based on at least one event marking moment and the stream time change information; the server device may also determine at least one event trigger moment based on at least one event marking moment and the stream time change information, and send at least one event trigger moment and the multimedia stream to be decoded to the playback device; etc. The embodiments of the present application do not limit this.

[0117] It should be noted that the server device obtains at least one event marking moment and stream time change information, which are used to obtain at least one event trigger moment corresponding to at least one event in the multimedia stream to be decoded. The process of obtaining at least one event trigger moment based on at least one event marking moment and stream time change information can be executed by the server device or by the playback device, and the embodiments of the present application do not limit this.

[0118] Here, after the server device sends the multimedia stream to be decoded to the playback device, the playback device receives the multimedia stream to be decoded.

[0119] S505. The playback device obtains at least one event trigger moment corresponding to the multimedia stream to be decoded.

[0120] It should be noted that at least one event trigger moment is obtained by adjusting at least one event marking moment based on the stream time change information. The stream time change information refers to the time change before and after the encoding of the multimedia stream, and each event marking moment refers to the event moment in the multimedia stream before encoding. Here, if the server device sends at least one event marking moment and stream time change information to the playback device, the playback device receives at least one event marking moment and stream time change information, and thus, the playback device can determine at least one event trigger moment based on at least one event marking moment and stream time change information; if the server device sends at least one event trigger moment to the playback device, the playback device receives at least one event trigger moment. That is to say, the playback device can obtain at least one event trigger moment by adjusting at least one event marking moment through the stream time change information, and can also obtain at least one event trigger moment by receiving at least one event trigger moment sent by the server device, and the embodiments of the present application do not limit this.

[0121] S506. The playback device decodes the multimedia stream to be decoded and plays the multimedia stream to be played.

[0122] In the embodiments of the present application, the playback device decodes the received multimedia stream to be decoded to obtain the multimedia stream to be played, and plays the multimedia stream to be played.

[0123] It should be noted that the multimedia stream to be played is the decoding result of the multimedia stream to be decoded, that is, the multimedia stream to be decoded after decoding. Here, due to different encoding methods, the multimedia stream to be played may be the same as the multimedia stream to be encoded or different from the multimedia stream to be encoded, and the embodiments of the present application do not limit this.

[0124] S507. When the multimedia stream to be played is played to the target event trigger moment, the playback device executes the specified process corresponding to the target event trigger moment.

[0125] In an embodiment of the present application, during the process of a playback device playing a multimedia stream to be played, if the multimedia stream to be played reaches any one of at least one event trigger moment, it indicates that the content currently being played on the playback device is the content of an event (for example, at least one of a picture and audio). At this time, the specified processing corresponding to the event trigger moment is triggered, so that the playback of the content of the event is synchronized with the triggering of the specified processing. Here, the target event trigger moment is any one of the at least one event trigger moment.

[0126] It should be noted that the server device sends the multimedia stream to be decoded to the playback device based on at least one event marking moment and stream time change information, so that the playback device plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and executes the specified processing corresponding to the target event trigger moment when the multimedia stream to be played reaches the target event trigger moment.

[0127] Exemplarily, refer to Figure 5b , Figure 5b FIG. is a schematic flowchart of an exemplary multimedia stream processing method provided by an embodiment of the present application; wherein, the exemplary multimedia stream processing method describes the process of a game server (referred to as a server device) sending a game live video stream (referred to as a multimedia stream to be encoded) to a game client on a terminal (referred to as a client device), and the client device playing the received game live video stream, and presenting a virtual red envelope (referred to as specified processing) when playing the picture of a "pentakill" event (referred to as a target event). The following will be described in conjunction with Figure 5b the steps shown.

[0128] S5b01. The game server marks the "pentakill" event in the game live video stream to obtain at least one "pentakill" event marking moment.

[0129] It should be noted that the game live video stream is the multimedia stream to be encoded in the embodiment of the present application; the process of marking the "pentakill" event is the process of event marking; the at least one "pentakill" event marking moment obtained corresponds to the at least one event marking moment in the embodiment of the present application.

[0130] S5b02. The game server encodes the game live video stream to obtain the game live video stream to be sent.

[0131] It should be noted that the game live video stream to be sent is the multimedia stream to be decoded.

[0132] S5b03. The game server obtains the stream time change information between the game live video stream and the game live video stream to be sent.

[0133] It should be noted that the process of obtaining the stream time change information between the game live video stream to be obtained and the game live video stream to be sent is the process of obtaining the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded in the embodiments of the present application.

[0134] S5b04. The game server sends at least one "pentakill" event marking moment, the stream time change information, and the game live video stream to be sent to the game client.

[0135] It should be noted that here, the example of the game client performing time adjustment processing is used for illustration.

[0136] S5b05. The game client adjusts at least one "pentakill" event marking moment based on the stream time change information to obtain at least one virtual red envelope event trigger moment.

[0137] It should be noted that when the game server sends at least one "pentakill" event marking moment, the stream time change information, and the game live video stream to be sent to the game client, the game client also receives at least one "pentakill" event marking moment, the stream time change information, and the received game live video stream (i.e., the game live video stream to be sent) sent by the game server; at this time, the game client adjusts each "pentakill" event marking moment based on the stream time change information to obtain the corresponding each virtual red envelope event trigger moment. Among them, at least one virtual red envelope event trigger moment is at least one event trigger moment in the embodiments of the present application.

[0138] S5b06. The game client decodes the received game live video stream and plays the game live video.

[0139] It should be noted that the process of the game client playing the game live video is the process of the server device playing the multimedia stream to be played.

[0140] S5b07. When the game live video is played to a virtual red envelope event trigger moment, the game client presents the picture of a corresponding "pentakill" event and presents a virtual red envelope.

[0141] It should be noted that when the game live video is played to a virtual red envelope event trigger moment, that is, when the multimedia stream to be played is played to the target event trigger moment, at this time, the specified processing executed by the game client is to present a virtual red envelope. Here, since the game client device adjusts each "pentakill" event marking moment based on the stream time change information, the playing moment of the picture of the "pentakill" event is consistent with the adjusted "pentakill" event marking moment (i.e., the virtual red envelope event trigger moment), so the synchronous presentation of the picture of the "pentakill" event and the virtual red envelope can be realized, improving the game video playing effect.

[0142] It can be understood that among the at least one event trigger moment obtained by the playback device, each event trigger moment is obtained by adjusting at least one event marking moment based on the stream time change information, and the stream time change information refers to the time change before and after multimedia stream encoding. Therefore, each event trigger moment obtained by the playback device is the occurrence moment of the event in the live stream to be decoded. Thus, during the playback of the multimedia stream to be played obtained by decoding the multimedia stream to be decoded, the playback content is matched with the event trigger moment. Therefore, when the multimedia stream to be played is played to any event trigger moment, it is exactly the moment to play the event content. At this time, triggering the specified processing can improve the accuracy of triggering the specified processing.

[0143] See Figure 6 , Figure 6 which is the flowchart of the multimedia stream processing method provided by the embodiment of the present application Figure 2 ; As Figure 6 shown, in the embodiment of the present application, S504 can be implemented through S5041; that is, the server device sends the multimedia stream to be decoded to the playback device based on at least one event marking moment and the stream time change information, including S5041. The following describes this step.

[0144] S5041. The server device sends at least one event marking moment, the stream time change information, and the multimedia stream to be decoded to the playback device.

[0145] In the embodiment of the present application, when the server device sends at least one event marking moment and the stream time change information to the playback device, the playback device also receives the stream time change information and at least one event marking moment sent by the server device.

[0146] It should be noted that the server device may include a broadcast service device. At this time, the server device can broadcast at least one event marking moment to each playback device through the broadcast service device. And the server device also includes a playback service device. Each playback device requests the corresponding encoding device to encode the multimedia stream to be encoded through the playback service device, and obtains the multimedia stream to be decoded encoded by the corresponding encoding device and the stream time change information of the multimedia stream to be decoded relative to the multimedia stream to be encoded through the playback service device.

[0147] Correspondingly, in the embodiment of the present application, S505 can be implemented through S5051; that is, the playback device obtains at least one event trigger moment corresponding to the multimedia stream to be decoded, including S5051. The following describes this step.

[0148] S5051. The playback device adjusts at least one event marking moment based on the stream time change information to obtain at least one event trigger moment.

[0149] In the embodiment of the present application, the playback device adjusts at least one actual marking moment by using the stream time change information so as to be consistent with the change of the time axis of the multimedia stream to be encoded during the encoding process, so that the time axis of at least one event trigger moment obtained by the adjustment is consistent with the time axis of the multimedia stream to be decoded.

[0150] In the embodiment of the present application, the stream time change information includes the timestamp to be encoded and the encoding timestamp of the target frame information; at this time, S5051 can be implemented through S50511 and S50512 (not shown in the figure); that is, the playback device adjusts at least one event marking moment based on the stream time change information to obtain at least one event trigger moment, including S50511 and S50512, and the following will explain each step separately.

[0151] S50511. The playback device obtains the time change amount between the encoding timestamp and the timestamp to be encoded.

[0152] It should be noted that both the multimedia stream to be encoded and the multimedia stream to be decoded include target frame information. Among them, the timestamp to be encoded is the timestamp corresponding to the target frame information in the multimedia stream to be encoded, the encoding timestamp is the corresponding timestamp of the target frame information in the multimedia stream to be decoded, the target frame information is the frame information including the same multimedia content in the multimedia stream to be encoded and the multimedia stream to be decoded, and the time change amount refers to the relative time difference between the encoding timestamp and the timestamp to be encoded.

[0153] S50512. The playback device fuses the time change amount with each event marking moment to obtain at least one event trigger moment corresponding to at least one event marking moment.

[0154] In the embodiment of the present application, the playback device applies the obtained time change amount to each event marking moment to adaptively adjust each event marking moment based on the change of the timestamp of the multimedia stream to be encoded during the encoding process, so that each event trigger moment obtained by the adjustment corresponds to the event content in the multimedia stream to be decoded. Among them, when the playback device fuses the time change amount and each event marking moment, it can be achieved by addition, or by offsetting each event marking moment based on a time variable. The embodiment of the present application does not limit this.

[0155] It can be understood that the server device records the timestamps of the target frame information in the multimedia stream before and after encoding (wherein, the multimedia stream to be encoded is the multimedia stream before encoding, and the multimedia stream to be decoded is the multimedia stream after encoding), and sends the two recorded timestamps to the playback device, so that the playback device adaptively adjusts the marked time of each received event, and obtains at least one event trigger time consistent with the timeline of the multimedia stream to be decoded; in this way, the consistency between the event content and the event playback time during the actual playback can be improved, and thus, the accuracy of the specified event trigger can be improved.

[0156] See Figure 7 , Figure 7 is a flowchart of the multimedia stream processing method provided by the embodiment of the present application Figure 3 ; as Figure 7 shown, in the embodiment of the present application, S504 and S505 can be implemented through S508 and S509; that is to say, the server device sends the multimedia stream to be decoded to the playback device based on at least one event marked time and the stream time change information, including S508 and S509, and the following will explain each step separately.

[0157] S508. The server device adjusts at least one event marked time based on the stream time change information to obtain at least one event trigger time.

[0158] It should be noted that the server device adjusts at least one event marked time based on the stream time change information to obtain at least one event trigger time, including: when the stream time change information includes the timestamp to be encoded and the encoded timestamp of the target frame information; the server device obtains the time change amount between the encoded timestamp and the timestamp to be encoded, and fuses the time change amount with each event marked time to obtain at least one event trigger time corresponding to at least one event marked time. Here, the corresponding adjustment process is similar to the processing process described in S50511 and S50512, but the execution subject is different.

[0159] S509. The server device sends at least one event trigger time and the multimedia stream to be decoded to the playback device.

[0160] In an embodiment of the present application, after the server device adjusts at least one event marking moment based on the stream time change information and obtains at least one event triggering moment, at this time, the server device sends at least one event triggering moment and the multimedia stream to be decoded to the playback device. Here, when the server device sends at least one event triggering moment to the playback device, the playback device also receives at least one event triggering moment corresponding to the multimedia stream to be decoded sent by the server device, and at least one event triggering moment is obtained by the server device adjusting at least one event marking moment based on the stream time change information.

[0161] It should be noted that when the server device includes an event marking device, a broadcasting device, an encoding device, and a playback service device, the playback service device obtains at least one event marking moment from the event marking device through the broadcasting device, obtains the stream time change information and the multimedia stream to be decoded from the encoding device, and adjusts at least one event marking moment to at least one event triggering moment based on the stream time change information, and then sends the multimedia stream to be decoded and at least one event triggering moment to the playback device.

[0162] It can be understood that by completing the adjustment of at least one event marking moment in the server device, the resource consumption of the playback device for adjustment is reduced; and the performance of the server device is better than that of the playback device. By implementing the adjustment process through the server device, the adjustment efficiency can be improved, and thus the processing efficiency of the multimedia stream can be improved.

[0163] In an embodiment of the present application, the specified processing in S507 includes at least one of presenting virtual resource information, presenting specified special effects, inserting multimedia information, presenting Q&A information, and presenting recommendation information.

[0164] It should be noted that the virtual resource information includes at least one of a virtual resource package and a virtual gift. When the specified processing includes presenting virtual resource information, the playback device executes the specified processing corresponding to the target event triggering moment, including: when the target event triggering moment refers to the occurrence moment of a virtual scene event, the playback device presents the virtual resource information; and after the playback device presents the virtual resource package, the multimedia stream processing method further includes: the playback device presents the obtained target virtual resource in response to a trigger operation on the virtual resource package, such as a grabbed virtual red envelope, a grabbed virtual gift, and so on.

[0165] The specified special effects played by the playback device can be special effects associated with at least one event. For example, when the event is a goal event, a special effect of applause is presented; when the event is a consecutive kill event, a special effect of scattering virtual coins is presented; when the event is a game winning streak event, an animation special effect of congratulating on the winning streak is presented, and so on. In addition, when the multimedia stream to be played includes subtitle information, when playing to the specified event time, the subtitle information associated with the event content can be marked and displayed.

[0166] The multimedia information inserted by the playback device is different from the multimedia stream to be played, such as advertisements, news, announcement information, and so on.

[0167] After the playback device presents interactive information such as question-and-answer information and recommendation information, the multimedia stream processing method further includes: the playback device responds to an interactive operation for the interactive information and executes a process corresponding to the interactive operation, such as application switching, switching of playback information, and so on.

[0168] In the embodiments of the present application, the event marking in S501 is implemented by at least one of a marking operation, a neural network model, and marking information; wherein, the marking operation refers to an operation for determining the event marking moment; the neural network model is used to determine the event marking moment by obtaining the content of the multimedia stream to be played, and the neural network model is trained based on multimedia stream samples and annotated event moments; the marking information is used to represent the event marking moment, and the marking information can be information carried in the multimedia stream to be encoded.

[0169] In the embodiments of the present application, S503 can be implemented through S5031 to S5034 (not shown in the figure); that is, the server device obtains the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded, including S5031 to S5034, and the following will explain each step separately.

[0170] S5031: The server device obtains the time stamp to be encoded corresponding to the target frame information of the multimedia stream to be encoded.

[0171] In the embodiments of the present application, after the encoding device in the server device obtains the multimedia stream to be encoded distributed by the multimedia stream distribution device, it records the time stamp corresponding to the target frame information of the multimedia stream to be encoded, and thus obtains the time stamp to be encoded.

[0172] S5032: The server device obtains the encoding method of the multimedia stream to be decoded.

[0173] It should be noted that the encoding method includes any one of a first specified encoding method, a second specified encoding method, and a third specified encoding method. Among them, the first specified encoding method refers to a method that does not change the playback timestamp during the encoding process. The second specified encoding method refers to a method that determines the playback timestamp based on synchronous time information (for example, the time information provided by a synchronous clock) during the encoding process. The third specified encoding method refers to a method that determines the playback timestamp based on specified time information (for example, 0 milliseconds) during the encoding process.

[0174] S5033. The server device determines the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded based on the encoding method.

[0175] In the embodiment of the present application, when the encoding method is the first specified encoding method, the timestamp to be encoded is determined as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded. When the encoding method is the second specified encoding method, the synchronous timestamp is obtained and determined as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded. When the encoding method is the third specified encoding method, the specified timestamp is determined as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded.

[0176] S5034. The server device determines the timestamp to be encoded and the encoding timestamp as the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded.

[0177] It should be noted that the server device sends the timestamp to be encoded and the encoding timestamp as the stream time change information to the playback device, so that the playback device adjusts each event marking moment based on the timestamp to be encoded and the encoding timestamp in the stream time change information. Alternatively, the server device takes the timestamp to be encoded and the encoding timestamp as the stream time change information, and adjusts each event marking moment based on the timestamp to be encoded and the encoding timestamp in the stream time change information, so as to send each event trigger moment to the playback device.

[0178] In the embodiment of the present application, the server device further includes a timestamp synchronization device, which is used to forward the stream time change information obtained by the encoding device to the playback service device, and is also used to provide a synchronous timestamp to the encoding device, so that the encoding device encodes the multimedia stream to be encoded based on the synchronous timestamp.

[0179] Next, an exemplary application of the embodiment of the present application in an actual application scenario will be described. This exemplary application describes that during a game live broadcast, when a terminal (referred to as a playback device) is playing an audio and video (referred to as a multimedia stream to be played), when playing the picture of a wonderful event (that is, when playing to the target event trigger moment), the presentation of a virtual resource package (referred to as a specified process) is accurately triggered.

[0180] See Figure 8 , Figure 8 which is an exemplary schematic diagram of an event screen provided by an embodiment of the present application; as Figure 8 shown, in the virtual scene 8-1, the virtual object 8-11 killed 5 virtual objects of the game team within the specified time, triggering the "Penta Kill" event.

[0181] Based on Figure 8 see Figure 9 , Figure 9 which is an exemplary schematic diagram of the change of the timestamp provided by an embodiment of the present application; as Figure 9 shown, it describes the process of changing the timestamp corresponding to the event in the multimedia stream due to encoding: in the live stream 9-1 before encoding, Figure 8 the timestamp 9-11 (referred to as the event marking moment) of the "Penta Kill" event in Figure 9 corresponds to PTS_0 in Figure 8 ; while in the live stream 9-2 (referred to as the event trigger moment) obtained by encoding the live stream 9-1, Figure 9 the timestamp 9-21 of the "Penta Kill" event in

[0182] See Figure 10 , Figure 10 which is an exemplary schematic diagram of the live stream system architecture provided by an embodiment of the present application; as Figure 10 shown, when the live stream 10-1 (live stream ID_0, referred to as the multimedia stream to be encoded) arrives at the stream distribution server 10-2 (referred to as the multimedia stream distribution device), the stream distribution server 10-2 distributes the live stream 10-1 in multiple paths. One path of distribution is to send the live stream 10-1 to the event marking server 10-3 (referred to as the event marking device), and the other path of distribution is to send the live stream 10-1 to n (n is a positive integer greater than 1) encoding servers 10-4 (encoding servers 10-41 to encoding servers 10-4n, referred to as the encoding devices).

[0183] The event marking server 10-3 marks events for the live stream 10-1 to obtain each marked moment on the time axis of the live stream 10-1 (for example, the "penta kill" moment, or other exciting moments, which are called event marked moments); among them, event marking can be achieved by receiving the operations of users (called marking operations), can also be achieved by artificial intelligence (such as a neural network model), and can also be directly obtained based on the indication information sent by the game server (such as the information sent synchronously with the live stream 10-1, which is called marking information). Here, the event marking server 10-3 sends each obtained marked moment to the broadcast server 10-5 (called the broadcast device).

[0184] The encoding server 10-4 is used to send the live stream that best matches the network conditions and playback capabilities to the terminal when there are differences in the network conditions and playback capabilities of the user's terminal, so as to improve playback effects such as playback smoothness; here, each encoding server i in the encoding server 10-4 (i is a positive integer variable greater than or equal to 1 and less than or equal to n, that is, any one of the encoding servers 10-41 to 10-4n) is used to encode the live stream 10-1 into multiple different live streams (live stream ID_1 to live stream ID_n); and each encoding server i in the encoding server 10-4 performs encoding based on at least one of different resolutions, encoding formats (such as H.265 encoding format, H.264 encoding format, etc.) and bitrates. In addition, each encoding server i can also correspond to a game live application.

[0185] Exemplarily, refer to Figure 11 , Figure 11 is a schematic diagram of the encoding result of an exemplary live stream provided by an embodiment of the present application; as Figure 11 shown, Figure 10 the 8 different live streams 11-1 output after the live stream 10-1 in

[0186] Based on Figure 11 , refer to Figure 12 , Figure 12 is a schematic diagram of an exemplary application scenario for obtaining different live streams provided by an embodiment of the present application; as Figure 12As shown, based on the area 12-11 on page 12-1, it can be determined that the live stream played on page 12-1 is a live stream with a definition of "HD 540P", and it is also possible to select a live stream with any one of the definitions of "Blu-ray 1080P", "Ultra HD 720P", and "SD 360P" for playback. Among them, the terminal corresponding to page 12-1 supports live streams in the H.264 encoding format. Thus, the "SD 360P" displayed on page 12-1 corresponds to Figure 12 a live stream with a definition of 360P in the H.264 encoding format in Figure 12 ; the "HD 540P" displayed on page 12-1 corresponds to Figure 12 a live stream with a definition of 540P in the H.264 encoding format in Figure 12 ; the "Ultra HD 720P" displayed on page 12-1 corresponds to

[0187] a live stream with a definition of 720P in the H.264 encoding format in

[0187] ; the "Blu-ray 1080P" displayed on page 12-1 corresponds to Figure 12 a live stream with a definition of 1080P in the H.264 encoding format in

[0187] It should be noted that each encoding server i encodes the live stream ID_0 to obtain the live stream ID_i. Here, if the encoding server i can pass through the live stream ID_0 without changing the presentation time stamp (PTS) corresponding to the frame information of the live stream ID_0 (corresponding to the first specified encoding method), then the time axis of the live stream ID_i is the same as that of the live stream ID_0. Thus, the presentation time stamp PTS_is (referred to as the encoding time stamp) corresponding to the start frame information of the live stream ID_i (which is at least one of the first frame picture and the first frame audio of the live stream ID_0 reaching the encoding server i, referred to as the target frame information) is the same as the presentation time stamp PTS_i0 (referred to as the time stamp to be encoded) corresponding to the start frame information of the live stream ID_0, and at this time, the marking time remains unchanged. In addition, the encoding server i can also use the synchronization time (referred to as the synchronization time stamp) provided by the time stamp synchronization server 10-6 (referred to as the time stamp synchronization device) as the presentation time stamp PTS_is corresponding to the start frame information of the live stream ID_i (corresponding to the second specified encoding method); and the encoding server i can also set it by itself (corresponding to the specified time stamp, for example, 0) as the presentation time stamp PTS_is corresponding to the start frame information of the live stream ID_i (corresponding to the third specified encoding method). Thus, in the embodiment of the present application, when the encoding server i receives the live stream ID_0 distributed by the stream distribution server 10-2, it records the presentation time stamp PTS_i0 corresponding to the start frame information of the live stream ID_0, and forms a time stamp pair <PTS_i0, PTS_is> (referred to as the stream time change information) by the presentation time stamp PTS_i0 and the presentation time stamp PTS_is corresponding to the start frame information of the live stream ID_i, and sends the time stamp pair <PTS_i0, PTS_is> to the time stamp synchronization server 10-6. Here, i is greater than or equal to and less than or equal to n, and the corresponding outputs of the encoding servers 10-41 to 10-4n are (ID_1, PTS_10, PTS_1s), (ID_2, PTS_20, PTS_2s),..., (ID_n, PTS_n0, PTS_ns); and any one of the live streams ID_1 to ID_n can be distributed to the terminal 10-7 through CDN distribution or peer-to-peer (P2P) services.

[0188] The broadcast server 10-5 broadcasts each marked moment to the terminal 10-7; the playback server 10-8 (referred to as the playback service device) obtains the timestamp pair <PTS_i0, PTS_is> from the timestamp synchronization server 10-6 and sends it to the terminal 10-7; the playback server 10-8 obtains the encoded live stream address from the encoding server 10-4 and sends it to the terminal 10-7. Thus, the terminal 10-7 can adjust each marked moment based on the timestamp pair, and obtain the live stream ID_i from the encoding server 10-4 based on the live stream address. For example, if the marked moment is PTS_ID0_t, then the marked moment at the moment PTS_IDi_t of the playback stream ID_i = PTS_id0_t + (PTS_is - PTS_i0). When the terminal plays to the moment PTS_IDi_t, the corresponding processing logic is activated, thereby triggering the specified processing.

[0189] Regarding Figure 10 , it should also be noted that the processing process of the live stream includes an information interaction stream (i.e., the interaction processing stream of the timestamp pair), a live stream processing stream, and an event stream (i.e., the processing stream of the marked moment). Also, the timestamps PTS_x0 and PTS_y0 of two frames of information in the live stream ID_0, and the timestamps PTS_xi and PTS_yi of two frames of information in the live stream ID_i, have the following corresponding relationship: PTS_yi - PTS_xi = PTS_y0 - PTS_x0; where the live content corresponding to the timestamp PTS_x0 is the same as the live content corresponding to the timestamp PTS_xi, and the live content corresponding to the timestamp PTS_y0 is the same as the live content corresponding to the timestamp PTS_yi.

[0190] Next, Figure 10 the interaction between the stream distribution server 10-2, the event marking server 10-3, the encoding server 10-4, the broadcast server 10-5, the timestamp synchronization server 10-6, the terminal 10-7, and the playback server 10-8 in Figure 13 will be described in detail. Refer to Figure 13 , Figure 13 which is an exemplary live stream interaction schematic diagram provided by an embodiment of the present application; as

[0191] shown, this exemplary live stream interaction schematic diagram includes S1301 to S1309, and each step will be described separately below.

[0192] Here, the game live stream is the Figure 10 live stream ID_0 in

[0193] S1302. The event marking server 10-3 sends highlight moments to the broadcast server 10-5.

[0194] It should be noted that the event marking server 10-3 marks events for the live stream ID_0, obtains each highlight moment (i.e., each marked moment in Figure 10 ), and sends each highlight moment to the broadcast server 10-5.

[0195] S1303. The terminal 10-7 sends a live stream request (referred to as a playback request) to the playback server 10-8.

[0196] S1304. The stream distribution server 10-2 distributes the game live stream to the encoding server 10-4.

[0197] It should be noted that in response to the live stream request, the playback server 10-8 enables the encoding server i in the encoding server 10-4 to access, thereby triggering the execution of S1304.

[0198] S1305. The encoding server 10-4 sends a timestamp pair to the timestamp synchronization server 10-6.

[0199] It should be noted that the encoding server 10-4 encodes the game live stream (live stream ID_0) to obtain the encoded game live stream (live stream ID_i); and sends the timestamp pair <PTS_i0, PTS_is> composed of the timestamp corresponding to the start frame information of the live stream ID_0 when responding to the live stream request and the timestamp corresponding to the start frame information of the live stream ID_i to the timestamp synchronization server 10-6.

[0200] S1306. The timestamp synchronization server 10-6 sends a timestamp pair to the terminal 10-7.

[0201] Here, the timestamp synchronization server 10-6 can send the timestamp pair to the terminal 10-7 through the playback server 10-8.

[0202] S1307. The broadcast server 10-5 broadcasts the highlight moments to the terminal 10-7.

[0203] S1308. The encoding server 10-4 sends the encoded game live stream to the terminal 10-7.

[0204] Here, it can be the live stream address sent to the terminal 10-7 in response to the live stream request, so that the encoding server 10-4 sends the encoded game live stream (i.e., the live stream ID_i in Figure 10 ) matching the live stream address to the terminal 10-7.

[0205] S1309. The terminal 10-7 plays the encoded game live stream, and when playing the picture of the exciting moment (such as the video picture of "pentakill" appears), presents virtual red envelopes or virtual gifts.

[0206] It can be understood that in the embodiments of the present application, by recording the change of the time stamp during the encoding process and adjusting the marked exciting moments based on the change, an accurate correspondence between events and time is established in the encoded live stream, which can ensure that the corresponding response actions of the events are performed only when a specific picture appears; even in the case of uncontrollable network delay, the consistency of the response logic of each terminal can be ensured, so that the matching degree between the specified processing and the playing content is relatively high. In addition, in the processing of the game live stream in the embodiments of the present application, no additional privatization processing is performed on the game live stream; and, in the processing of the game live stream in the embodiments of the present application, the processing of events and the processing of the game live stream are decoupled and do not affect each other; thus, the generality is relatively high.

[0207] Next, continue to describe the exemplary structure of the first multimedia stream processing device 455 provided by the embodiments of the present application as a software module. In some embodiments, as Figure 3 shown, the software module in the first multimedia stream processing device 455 stored in the first memory 450 may include:

[0208] An information acquisition module 4551, configured to receive the multimedia stream to be decoded sent by the server device, and acquire at least one event trigger moment corresponding to the multimedia stream to be decoded, where at least one of the event trigger moments is obtained by adjusting at least one event marking moment based on the stream time change information, and the stream time change information refers to the time change before and after the encoding of the multimedia stream, and each event marking moment refers to the event moment in the multimedia stream before encoding;

[0209] An information playback module 4552, configured to play the multimedia stream to be played by decoding the multimedia stream to be decoded;

[0210] A processing trigger module 4553, configured to perform a specified process corresponding to the target event trigger moment when the multimedia stream to be played plays to the target event trigger moment, where the target event trigger moment is any one of at least one of the event trigger moments.

[0211] In the embodiments of the present application, the information playback module 4552 is further configured to receive the stream time change information and at least one of the event marking moments sent by the server device; adjust at least one of the event marking moments based on the stream time change information to obtain at least one of the event trigger moments.

[0212] In the embodiment of the present application, the stream time change information includes the timestamp to be encoded and the encoded timestamp of the target frame information; the information playback module 4552 is further configured to obtain the time change amount between the encoded timestamp and the timestamp to be encoded; fuse the time change amount with each of the event marking times to obtain at least one event trigger time corresponding to at least one of the event marking times.

[0213] In the embodiment of the present application, the information playback module 4552 is further configured to receive at least one of the event trigger times corresponding to the multimedia stream to be decoded sent by the server device, where at least one of the event trigger times is obtained by the server device adjusting at least one of the event marking times based on the stream time change information.

[0214] In the embodiment of the present application, the specified processing includes at least one of presenting virtual resource information, presenting specified special effects, inserting multimedia information, presenting Q&A information, and presenting recommendation information.

[0215] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A first processor of a computer device (referred to as a playback device) reads the computer instructions from the computer-readable storage medium, and the first processor executes the computer instructions, so that the computer device executes the multimedia stream processing method applied to the playback device in the embodiment of the present application.

[0216] Next, the exemplary structure of the second multimedia stream processing device 255 provided in the embodiment of the present application implemented as a software module will be further described. In some embodiments, as Figure 4 shown, the software module stored in the second multimedia stream processing device 255 in the second memory 450 may include:

[0217] An event marking module 2551, configured to perform event marking on the multimedia stream to be encoded to obtain at least one event marking time;

[0218] An information encoding module 2552, configured to encode the multimedia stream to be encoded to obtain a multimedia stream to be decoded;

[0219] A time acquisition module 2553, configured to acquire the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded;

[0220] An information sending module 2554, configured to send the multimedia stream to be decoded to a playback device based on at least one of the event marking moments and the stream time change information, so that the playback device plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and perform a specified process corresponding to the target event triggering moment when the multimedia stream to be played is played to the target event triggering moment, where the target event triggering moment is any one of at least one event triggering moment, and at least one of the event triggering moments is obtained by adjusting at least one of the event marking moments based on the stream time change information.

[0221] In an embodiment of the present application, the information sending module 2554 is further configured to send at least one of the event marking moments, the stream time change information, and the multimedia stream to be decoded to the playback device, so that the playback device adjusts at least one of the event marking moments based on the stream time change information to obtain at least one of the event triggering moments.

[0222] In an embodiment of the present application, the information sending module 2554 is further configured to adjust at least one of the event marking moments based on the stream time change information to obtain at least one of the event triggering moments; and send at least one of the event triggering moments and the multimedia stream to be decoded to the playback device.

[0223] In an embodiment of the present application, the event marking is implemented by at least one of a marking operation, a neural network model, and marking information, where the marking operation refers to an operation for determining the event marking moment, the neural network model is used to determine the event marking moment by obtaining the content of the multimedia stream to be played, and the neural network model is trained based on a multimedia stream sample and an annotated event moment, and the marking information is used to represent the event marking moment.

[0224] In an embodiment of the present application, the time acquisition module 2553 is further configured to acquire a timestamp to be encoded corresponding to the target frame information of the multimedia stream to be encoded; acquire the encoding method of the multimedia stream to be decoded; determine the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded based on the encoding method; and determine the timestamp to be encoded and the encoding timestamp as the stream time change information between the multimedia stream to be encoded and the multimedia stream to be decoded.

[0225] In an embodiment of the present application, the time acquisition module 2553 is further configured to, when the encoding method is the first specified encoding method, determine the timestamp to be encoded as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the first specified encoding method refers to a method that does not change the playback timestamp during the encoding process; when the encoding method is the second specified encoding method, obtain a synchronization timestamp and determine the synchronization timestamp as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the second specified encoding method refers to a method that determines the playback timestamp based on synchronized time information during the encoding process; when the encoding method is the third specified encoding method, determine the specified timestamp as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the third specified encoding method refers to a method that determines the playback timestamp based on specified time information during the encoding process.

[0226] An embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A second processor of a computer device (referred to as a server device) reads the computer instructions from the computer-readable storage medium, and the second processor executes the computer instructions, so that the computer device executes the multimedia stream processing method applied to the server device in the embodiments of the present application.

[0227] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, where the executable instructions, when executed by a first processor, cause the first processor to execute the multimedia stream processing method applied to a playback device in the embodiments of the present application; or, when the executable instructions are executed by a second processor, cause the second processor to execute the multimedia stream processing method applied to a server device in the embodiments of the present application; for example, as Figure 5a shown in the multimedia stream processing method.

[0228] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0229] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0230] As an example, the executable instructions may or may not correspond to files in a file system, and may be stored as part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program under discussion, or stored in multiple cooperating files (e.g., files that store one or more modules, subroutines, or code portions).

[0231] As an example, the executable instructions may be deployed to execute on one computer device (in this case, this one computer device is both the playback device and the server device), or on multiple computer devices located at one location (in this case, the multiple computer devices located at one location are both the playback device and the server device), or, execute on multiple computer devices distributed at multiple locations and interconnected by a communication network (in this case, the multiple computer devices distributed at multiple locations and interconnected by a communication network are both the playback device and the server device).

[0232] In summary, in the embodiments of the present application, since among the at least one event trigger moment obtained by the playback device, each event trigger moment is obtained by adjusting at least one event marking moment based on the stream time change information, and the stream time change information refers to the time change before and after the live stream is encoded; therefore, each event trigger moment obtained by the playback device is the occurrence moment of the event in the live stream to be decoded. Thus, in the multimedia stream to be played obtained by decoding the multimedia stream to be decoded, the event content matches the event trigger moment; thereby, when the multimedia stream to be played is played to any event trigger moment, the specified process corresponding to the event trigger moment can be accurately triggered, so the accuracy of triggering the specified process can be improved; in addition, the multimedia stream triggering method provided by the embodiments of the present application has high generality.

[0233] The above description is only for the embodiments of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are all included in the protection scope of the present application.

Claims

1. A method for processing multimedia streams, characterized in that, applied to a playback device, the method includes: Receiving a multimedia stream to be decoded sent by a server device, and obtaining at least one event trigger moment corresponding to the multimedia stream to be decoded, wherein at least one of the event trigger moments is obtained by adjusting at least one event marking moment based on stream time change information, the stream time change information refers to the time change before and after multimedia stream encoding, each event marking moment refers to an event moment in the multimedia stream before encoding, the multimedia stream to be decoded refers to the multimedia stream after encoding, the event trigger moment refers to the moment in the multimedia stream to be decoded corresponding to the event marking moment, and the event trigger moment is the occurrence moment of the event in the multimedia stream to be decoded; Decoding the multimedia stream to be decoded and playing the multimedia stream to be played; When the multimedia stream to be played plays to the target event trigger moment, playing the event content corresponding to the target event trigger moment and performing the specified processing corresponding to the target event trigger moment, wherein the target event trigger moment is any one of at least one of the event trigger moments; At least one of the event trigger moments is obtained by fusing at least one of the event marking moments and the time change amount corresponding to the stream time change information, and at least one of the event trigger moments or the stream time change information and at least one of the event marking moments are obtained by broadcasting of the server device; The server device is configured to perform event marking on the multimedia stream before encoding through at least one of a marking operation, a neural network model, and marking information to obtain at least one of the event marking moments, the marking operation refers to an operation for determining the event marking moment, the neural network model is used to determine the event marking moment by obtaining the content of the multimedia stream before encoding, and the neural network model is trained based on multimedia stream samples and labeled event moments, and the marking information is used to represent the event marking moment; The server device is further configured to obtain the timestamp to be encoded corresponding to the target frame information of the multimedia stream before encoding, obtain the encoding method of the multimedia stream to be decoded, determine the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded based on the encoding method, and determine the stream time change information based on the timestamp to be encoded and the encoding timestamp.

2. The method according to claim 1, characterized in that, The obtaining at least one event trigger moment corresponding to the multimedia stream to be decoded includes: Receiving the stream time change information and at least one of the event marking moments sent by the server device; Adjusting at least one of the event marking moments based on the stream time change information to obtain at least one of the event trigger moments.

3. The method according to claim 2, characterized in that, The stream time change information includes the timestamp to be encoded and the encoding timestamp of the target frame information; Adjusting at least one of the event marking moments based on the flow time change information to obtain at least one of the event triggering moments includes: Obtaining the time change amount between the encoding timestamp and the timestamp to be encoded; Fusing the time change amount with each of the event marking moments to obtain at least one of the event triggering moments corresponding to at least one of the event marking moments.

4. The method according to claim 2, wherein, the flow time change information includes the time change amount between the encoding timestamp and the timestamp to be encoded; Adjusting at least one of the event marking moments based on the flow time change information to obtain at least one of the event triggering moments includes: Fusing the time change amount with each of the event marking moments to obtain at least one of the event triggering moments corresponding to at least one of the event marking moments.

5. The method according to claim 1, wherein, Obtaining at least one event triggering moment corresponding to the multimedia stream to be decoded includes: Receiving at least one of the event triggering moments corresponding to the multimedia stream to be decoded sent by the server device, wherein at least one of the event triggering moments is obtained by the server device adjusting at least one of the event marking moments based on the flow time change information.

6. The method according to any one of claims 1 to 4, wherein, The specified processing includes at least one of presenting virtual resource information, presenting specified special effects, inserting multimedia information, presenting Q&A information, and presenting recommendation information.

7. A multimedia stream processing method, wherein, Applied to a server device, the method includes: Performing event marking on the multimedia stream to be encoded through at least one of a marking operation, a neural network model, and marking information to obtain at least one event marking moment, where the marking operation refers to an operation for determining the event marking moment, the neural network model is used to determine the event marking moment by obtaining the content of the multimedia stream to be encoded, and the neural network model is trained based on multimedia stream samples and annotated event moments, and the marking information is used to represent the event marking moment; Encoding the multimedia stream to be encoded to obtain a multimedia stream to be decoded, where the multimedia stream to be decoded refers to the encoded multimedia stream; Obtaining the timestamp to be encoded corresponding to the target frame information of the multimedia stream to be encoded; obtaining the encoding method of the multimedia stream to be decoded; determining the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded based on the encoding method; determining the flow time change information based on the timestamp to be encoded and the encoding timestamp; Based on at least one of the event marking moments and the stream time change information, send the multimedia stream to be decoded to the playback device, so that the playback device plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and plays the event content corresponding to the target event trigger moment when the multimedia stream to be played reaches the target event trigger moment, and executes the specified processing corresponding to the target event trigger moment. The target event trigger moment is any one of at least one event trigger moment. At least one of the event trigger moments is obtained by adjusting at least one of the event marking moments based on the stream time change information. The event trigger moment refers to the moment in the multimedia stream to be decoded corresponding to the event marking moment, and the event trigger moment is the occurrence moment of the event in the multimedia stream to be decoded. The playback device obtains the stream time change information and at least one of the event marking moments or at least one of the event trigger moments through the broadcast of the server device; At least one of the event trigger moments is obtained by fusing at least one of the event marking moments and the time change amount corresponding to the stream time change information.

8. The method according to claim 7, wherein, the step of sending the multimedia stream to be decoded to the playback device based on at least one of the event marking moments and the stream time change information includes: sending at least one of the event marking moments, the stream time change information, and the multimedia stream to be decoded to the playback device, so that the playback device adjusts at least one of the event marking moments based on the stream time change information to obtain at least one of the event trigger moments.

9. The method according to claim 7, wherein, the step of sending the multimedia stream to be decoded to the playback device based on at least one of the event marking moments and the stream time change information includes: adjusting at least one of the event marking moments based on the stream time change information to obtain at least one of the event trigger moments; sending at least one of the event trigger moments and the multimedia stream to be decoded to the playback device.

10. The method according to any one of claims 7 to 9, wherein, the step of determining the stream time change information based on the timestamp to be encoded and the encoded timestamp includes: determining the timestamp to be encoded and the encoded timestamp as the stream time change information; or, determining the time change amount between the timestamp to be encoded and the encoded timestamp as the stream time change information.

11. The method according to claim 7, wherein, the step of determining the encoded timestamp corresponding to the target frame information in the multimedia stream to be decoded based on the encoding method includes: when the encoding method is the first specified encoding method, determining the timestamp to be encoded as the encoded timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the first specified encoding method refers to the method that does not change the playback timestamp during the encoding process; When the encoding method is the second specified encoding method, obtain a synchronization timestamp, and determine the synchronization timestamp as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the second specified encoding method refers to a method of determining a presentation timestamp based on synchronized time information during the encoding process; When the encoding method is the third specified encoding method, determine a specified timestamp as the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded, where the third specified encoding method refers to a method of determining a presentation timestamp based on specified time information during the encoding process.

12. A playback device, characterized in that, the playback device includes: a first memory for storing executable instructions; a first processor for implementing the multimedia stream processing method according to any one of claims 1 to 6 when executing the executable instructions stored in the first memory.

13. A server device, characterized in that, the server device includes: a second memory for storing executable instructions; a second processor for implementing the multimedia stream processing method according to any one of claims 7 to 11 when executing the executable instructions stored in the second memory.

14. A computer-readable storage medium storing executable instructions, characterized in that, the executable instructions are used to implement the multimedia stream processing method according to any one of claims 1 to 6 when executed by a first processor; or the executable instructions are used to implement the multimedia stream processing method according to any one of claims 7 to 11 when executed by a second processor.

15. A computer program product including a computer program or instructions, characterized in that, the computer program or instructions are used to implement the multimedia stream processing method according to any one of claims 1 to 6 when executed by a first processor; or the computer program or instructions are used to implement the multimedia stream processing method according to any one of claims 7 to 11 when executed by a second processor.

16. A first multimedia stream processing apparatus, characterized in that, comprising: an information acquisition module for receiving a multimedia stream to be decoded sent by a server device, and acquiring at least one event trigger moment corresponding to the multimedia stream to be decoded, where at least one of the event trigger moments is obtained by adjusting at least one event marker moment based on stream time change information, the stream time change information refers to the time change before and after multimedia stream encoding, each event marker moment refers to an event moment in the multimedia stream before encoding, the multimedia stream to be decoded refers to the multimedia stream after encoding, the event trigger moment refers to the moment corresponding to the event marker moment in the multimedia stream to be decoded, and the event trigger moment is the occurrence moment of an event in the multimedia stream to be decoded; an information playback module for playing a multimedia stream to be played by decoding the multimedia stream to be decoded; A processing trigger module, configured to play event content corresponding to the target event trigger moment and execute specified processing corresponding to the target event trigger moment when the multimedia stream to be played reaches the target event trigger moment, where the target event trigger moment is any one of at least one of the event trigger moments; At least one of the event trigger moments is obtained by fusing at least one of the event marking moments and the time change amount corresponding to the stream time change information, and at least one of the event trigger moments or the stream time change information and at least one of the event marking moments are obtained through the broadcast of the server device; The server device is configured to perform event marking on the multimedia stream before encoding through at least one of a marking operation, a neural network model, and marking information to obtain at least one of the event marking moments, where the marking operation refers to an operation for determining the event marking moment, the neural network model is configured to determine the event marking moment by obtaining the content of the multimedia stream before encoding, and the neural network model is trained based on multimedia stream samples and annotated event moments, and the marking information is used to represent the event marking moment; The server device is further configured to obtain the timestamp to be encoded corresponding to the target frame information of the multimedia stream before encoding, obtain the encoding method of the multimedia stream to be decoded, determine the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded based on the encoding method, and determine the stream time change information based on the timestamp to be encoded and the encoding timestamp.

17. A second multimedia stream processing device Characterized in that It includes: An event marking module, configured to perform event marking on the multimedia stream to be encoded through at least one of a marking operation, a neural network model, and marking information to obtain at least one event marking moment, where the marking operation refers to an operation for determining the event marking moment, the neural network model is configured to determine the event marking moment by obtaining the content of the multimedia stream to be encoded, and the neural network model is trained based on multimedia stream samples and annotated event moments, and the marking information is used to represent the event marking moment; An information encoding module, configured to encode the multimedia stream to be encoded to obtain a multimedia stream to be decoded, where the multimedia stream to be decoded refers to the encoded multimedia stream; A time acquisition module, configured to obtain the timestamp to be encoded corresponding to the target frame information of the multimedia stream to be encoded; Obtain the encoding method of the multimedia stream to be decoded; Based on the encoding method, determine the encoding timestamp corresponding to the target frame information in the multimedia stream to be decoded; Based on the timestamp to be encoded and the encoding timestamp, determine the stream time change information; An information sending module, configured to send the multimedia stream to be decoded to a playback device based on at least one of the event marking times and the stream time change information, so that the playback device plays the multimedia stream to be played by decoding the multimedia stream to be decoded, and plays event content corresponding to a target event triggering time when the multimedia stream to be played is played to the target event triggering time, and executes specified processing corresponding to the target event triggering time. The target event triggering time is any one of at least one event triggering time, and at least one of the event triggering times is obtained by adjusting at least one of the event marking times based on the stream time change information. The event triggering time refers to the time corresponding to the event marking time in the multimedia stream to be decoded, and the event triggering time is the occurrence time of an event in the multimedia stream to be decoded. The playback device obtains the stream time change information and at least one of the event marking times or at least one of the event triggering times through the broadcast of a server device; at least one of the event triggering times is obtained by fusing at least one of the event marking times and the time change amount corresponding to the stream time change information.

Images