Multimedia stream switching method, device and system

Abstract

The present application provides a multimedia stream switching method, device and system, wherein the multimedia stream switching method includes: receiving a first multimedia stream sent by a first user terminal, caching the first multimedia stream based on a set delay time, and pushing the cached first multimedia stream based on the current moment; during the push process of the first multimedia stream, if an interaction request is received between the first user terminal and the second user terminal, obtaining the second multimedia stream of the second user terminal; mixing the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein the mixing time for the mixing process is less than or equal to the set delay time; switching the first multimedia stream to the mixed multimedia stream for pushing. The problem of stream interruption and waste of resources is avoided, and it can ensure that the multimedia content corresponding to the output mixed multimedia stream is completely aligned and smooth without any jamming, thereby improving the smoothness of multimedia switching.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimedia stream switching method, a multimedia stream switching device, a multimedia stream switching system, a computing device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology, live broadcast technology has become more and more mature. In order to enhance the interactivity of live broadcast, live broadcast connection came into being.

[0003] When the host initiates a live broadcast, the server needs to mix the live streams of the host and the users participating in the live broadcast, and then push the mixed live stream to the audience clients watching the live broadcast of each live broadcast client. In this way, the live broadcast content of the users' live broadcast interaction can be displayed on the audience clients.

[0004] However, since the server needs to mix the live streams of the anchor and the user who is connected to the microphone, it takes a certain amount of time. Directly switching the multimedia stream corresponding to the anchor to the live stream mixed by the server will cause the stream to be disconnected or the picture to freeze, thereby reducing the experience when connected to the microphone. Therefore, a solution with smoother multimedia stream switching is needed. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a multimedia stream switching method. The present application also relates to a multimedia stream switching device, a multimedia stream switching system, a computing device, and a computer-readable storage medium to solve the technical defect of multimedia stream switching being stuck in the prior art.

[0006] According to a first aspect of an embodiment of the present application, a multimedia stream switching method is provided, including:

[0007] receiving a first multimedia stream sent by a first user terminal, caching the first multimedia stream based on a set delay time, and pushing the cached first multimedia stream based on a current moment;

[0008] During the push process of the first multimedia stream, if an interaction request between the first user terminal and the second user terminal is received, obtaining a second multimedia stream of the second user terminal;

[0009] Performing a mixed stream processing on the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein a mixed stream time of the mixed stream processing is less than or equal to the set delay time;

[0010] The first multimedia stream is switched to the mixed multimedia stream for pushing.

[0011] Optionally, the pushing the cached first multimedia stream based on the current moment includes:

[0012] Subtract the current time from the set delay time to obtain the time to be pushed;

[0013] Push the first multimedia stream corresponding to the time to be pushed.

[0014] Optionally, the first multimedia stream consists of a plurality of first multimedia frames carrying timestamps, and the second multimedia stream consists of a plurality of second multimedia frames carrying timestamps;

[0015] The mixing the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream includes:

[0016] The first multimedia frame and the second multimedia frame carrying the same timestamp are mixed to obtain a mixed multimedia frame carrying the timestamp, wherein a plurality of mixed multimedia frames constitute a mixed multimedia stream.

[0017] Optionally, the first multimedia stream is composed of a plurality of first multimedia frames carrying timestamps, and the mixed multimedia stream is composed of a plurality of mixed multimedia frames carrying timestamps;

[0018] The switching the first multimedia stream to the mixed multimedia stream for pushing includes:

[0019] Compare the cached first multimedia frame to be pushed with the obtained mixed multimedia frame to check whether there is a target mixed multimedia frame with the same timestamp as that carried by the first multimedia frame to be pushed;

[0020] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0021] Optionally, the first multimedia frame carries identification information, where the identification information indicates whether the first multimedia frame is a key frame;

[0022] The switching the cached first multimedia frame to be pushed to the target mixed multimedia frame for pushing includes:

[0023] Identify, according to identification information carried by the cached first multimedia frame to be pushed, whether the cached first multimedia frame to be pushed is a key frame;

[0024] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0025] Optionally, the performing mixed stream processing on the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream includes:

[0026] Decoding the first multimedia stream and the second multimedia stream respectively to obtain decoded first multimedia stream and second multimedia stream;

[0027] Mixing the decoded first multimedia stream and the second multimedia stream to obtain an initial mixed multimedia stream;

[0028] The initial mixed multimedia stream is encoded to obtain a target mixed multimedia stream.

[0029] Optionally, after switching the first multimedia stream to the mixed multimedia stream for pushing, the method further includes:

[0030] If it is detected that the interaction between the first user terminal and the second user terminal is disconnected, the mixed multimedia stream is switched to the first multimedia stream for push.

[0031] According to a second aspect of an embodiment of the present application, a multimedia stream switching device is provided, including:

[0032] A receiving module, configured to receive a first multimedia stream sent by a first user terminal, cache the first multimedia stream based on a set delay time, and push the cached first multimedia stream based on a current moment;

[0033] an acquisition module, configured to acquire a second multimedia stream of the second user terminal if an interaction request between the first user terminal and the second user terminal is received during the push process of the first multimedia stream;

[0034] a stream mixing module, configured to perform a stream mixing process on the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein a stream mixing time for performing the stream mixing process is less than or equal to the set delay time;

[0035] The switching module is configured to switch the first multimedia stream to the mixed multimedia stream for pushing.

[0036] According to a second aspect of an embodiment of the present application, a multimedia stream switching system is provided, including:

[0037] A first client terminal is used to send a first multimedia stream to a server terminal;

[0038] The server is used to cache the first multimedia stream based on a set delay time, and push the cached first multimedia stream based on the current moment; during the push process of the first multimedia stream, if an interaction request is received from the first user end and the second user end, the second multimedia stream of the second user end is obtained; the first multimedia stream and the second multimedia stream are mixed to obtain a mixed multimedia stream, wherein the mixing time for the mixing process is less than or equal to the set delay time; and the first multimedia stream is switched to the mixed multimedia stream for pushing.

[0039] According to a fourth aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the multimedia stream switching method when executing the computer instructions.

[0040] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and when the computer instructions are executed by a processor, the steps of the multimedia stream switching method are implemented.

[0041] The multimedia stream switching method provided in the present application receives a first multimedia stream sent by a first user terminal, caches the first multimedia stream based on a set delay time, and pushes the cached first multimedia stream based on the current moment; during the push process of the first multimedia stream, if an interactive request is received between the first user terminal and the second user terminal, the second multimedia stream of the second user terminal is obtained; the first multimedia stream and the second multimedia stream are mixed to obtain a mixed multimedia stream, wherein the mixed stream time for the mixed stream processing is less than or equal to the set delay time; the first multimedia stream is switched to the mixed multimedia stream for pushing. In this way, by pushing the cached first multimedia stream based on the current moment, the first multimedia stream does not need to be interrupted and re-pushed to different modules, avoiding the problem of interruption, and does not need to decode and encode the first multimedia stream, does not need to consume processor and memory resources, and avoids resource waste. By setting the delay time, so that the mixed stream time of the mixed stream processing is less than or equal to the set delay time, it can be ensured that the multimedia content corresponding to the output mixed multimedia stream is completely aligned and smooth, without jamming, and improves the smoothness of multimedia switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1A It is a schematic diagram of a live broadcast process in the prior art;

[0043] Figure 1B This is a schematic diagram of a client mixed stream connection process in the prior art;

[0044] Figure 1CThis is a schematic diagram of a server-side mixed stream connection process in the prior art;

[0045] Figure 2 is a flowchart of a multimedia stream switching method provided by an embodiment of the present application;

[0046] Figure 3A It is a schematic diagram of a process of pushing a first multimedia stream in a multimedia stream switching method provided in an embodiment of the present application;

[0047] Figure 3B It is a schematic diagram of a process of pushing a first multimedia stream in another multimedia stream switching method provided in an embodiment of the present application;

[0048] Figure 3C It is a schematic diagram of a process of pushing a mixed multimedia stream in a multimedia stream switching method provided in an embodiment of the present application;

[0049] Figure 3D It is a schematic diagram of a process of pushing a mixed multimedia stream in another multimedia stream switching method provided in an embodiment of the present application;

[0050] Figure 3E It is a schematic diagram of a first multimedia stream carrying a frame number in a multimedia stream switching method provided by an embodiment of the present application;

[0051] Figure 3F is a schematic diagram of a switching process in a multimedia stream switching method provided in an embodiment of the present application;

[0052] Figure 4 This is a processing flow chart of a multimedia stream switching method applied to live broadcast and microphone connection provided by an embodiment of the present application;

[0053] Figure 5 It is a structural schematic diagram of a multimedia stream switching device provided by an embodiment of the present application;

[0054] Figure 6 It is a structural schematic diagram of a multimedia stream switching system provided by an embodiment of the present application;

[0055] Figure 7 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0057] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0058] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] First, the terms involved in one or more embodiments of the present application are explained.

[0060] Anchor: The user who initiates the live broadcast. The audience can see the live broadcast images and sounds from the anchor's live broadcast room. Or hear the live broadcast sound.

[0061] Microphone user: A user who interacts with the anchor through a microphone. During the microphone connection, other viewers in the live broadcast room can also see the picture and sound of the microphone user, or hear the voice of the microphone user; before the microphone connection, it may be the anchor or an ordinary viewer; it may be one person or multiple people.

[0062] Multimedia stream: also known as stream, video stream or audio stream, usually contains both video images and audio sounds, or may contain only video images or only audio sounds.

[0063] Live stream: a stream distributed to the live broadcast room so that the audience can actually see it. In the non-connected microphone state, it is the same as the host's stream, that is, the image and sound, or sound, of the host's live broadcast. In the connected microphone state, it is a mixed stream that contains the image and sound, or sound, of both the host and the connected microphone user.

[0064] Mixed stream: Mix multiple multimedia streams from the host and the connected users into one multimedia stream, which is then distributed to the live broadcast room as a live stream for viewers to watch.

[0065] In the present application, a multimedia stream switching method is provided. The present application also relates to a multimedia stream switching device, a multimedia stream switching system, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0066] With the rapid development of computer technology, Internet technology has become more and more mature, among which live broadcast technology is the most prominent. Figure 1A , Figure 1A The figure shows a schematic diagram of a live broadcast process in the prior art. During the live broadcast, the client corresponding to the host sends the host stream to the server, which directly distributes it to the live broadcast room, that is, the client corresponding to the audience. In order to enhance the interactivity of the live broadcast, live broadcast connection came into being.

[0067] When the host initiates a live broadcast, the host's own video and audio must be mixed together with the other party's video and audio. Figure 1B , Figure 1B The schematic diagram of the process of connecting microphones with a client-side mixed stream in the prior art is shown. When the host and the connected microphone user are connected microphone, a mixed stream module is added to the client corresponding to the host and the connected microphone user stream sent by the connected microphone user terminal corresponding to the connected microphone user is received. The host stream and the connected microphone user stream are mixed on the client corresponding to the host to obtain a mixed live stream. The client corresponding to the host then sends the mixed stream to the server, which directly distributes it to the live broadcast room, that is, the client corresponding to the audience. However, the mixed stream on the client is limited by the performance of the client corresponding to the host. In the mixed stream or connected microphone process of multiple people, the client corresponding to the host also needs to run other tasks, such as playing games, superimposing some special effects, etc., and the performance of the client is difficult to support these.

[0068] In order to reduce the resource consumption of the client corresponding to the anchor and make full use of the performance of the server to support multi-person mixed streaming, the existing technology usually adopts the server-side mixed streaming method. Figure 1C , Figure 1C The schematic diagram of the process of mixing streams at the server side in the prior art is shown: when the host and the connected-microphone user are connected to the microphone, a mixed stream module is added to the server side and the connected-microphone user stream sent by the connected-microphone user side corresponding to the connected-microphone user is received. The server side mixes the host stream corresponding to the host and the connected-microphone user stream corresponding to the connected-microphone user to obtain a mixed live stream, and then distributes the mixed stream to the live broadcast room, that is, the server side needs to mix the live streams of the host and the users participating in the connected-microphone together, and then push them to the audience clients watching the live broadcasts of each connected-microphone client. In this way, the live broadcast content of the user's connected-microphone interaction can be displayed on the audience client.

[0069] However, since the server needs to mix the live streams of the anchor and the connected user, it takes a certain amount of time. Directly switching the multimedia stream corresponding to the anchor to the live stream mixed and processed by the server will cause the stream to be interrupted or the picture to be stuck and then restored, thereby reducing the experience when connected.

[0070] Therefore, the present specification provides a multimedia stream switching method, receiving a first multimedia stream sent by a first user terminal, caching the first multimedia stream based on a set delay time, and pushing the cached first multimedia stream based on the current moment; in the process of pushing the first multimedia stream, if an interactive request is received between the first user terminal and the second user terminal, obtaining the second multimedia stream of the second user terminal; mixing the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein the mixing time for the mixing process is less than or equal to the set delay time; switching the first multimedia stream to the mixed multimedia stream for pushing. In this way, by pushing the cached first multimedia stream based on the current moment, the first multimedia stream does not need to be interrupted and re-pushed to different modules, avoiding the problem of interruption, and does not need to decode and encode the first multimedia stream, does not need to consume processor and memory resources, and avoids resource waste. By setting the delay time, so that the mixing time of the mixing process is less than or equal to the set delay time, it can be ensured that the multimedia content corresponding to the output mixed multimedia stream is completely aligned and smooth, without jamming, and improves the smoothness of multimedia switching.

[0071] The multimedia stream switching method provided in one or more embodiments of this specification can be applied to multimedia stream switching in various scenarios: online video conferencing, live broadcast, online voice conferencing, live PK, etc. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any limitations on this.

[0072] Figure 2 A flowchart of a multimedia stream switching method provided by an embodiment of the present application is shown, which specifically includes the following steps:

[0073] Step 202: receiving a first multimedia stream sent by a first user terminal, caching the first multimedia stream based on a set delay time, and pushing the cached first multimedia stream based on the current moment.

[0074] Specifically, the user terminal can be any electronic product that can perform human-computer interaction with the user, such as a PC (Personal Computer), a mobile phone, a PPC (Pocket PC), a tablet computer, etc. The first user terminal refers to any user terminal that does not interact. The server can be a server, or a server cluster composed of multiple servers, or a cloud computing service center, which is not limited in the embodiments of the present application. Setting the delay time refers to the time for delaying the sending of the first multimedia stream set on the server. For example, if the delay time is set to 3 seconds, the server will push the first multimedia stream received at the 1st second at the 4th second, and push the first multimedia stream received at the 2nd second at the 5th second; in addition, setting the delay time can also be understood as setting the cache time, that is, the length of time that the first multimedia stream that the server needs to cache can be played.

[0075] In actual applications, users conduct live broadcasts, conferences, etc. through the first user terminal. Accordingly, the first user terminal will establish a link with the server terminal and send the generated first multimedia stream to the server terminal, which is the process of pulling the stream. Furthermore, when the server terminal receives the first multimedia stream, it caches the continuous first multimedia stream based on the set delay time and pushes the cached first multimedia stream based on the current moment, which is the process of pushing the stream.

[0076] Combination Figure 3A and Figure 3B To explain, Figure 3A is a schematic diagram of a process of pushing a first multimedia stream in a multimedia stream switching method provided in an embodiment of the present application, Figure 3B This is a flow chart of pushing the first multimedia stream in another multimedia stream switching method provided by an embodiment of the present application. When the first user end pushes the stream, the first multimedia stream is pushed to the mixing module of the server end. The mixing module uses the copy mode by default, that is, when only a single multimedia stream is input, no decoding and encoding are performed, and the input first multimedia stream is directly copied and forwarded, so that compared with the decoding and encoding actions, almost no central processing unit CPU and memory resources are consumed; at this time, the switching unit in the mixing module also performs pure transparent transmission and forwarding of the first multimedia stream in the cache unit to the live broadcast room, that is, the audience end, without doing anything.

[0077] In one or more optional embodiments of the present specification, a cache unit is provided in the server, and the cache unit can cache the first multimedia stream with a set delay time. When the server receives the first multimedia stream, the first multimedia stream is stored through the cache unit. When the cache unit stores enough first multimedia streams for the set delay time and the server is still receiving the first multimedia stream, the first multimedia stream cached first is pushed according to the first-in-first-out rule, and the first multimedia stream received later is added to the cache unit.

[0078] For example, the delay time is set to 2 seconds, that is, the cache unit can store the first multimedia stream for 2 seconds. The server starts to receive the first multimedia stream at the 1st second, caches the first multimedia stream of the 1st second in the cache unit, and does not push the first multimedia stream; at the 2nd second, caches the first multimedia stream of the 2nd second in the cache unit and does not push the first multimedia stream; at the 3rd second, pushes the first multimedia stream of the 1st second cached in the cache unit, and caches the first multimedia stream of the 3rd second in the cache unit; at the 4th second, pushes the first multimedia stream of the 2nd second cached in the cache unit, and caches the first multimedia stream of the 4th second in the cache unit, and so on.

[0079] In one or more optional embodiments of the present specification, a cache unit is provided in the server, and the cache unit can cache the first multimedia stream of the set push time, wherein the set push time is greater than or equal to the set delay time. When the server receives the first multimedia stream, it stores the first multimedia stream through the cache unit, and when the cache time of the first multimedia stream in the cache unit reaches the set push time, the server takes out the first multimedia stream whose cache time reaches the set push time from the cache unit and pushes it.

[0080] Optionally, in order to ensure the push timeliness of the first multimedia stream, the first multimedia stream may be pushed according to the current time. That is, the first multimedia stream cached is pushed based on the current time, and the specific implementation process may be as follows:

[0081] Subtract the current time from the set delay time to obtain the time to be pushed;

[0082] Push the first multimedia stream corresponding to the time to be pushed.

[0083] In practical applications, the real time, that is, the current time, can be obtained, and the current time can be subtracted from the set delay time to obtain the time to be pushed, and then the first multimedia stream corresponding to the time to be pushed can be pushed. In this way, the time to be pushed can be determined by subtracting the current time from the set delay time, which is simple and fast, and can simplify the calculation while ensuring the push timeliness of the first multimedia stream, reduce the amount of calculation, reduce the data processing pressure on the server, and thus improve the switching efficiency of the multimedia stream.

[0084] For example, the current time is 00:10, the delay time is set to 5 seconds, and the time to be pushed is 00:05, then the server will push the first multimedia stream received at 00:05.

[0085] Optionally, the first cached multimedia stream is pushed based on the current time, and can also be pushed as follows: subtract the current time from the set delay time to obtain an initial time to be pushed; subtract the initial time to be pushed from the set interval length to obtain a target time to be pushed, and push the first multimedia stream corresponding to the target time to be pushed.

[0086] For example, if the current time is 00:10, the delay time is set to 1 second, the interval time is set to 1 second, the initial push time is 00:09, the target push time is 00:08, and the server pushes the first multimedia stream received at 00:08.

[0087] Step 204: During the push process of the first multimedia stream, if an interaction request between the first user terminal and the second user terminal is received, the second multimedia stream of the second user terminal is obtained.

[0088] Specifically, the second user terminal refers to any user terminal that establishes interaction with the first user terminal. An interaction request refers to a request initiated by two users through a client to interact, such as a request to connect to a microphone, a request to participate in an online meeting, etc. The initiator of the interaction request can be the first user terminal or the second user terminal, and this application does not limit this.

[0089] In actual applications, when the server is pushing the first multimedia stream, if it receives an interaction request between the first user terminal and the second user terminal, the server obtains the second multimedia stream of the second user terminal, and can allow the second user terminal to obtain the second multimedia stream, or can obtain the second multimedia stream from the server that pushes the stream from the second user terminal. This application does not limit this.

[0090] After receiving the interaction request, the server can obtain the second multimedia stream of the second user terminal in real time; the interaction request can also carry a request timestamp, and the server can obtain the request timestamp and the second multimedia stream after the request timestamp based on the request timestamp.

[0091] It should be noted that, in order to ensure the correctness of the mixed stream, before obtaining the second multimedia stream of the second user terminal, it is also necessary to verify whether the interaction request between the first user terminal and the second user terminal is passed. If it is passed, the second multimedia stream of the second user terminal is obtained. If it is not passed, the second multimedia stream of the second user terminal is not obtained. For example, the initiator of a video conference invites a person to join a video conference through the first client. If the person rejects the invitation of the initiator of the video conference through the second client, the second multimedia stream of the second client is not obtained. If the person agrees to the invitation of the initiator of the video conference through the second client, the second multimedia stream of the second client is obtained.

[0092] Step 206: Mix the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein the mixing time of the mixing process is less than or equal to the set delay time.

[0093] In practical applications, after obtaining the first multimedia stream and the second multimedia stream, the first multimedia stream and the second multimedia stream need to be mixed to obtain a mixed multimedia stream. It should be noted that the mixing time of the mixing process is less than or equal to the set delay time.

[0094] In one or more optional embodiments of the present specification, a multimedia stream is composed of a plurality of multimedia frames carrying timestamps. In order to improve the quality of the mixed multimedia stream, the multimedia streams with the same timestamps need to be mixed. That is, when the first multimedia stream is composed of a plurality of first multimedia frames carrying timestamps, and the second multimedia stream is composed of a plurality of second multimedia frames carrying timestamps, the first multimedia stream and the second multimedia stream are mixed to obtain a mixed multimedia stream. The specific implementation process may be as follows:

[0095] The first multimedia frame and the second multimedia frame carrying the same timestamp are mixed to obtain a mixed multimedia frame carrying the timestamp, wherein a plurality of mixed multimedia frames constitute a mixed multimedia stream.

[0096] In practical applications, each first multimedia frame in the first multimedia stream and each second multimedia frame in the second multimedia stream carry a timestamp. The essence of mixed stream processing is to mix the first multimedia frame with the second multimedia frame. In order to ensure the authenticity and accuracy of the mixed multimedia frame, it is necessary to compare the timestamps of the first multimedia frame and the second multimedia frame, and mix the first multimedia frame and the second multimedia frame carrying the same timestamp to obtain a mixed multimedia frame that also carries the same timestamp, and then obtain a mixed multimedia stream.

[0097] In one or more optional embodiments of the present specification, the first multimedia stream and the second multimedia stream may be decoded, mixed, and encoded to obtain a mixed multimedia stream. That is, the first multimedia stream and the second multimedia stream are mixed to obtain a mixed multimedia stream. The specific implementation process may be as follows:

[0098] Decoding the first multimedia stream and the second multimedia stream respectively to obtain decoded first multimedia stream and second multimedia stream;

[0099] Mixing the decoded first multimedia stream and the second multimedia stream to obtain an initial mixed multimedia stream;

[0100] The initial mixed multimedia stream is encoded to obtain a target mixed multimedia stream.

[0101] In practical applications, when performing mixed stream processing, the first multimedia stream can be first decoded to obtain a decoded first multimedia stream, and the second multimedia stream can be decoded to obtain a decoded second multimedia stream. Then, the decoded first multimedia stream and the decoded second multimedia stream are mixed to obtain an initial mixed multimedia stream. Further, the initial mixed multimedia stream is encoded to obtain a target mixed multimedia stream.

[0102] In this way, decoding the first multimedia stream and the second multimedia stream can make it easier to mix the first multimedia stream and the second multimedia stream, and make the mixed multimedia stream more accurate.

[0103] exist Figure 3B Based on Figure 3C and Figure 3D To explain, Figure 3C A schematic diagram of a process of pushing a mixed multimedia stream in a multimedia stream switching method provided in an embodiment of the present application is shown. Figure 3D A schematic diagram of the process of pushing a mixed multimedia stream in another multimedia stream switching method provided by an embodiment of the present application is shown: when the first user terminal interacts with the second user terminal, that is, the server terminal receives the interaction request between the first user terminal and the second user terminal, the second multimedia of the second user terminal is also directly pushed to the mixing module of the server terminal. At this time, the mixing module has more than one multimedia stream input, and the mixing mode is automatically turned on, that is, the two decoding units decode the first multimedia stream and the second multimedia stream respectively, and then the mixing unit performs mixing processing, and then the encoding unit performs encoding processing to obtain a mixed multimedia stream.

[0104] Since decoding, mixing and encoding actions will bring additional delay (typically 1 to 2 seconds), that is, the mixing duration, directly switching from the copy mode to the mixing mode will cause the output of the mixing module to be discontinuous; therefore, in the copy mode, the cache unit will first cache a small segment of data (typically 1 to 2 seconds) according to the corresponding delay, that is, cache the first multimedia stream with the set delay push time; when switching from the copy mode to the mixing mode, the output data still comes from the first multimedia stream cached by the cache unit until the encoding unit after mixing starts to output, so that the output can be smooth and continuous.

[0105] Step 208: Switch the first multimedia stream to the mixed multimedia stream for pushing.

[0106] In actual applications, after the mixed multimedia stream is obtained, the first multimedia stream can be switched to the mixed multimedia stream for push. Figure 3C and Figure 3D The mixing module pushes the mixed multimedia stream to the live broadcast room. Specifically, the encoding unit performs encoding processing to obtain the mixed multimedia stream, and then the switching module switches the first multimedia stream to the mixed multimedia stream and pushes it to the live broadcast room.

[0107] In one or more optional embodiments of the present specification, in order to ensure the continuity of push and thus improve user viewing, after obtaining the mixed multimedia stream, the request timestamp carried by the interactive request can be added to the set delay time to obtain the switching time, and then when the switching time comes, the first multimedia stream can be switched to the mixed multimedia stream for push.

[0108] For example, the request timestamp is 10:00, the delay time is set to 5 seconds, and the switching time is 10:05, that is, at 10:05, the first multimedia stream is switched to the mixed multimedia stream for push.

[0109] In one or more optional embodiments of the present specification, the media stream is composed of multiple multimedia frames carrying timestamps. In order to ensure the continuity of push and thus improve the user's viewing experience, the timestamp carried by the cached first multimedia frame to be pushed can be compared with the timestamp carried by the mixed multimedia frame, and whether to switch is determined based on the comparison result. That is, when the first multimedia stream is composed of multiple first multimedia frames carrying timestamps, and the mixed multimedia stream is composed of multiple mixed multimedia frames carrying timestamps, the specific implementation process of switching the first multimedia stream to the mixed multimedia stream for push can be as follows:

[0110] Compare the cached first multimedia frame to be pushed with the obtained mixed multimedia frame to check whether there is a target mixed multimedia frame with the same timestamp as that carried by the first multimedia frame to be pushed;

[0111] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0112] In practical applications, each multimedia frame carries a timestamp. For the first multimedia frame to be pushed, that is, the cached first multimedia frame to be pushed, the timestamp carried by the first multimedia frame is compared with the timestamp carried by the mixed multimedia frame. If there is a target mixed multimedia frame with the same timestamp, the target mixed multimedia frame is used to replace the first multimedia frame for pushing, that is, the target mixed multimedia frame is switched to the target mixed multimedia frame for pushing, and then, each mixed multimedia frame is pushed in sequence from front to back in the order of timestamps. If there is no target mixed multimedia frame with the same timestamp, the first multimedia frame is pushed, and the above process is repeated for the next cached first multimedia frame to be pushed.

[0113] In this way, it can be ensured that the multimedia stream pushed to the audience is coherent and connected, avoiding the problem of multimedia stream freezing or jumping, thereby improving switching efficiency and user stickiness.

[0114] Optionally, based on the determination of the target mixed multimedia frame, the cached first multimedia frame to be pushed can be directly switched to the target mixed multimedia frame for pushing. In this way, while improving the switching efficiency, the switching process is slowed down and the processing efficiency of the server is improved.

[0115] Optionally, based on the determination of the target mixed multimedia frame, it is further possible to determine whether the target mixed multimedia frame or the first multimedia frame is a key frame. Determine whether to switch based on the judgment result. That is, the first multimedia frame carries identification information, and the identification information indicates whether the first multimedia frame is a key frame. At this time, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing. The specific implementation process can be as follows:

[0116] Identify, according to identification information carried by the cached first multimedia frame to be pushed, whether the cached first multimedia frame to be pushed is a key frame;

[0117] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0118] In practical applications, the first multimedia frame carries identification information. After determining the target mixed multimedia frame, it is possible to identify whether the cached first multimedia frame to be pushed is a key frame, that is, to identify it according to the identification information carried by the cached first multimedia frame to be pushed. If so, the target mixed multimedia frame is switched to the target mixed multimedia frame for pushing, and thereafter, each mixed multimedia frame is pushed in sequence from front to back in the order of timestamps. If not, the cached first multimedia frame to be pushed is pushed, and the above process is repeated for the next cached first multimedia frame to be pushed.

[0119] In addition, since the mixed multimedia frame is obtained by mixing the first multimedia frame and the second multimedia frame, the mixed multimedia frame also carries identification information. The cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing, and it can also be based on the identification information carried by the target mixed multimedia frame, to identify whether the mixed multimedia frame is a key frame; if so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing; if not, the cached first multimedia frame to be pushed is pushed.

[0120] Since non-key frames have forward dependencies, even if the timestamps are the same, direct use may cause problems such as distorted screens. Therefore, switching based on the judgment results of key frames can avoid problems such as distorted screens, further improve the accuracy and efficiency of multimedia stream switching, and improve user viewing and user stickiness.

[0121] It should be noted that if the interaction between the first user terminal and the second user terminal is disconnected, the reverse operation of the multimedia stream switching can be performed. That is, after the first multimedia stream is switched to the mixed multimedia stream for pushing, it also includes: if it is detected that the interaction between the first user terminal and the second user terminal is disconnected, the mixed multimedia stream is switched to the first multimedia stream for pushing.

[0122] For example, see Figure 3E , Figure 3E A schematic diagram of the first multimedia stream carrying frame numbers in a multimedia stream switching method provided by an embodiment of the present application is shown: the first multimedia stream input by the mixing module can be numbered for each first multimedia frame; the numbering method is not limited to a continuously increasing numerical value or a display timestamp, etc., preferably, the label is a timestamp; and it is recorded whether each frame is a key frame, that is, the first multimedia frame carries identification information.

[0123] In the copy mode, the first multimedia stream with the set delay time is cached in the cache unit, and the number of cached first multimedia frames can be determined by the difference of the timestamp carried by the first multimedia frame, or obtained by dividing the set delay time by the frame rate (number of frames per second), etc.; the set delay time is always guaranteed to be greater than the delay caused by decoding, mixing, and encoding, that is, the mixing time of the mixing process is less than or equal to the set delay time;

[0124] When the mixed stream mode is turned on, the first multimedia frames with the same number are sent to the decoding unit, mixing unit, and encoding unit modules, and each module retains the timestamp carried by the first multimedia frame;

[0125] See also Figure 3F , Figure 3FA schematic diagram of a switching process in a multimedia stream switching method provided by an embodiment of the present application is shown: after the encoding unit in the mixed stream mode starts to output, the switching unit will receive a mixed multimedia stream, wherein each mixed multimedia frame of the mixed multimedia stream carries a timestamp (such as 1, 2, ..., N) and identification information. At this time, the switching unit enters the switching logic: first, the switching unit caches all data (mixed multimedia frames) from the mixed multimedia stream; then, before the switching unit sends a first multimedia frame (a multimedia frame in the first multimedia stream from the cache unit), it always searches for a mixed multimedia frame with the same timestamp as the first multimedia frame from the mixed multimedia frames of the mixed multimedia stream, if not, the next frame continues to search; if so, an aligned mixed multimedia frame has been found, such as the first cache frame 2 and the mixed multimedia frame 2, and then the next mixed multimedia frame or the first video frame carrying the identification information as a key frame is switched, that is, the next key frame is switched; after the switch, the expired cache (first multimedia frame) before the mixed stream multimedia frame can be cleared; thus, the multimedia stream completes the seamless switching process.

[0126] It should be noted that multimedia frames can be audio frames or video frames, and audio frames and video frames can follow the switching logic. In addition, each audio frame can be considered a key frame, so as long as the aligned timestamp is found, the switch can be made. If it is a video frame, in order to ensure the synchronization of sound and picture, the actual switching process should occur when the next switching point has been found, that is, the next key frame after the picture is aligned, and the key frame in the same position after the sound is aligned, and the switch should be performed at the same time to ensure the synchronization of sound and picture;

[0127] After the switch, the timestamps of the first multimedia stream from the cache unit and the mixed multimedia stream are always aligned, and the mode can be switched back to the copy mode at any time, such as when an exception occurs or the interaction ends. During the interaction, the first multimedia stream from the cache unit enters the switching unit and can be directly discarded without any additional processing.

[0128] Optionally, the multimedia frames can be numbered only for key frames. For example, there is a key frame every 1 to 10 seconds during live broadcast. Then, when querying alignment in the switching module, only the key frames need to be queried and aligned, and non-key frames (unnumbered) can be skipped, which can greatly optimize the query efficiency during alignment.

[0129] When the interaction ends, the second multimedia stream is disconnected, and the input of the stream mixing module is reduced back to one stream, namely the first multimedia stream, and the reverse operation is followed to switch from the stream mixing mode back to the copy mode, so no additional resources are consumed. This stream mixing module can make full use of server resources and conveniently support multi-person interaction.

[0130] In this way, when the host is broadcasting live, the host stream always passes through the optimized mixing module, so the host stream does not need to be interrupted and re-pushed to different modules, which solves the problem of interruption; the optimized mixing module defaults to copy mode, that is, when only a single stream is input, no decoding and encoding are performed, and the input stream is directly copied and forwarded, so compared with the decoding and encoding actions, almost no CPU and memory resources are consumed, so there is no obvious waste of resources. When switching between copy mode and mixing mode inside the mixing module, seamless switching can be achieved through cache and frame number alignment, that is, the output audio and video content is completely aligned and smooth, without any jamming.

[0131] The multimedia stream switching method provided in the present application receives a first multimedia stream sent by a first user terminal, caches the first multimedia stream based on a set delay time, and pushes the cached first multimedia stream based on the current moment; during the push process of the first multimedia stream, if an interactive request is received between the first user terminal and the second user terminal, the second multimedia stream of the second user terminal is obtained; the first multimedia stream and the second multimedia stream are mixed to obtain a mixed multimedia stream, wherein the mixed stream time for the mixed stream processing is less than or equal to the set delay time; the first multimedia stream is switched to the mixed multimedia stream for pushing. In this way, by pushing the cached first multimedia stream based on the current moment, the first multimedia stream does not need to be interrupted and re-pushed to different modules, avoiding the problem of interruption, and does not need to decode and encode the first multimedia stream, does not need to consume processor and memory resources, and avoids resource waste. By setting the delay time, so that the mixed stream time of the mixed stream processing is less than or equal to the set delay time, it can be ensured that the multimedia content corresponding to the output mixed multimedia stream is completely aligned and smooth, without jamming, and improves the smoothness of multimedia switching.

[0132] The following combination Figure 4 Taking the application of the multimedia stream switching method provided by this application in live broadcast and microphone connection as an example, the multimedia stream method is further explained. Figure 4 A processing flow chart of a multimedia stream switching method applied to live broadcast and microphone connection provided by an embodiment of the present application is shown, which specifically includes the following steps:

[0133] Step 402: Receive a first video stream sent by the anchor end, and cache the first video stream based on a set delay time, wherein the first video stream is composed of a plurality of first video frames carrying timestamps.

[0134] Step 404: Subtract the current time from the set delay time to obtain the time to be pushed; and push the first video frame corresponding to the time to be pushed.

[0135] Step 406: During the push process of the first video stream, if an interaction request is received between the host end and the microphone-connected user end, a second video stream of the microphone-connected user end is obtained, wherein the second video stream is composed of a plurality of second video frames carrying timestamps.

[0136] Step 408: Decode the first video frame and the second video frame that carry the same timestamp to obtain decoded first video frame and second video frame.

[0137] Step 410: Mix the decoded first video stream and the second video stream to obtain an initial mixed video stream; encode the initial mixed video stream to obtain a target mixed video stream, wherein a plurality of mixed video frames constitute the mixed video stream.

[0138] Step 412: Compare the cached first video frame to be pushed with the obtained mixed video frame to check whether there is a target mixed video frame with the same timestamp as that carried by the first video frame to be pushed.

[0139] Step 414: If it exists, identify whether the first video frame cached and to be pushed is a key frame according to the identification information carried by the first video frame cached and to be pushed.

[0140] Step 416: If yes, the first video frame cached and to be pushed is switched to the target mixed video frame for pushing.

[0141] Step 418: If it is detected that the interaction between the host end and the microphone-connected user end is disconnected, the mixed video stream is switched to the first video frame for push.

[0142] The multimedia stream switching method provided by the present application pushes the cached first multimedia stream based on the current moment, so that the first multimedia stream does not need to be disconnected and re-pushed to different modules, thereby avoiding the problem of disconnection, and does not need to decode and encode the first multimedia stream, and does not need to consume processor and memory resources, thereby avoiding resource waste. By setting the delay time so that the mixing time of the mixed stream processing is less than or equal to the set delay time, it can be ensured that the multimedia content corresponding to the output mixed multimedia stream is completely aligned and smooth without jamming, thereby improving the smoothness of multimedia switching.

[0143] Corresponding to the above method embodiment, the present application also provides a multimedia stream switching device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a multimedia stream switching device provided by an embodiment of the present application. Figure 5 As shown, the device comprises:

[0144] The receiving module 502 is configured to receive a first multimedia stream sent by a first user terminal, cache the first multimedia stream based on a set delay time, and push the cached first multimedia stream based on a current moment;

[0145] The acquisition module 504 is configured to acquire the second multimedia stream of the second user terminal if an interaction request between the first user terminal and the second user terminal is received during the push process of the first multimedia stream;

[0146] The mixing module 506 is configured to mix the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein the mixing time of the mixing process is less than or equal to the set delay time;

[0147] The switching module 508 is configured to switch the first multimedia stream to the mixed multimedia stream for pushing.

[0148] Optionally, the switching module 508 is further configured to:

[0149] Subtract the current time from the set delay time to obtain the time to be pushed;

[0150] Push the first multimedia stream corresponding to the time to be pushed.

[0151] Optionally, the first multimedia stream consists of a plurality of first multimedia frames carrying timestamps, and the second multimedia stream consists of a plurality of second multimedia frames carrying timestamps;

[0152] The mixing module 506 is further configured to:

[0153] The first multimedia frame and the second multimedia frame carrying the same timestamp are mixed to obtain a mixed multimedia frame carrying the timestamp, wherein a plurality of mixed multimedia frames constitute a mixed multimedia stream.

[0154] Optionally, the first multimedia stream is composed of a plurality of first multimedia frames carrying timestamps, and the mixed multimedia stream is composed of a plurality of mixed multimedia frames carrying timestamps;

[0155] The switching module 508 is further configured to:

[0156] Compare the cached first multimedia frame to be pushed with the obtained mixed multimedia frame to check whether there is a target mixed multimedia frame with the same timestamp as that carried by the first multimedia frame to be pushed;

[0157] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0158] Optionally, the first multimedia frame carries identification information, where the identification information indicates whether the first multimedia frame is a key frame;

[0159] The switching module 508 is further configured to:

[0160] Identify, according to identification information carried by the cached first multimedia frame to be pushed, whether the cached first multimedia frame to be pushed is a key frame;

[0161] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0162] Optionally, the stream mixing module 506 is further configured to:

[0163] Decoding the first multimedia stream and the second multimedia stream respectively to obtain decoded first multimedia stream and second multimedia stream;

[0164] Mixing the decoded first multimedia stream and the second multimedia stream to obtain an initial mixed multimedia stream;

[0165] The initial mixed multimedia stream is encoded to obtain a target mixed multimedia stream.

[0166] Optionally, the switching module 508 is further configured to:

[0167] If it is detected that the interaction between the first user terminal and the second user terminal is disconnected, the mixed multimedia stream is switched to the first multimedia stream for push.

[0168] The multimedia stream switching device provided by the present application pushes the cached first multimedia stream based on the current moment, so that the first multimedia stream does not need to be disconnected and re-pushed to different modules, avoiding the problem of disconnection, and does not need to decode and encode the first multimedia stream, and does not consume processor and memory resources, thus avoiding resource waste. By setting the delay time so that the mixing time of the mixed stream processing is less than or equal to the set delay time, it can be ensured that the multimedia content corresponding to the output mixed multimedia stream is completely aligned and smooth without jamming, thereby improving the smoothness of multimedia switching.

[0169] The above is a schematic scheme of a multimedia stream switching device of this embodiment. It should be noted that the technical scheme of the multimedia stream switching device and the technical scheme of the multimedia stream switching method described above belong to the same concept, and the details not described in detail in the technical scheme of the multimedia stream switching device can be referred to the description of the technical scheme of the multimedia stream switching method described above.

[0170] Corresponding to the above method embodiment, the present application also provides a multimedia stream switching system embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of a multimedia stream switching system provided by an embodiment of the present application. Figure 6 As shown, the system includes:

[0171] A first user terminal 602, a server terminal 604, a second user terminal 606 and a viewer terminal 608;

[0172] A first client 602 is used to send a first multimedia stream to a server 604;

[0173] The server end 604 is used to cache the first multimedia stream based on the set delay time, and push the cached first multimedia stream based on the current moment; during the push process of the first multimedia stream, if an interaction request is received between the first user end and the second user end, the second multimedia stream of the second user end is obtained; the first multimedia stream and the second multimedia stream are mixed to obtain a mixed multimedia stream, wherein the mixing time for the mixing process is less than or equal to the set delay time; the first multimedia stream is switched to the mixed multimedia stream for pushing to the audience end 608.

[0174] Optionally, the server 606 is further configured to:

[0175] Subtract the current time from the set delay time to obtain the time to be pushed;

[0176] Push the first multimedia stream corresponding to the time to be pushed.

[0177] Optionally, the first multimedia stream consists of a plurality of first multimedia frames carrying timestamps, and the second multimedia stream consists of a plurality of second multimedia frames carrying timestamps;

[0178] The server 606 is also used for:

[0179] The first multimedia frame and the second multimedia frame carrying the same timestamp are mixed to obtain a mixed multimedia frame carrying the timestamp, wherein a plurality of mixed multimedia frames constitute a mixed multimedia stream.

[0180] Optionally, the first multimedia stream is composed of a plurality of first multimedia frames carrying timestamps, and the mixed multimedia stream is composed of a plurality of mixed multimedia frames carrying timestamps;

[0181] The server 606 is also used for:

[0182] Compare the cached first multimedia frame to be pushed with the obtained mixed multimedia frame to check whether there is a target mixed multimedia frame with the same timestamp as that carried by the first multimedia frame to be pushed;

[0183] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0184] Optionally, the first multimedia frame carries identification information, where the identification information indicates whether the first multimedia frame is a key frame;

[0185] The server 606 is also used for:

[0186] Identify, according to identification information carried by the cached first multimedia frame to be pushed, whether the cached first multimedia frame to be pushed is a key frame;

[0187] If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

[0188] Optionally, the server 606 is further configured to:

[0189] Decoding the first multimedia stream and the second multimedia stream respectively to obtain decoded first multimedia stream and second multimedia stream;

[0190] Mixing the decoded first multimedia stream and the second multimedia stream to obtain an initial mixed multimedia stream;

[0191] The initial mixed multimedia stream is encoded to obtain a target mixed multimedia stream.

[0192] Optionally, the server 606 is further configured to:

[0193] If it is detected that the interaction between the first user terminal and the second user terminal is disconnected, the mixed multimedia stream is switched to the first multimedia stream for push.

[0194] The multimedia stream switching system provided by the present application pushes the cached first multimedia stream based on the current moment, so that the first multimedia stream does not need to be disconnected and re-pushed to different modules, thereby avoiding the problem of disconnection, and does not need to decode and encode the first multimedia stream, and does not need to consume processor and memory resources, thereby avoiding resource waste. By setting the delay time so that the mixing time of the mixed stream processing is less than or equal to the set delay time, it can be ensured that the multimedia content corresponding to the output mixed multimedia stream is completely aligned and smooth without jamming, thereby improving the smoothness of multimedia switching.

[0195] The above is a schematic scheme of a multimedia stream switching system of this embodiment. It should be noted that the technical scheme of the multimedia stream switching system and the technical scheme of the multimedia stream switching method described above belong to the same concept, and the details not described in detail in the technical scheme of the multimedia stream switching system can be referred to the description of the technical scheme of the multimedia stream switching method described above.

[0196] Figure 7 The block diagram of a computing device provided in an embodiment of the present application is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0197] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0198] In one embodiment of the present application, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0199] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 700 may also be a mobile or stationary server.

[0200] When the processor 720 executes the computer instructions, the steps of the multimedia stream switching method are implemented.

[0201] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the multimedia stream switching method described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the multimedia stream switching method described above.

[0202] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the multimedia stream switching method as described above.

[0203] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multimedia stream switching method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the multimedia stream switching method described above.

[0204] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0205] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0206] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0207] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0208] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A multimedia stream switching method, characterized in that: include: receiving a first multimedia stream sent by a first user terminal, and caching the first multimedia stream based on a set delay time, Pushing the first cached multimedia stream based on the current moment; During the push process of the first multimedia stream, if an interaction request between the first user terminal and the second user terminal is received, Acquire a second multimedia stream of the second user terminal; Performing a mixed stream processing on the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein a mixed stream time of the mixed stream processing is less than or equal to the set delay time; Switching the first multimedia stream to the mixed multimedia stream for pushing; The first multimedia stream is composed of a plurality of first multimedia frames carrying timestamps, and the mixed multimedia stream is composed of a plurality of mixed multimedia frames carrying timestamps; The switching the first multimedia stream to the mixed multimedia stream for pushing includes: Compare the cached first multimedia frame to be pushed with the obtained mixed multimedia frame to check whether there is a target mixed multimedia frame with the same timestamp as that carried by the first multimedia frame to be pushed; If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

2. The method according to claim 1, characterized in that: The pushing the first multimedia stream cached based on the current moment includes: Subtract the current time from the set delay time to obtain the time to be pushed; Push the first multimedia stream corresponding to the time to be pushed.

3. The method according to claim 1, characterized in that The first multimedia stream consists of a plurality of first multimedia frames carrying timestamps, and the second multimedia stream consists of a plurality of second multimedia frames carrying timestamps; The mixing the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream includes: The first multimedia frame and the second multimedia frame carrying the same timestamp are mixed to obtain a mixed multimedia frame carrying the timestamp, wherein a plurality of mixed multimedia frames constitute a mixed multimedia stream.

4. The method according to claim 1, characterized in that: The first multimedia frame carries identification information, where the identification information indicates whether the first multimedia frame is a key frame; The switching the cached first multimedia frame to be pushed to the target mixed multimedia frame for pushing includes: Identify, according to identification information carried by the cached first multimedia frame to be pushed, whether the cached first multimedia frame to be pushed is a key frame; If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

5. The method according to claim 1, characterized in that: The mixing the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream includes: Decoding the first multimedia stream and the second multimedia stream respectively to obtain decoded first multimedia stream and second multimedia stream; Mixing the decoded first multimedia stream and the second multimedia stream to obtain an initial mixed multimedia stream; The initial mixed multimedia stream is encoded to obtain a target mixed multimedia stream.

6. The method according to claim 1, characterized in that After the first multimedia stream is switched to the mixed multimedia stream for pushing, the method further includes: If it is detected that the interaction between the first user terminal and the second user terminal is disconnected, the mixed multimedia stream is switched to the first multimedia stream for push.

7. A multimedia stream switching device, characterized in that: include: A receiving module, configured to receive a first multimedia stream sent by a first user terminal, cache the first multimedia stream based on a set delay time, and push the cached first multimedia stream based on a current moment; an acquisition module, configured to acquire a second multimedia stream of the second user terminal if an interaction request between the first user terminal and the second user terminal is received during the push process of the first multimedia stream; a stream mixing module, configured to perform a stream mixing process on the first multimedia stream and the second multimedia stream to obtain a mixed multimedia stream, wherein a stream mixing time for performing the stream mixing process is less than or equal to the set delay time; A switching module, configured to switch the first multimedia stream to the mixed multimedia stream for pushing; The first multimedia stream is composed of a plurality of first multimedia frames carrying timestamps, and the mixed multimedia stream is composed of a plurality of mixed multimedia frames carrying timestamps; The switching the first multimedia stream to the mixed multimedia stream for pushing includes: Compare the cached first multimedia frame to be pushed with the obtained mixed multimedia frame to check whether there is a target mixed multimedia frame with the same timestamp as that carried by the first multimedia frame to be pushed; If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

8. A multimedia stream switching system, characterized in that: include: A first client terminal is used to send a first multimedia stream to a server terminal; The server is configured to cache the first multimedia stream based on a set delay time, and push the cached first multimedia stream based on the current moment; during the push process of the first multimedia stream, if an interaction request is received from the first user end and the second user end, the second multimedia stream of the second user end is obtained; the first multimedia stream and the second multimedia stream are mixed to obtain a mixed multimedia stream, wherein the mixing time for the mixed stream processing is less than or equal to the set delay time; the first multimedia stream is switched to the mixed multimedia stream for pushing to the viewer end; The first multimedia stream is composed of a plurality of first multimedia frames carrying timestamps, and the mixed multimedia stream is composed of a plurality of mixed multimedia frames carrying timestamps; The switching the first multimedia stream to the mixed multimedia stream for pushing includes: Compare the cached first multimedia frame to be pushed with the obtained mixed multimedia frame to check whether there is a target mixed multimedia frame with the same timestamp as that carried by the first multimedia frame to be pushed; If so, the cached first multimedia frame to be pushed is switched to the target mixed multimedia frame for pushing.

9. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer instructions, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images