Audio and video stream processing method and device for cross-platform live broadcast microphone connection, equipment and medium

Abstract

The invention relates to the technical field of network live broadcast, and provides an audio and video stream processing method and device for cross-platform live broadcast microphone connection, electronic equipment and a storage medium. According to the method, the audio frame information and the video frame information are extracted from the audio and video file, the reason of the abnormal playing event of the real-time microphone-connected audio and video stream can be automatically and quickly determined according to the audio frame information and the video frame information, manual participation is not needed, and the efficiency of determining the reason of the abnormal playing of the audio and video stream is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of network live broadcast technology, and in particular to a method, device, electronic device and storage medium for processing audio and video streams for cross-platform live broadcast and microphone connection. Background Art

[0002] With the advancement of network communication technology, live streaming has become a new form of online interaction. Live streaming refers to the technology of sharing live audio and video streams with the audience on the Internet through a live streaming platform. Live streaming is a new form of Internet business that reflects the open and shared characteristics of the Internet, giving every ordinary person the opportunity to show their talents on the Internet.

[0003] At present, in the process of live broadcasting, there are problems with the playback of audio and video streams, such as asynchrony, freeze, frame loss, etc. In the prior art, the test of abnormal playback of audio and video streams often requires subjective judgment and manual operation, resulting in low efficiency in determining the cause of abnormal playback of audio and video streams. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, and storage medium for processing audio and video streams for cross-platform live broadcast and microphone connection, which can improve the efficiency of determining the cause of abnormal audio and video stream playback. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides an audio and video stream processing method for a cross-platform live broadcast and microphone connection, which is applied to a cross-platform live broadcast and microphone connection system. The cross-platform live broadcast and microphone connection system includes a first live broadcast platform server and a second live broadcast platform server. The first live broadcast platform server and the second live broadcast platform server are connected in communication. The first live broadcast platform server pushes the audio and video stream of the first anchor client to the second live broadcast platform server, and the second live broadcast platform server sends the audio and video stream to the second anchor client; the method includes the following steps:

[0006] Acquire the abnormal event information of real-time audio and video stream playback of the second anchor client during the live broadcast; wherein the abnormal event information of real-time audio and video stream playback includes the start time and end time of the abnormal event of real-time audio and video stream playback; wherein the real-time audio and video stream is the audio and video stream received in real time by the second anchor client from the second live broadcast platform server;

[0007] Obtaining the live broadcast playback video of the second anchor client; wherein the live broadcast playback video is a video obtained after the second anchor client receives and stores the audio and video stream;

[0008] If there is no abnormality in the playback of the live broadcast with microphones, then determine a number of audio frame information and a number of video frame information corresponding to the start time and the end time according to the audio and video files obtained from the second live broadcast platform server; wherein the audio and video files are files obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server;

[0009] A video in a preset format is obtained according to a plurality of audio frame information and a plurality of video frame information; if the video in the preset format is synchronized with a video screen of a live broadcast playback video connected to a microphone, the video in the preset format is parsed.

[0010] In a second aspect, an embodiment of the present application provides an audio and video stream processing device for cross-platform live broadcast and microphone connection, including:

[0011] An event information acquisition module is used to obtain abnormal event information of real-time audio and video stream playback of the second anchor client during the live broadcast of the live broadcast; wherein the abnormal event information of real-time audio and video stream playback of the second anchor client includes the start time and end time of the abnormal event of real-time audio and video stream playback of the second anchor client; wherein the real-time audio and video stream of the second anchor client is the audio and video stream received in real time by the second anchor client from the second live broadcast platform server;

[0012] A playback video acquisition module is used to obtain the live broadcast playback video of the second anchor client; wherein the live broadcast playback video is a video obtained after the second anchor client receives and stores the audio and video stream;

[0013] A frame information determination module is used to determine a number of audio frame information and a number of video frame information corresponding to the start time and the end time according to the audio and video files obtained from the second live broadcast platform server if there is no abnormality in the playback video of the live broadcast with microphones connected; wherein the audio and video files are files obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server;

[0014] The video parsing module is used to obtain a video in a preset format according to a plurality of audio frame information and a plurality of video frame information; if the video in the preset format is synchronized with the video screen of the live broadcast playback video with microphone connection, the video in the preset format is parsed.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0017] The embodiment of the present application obtains the abnormal event information of the real-time audio and video stream playback of the second anchor client during the live broadcast process; wherein, the abnormal event information of the real-time audio and video stream playback includes the start time and end time of the abnormal event of the real-time audio and video stream playback; wherein, the real-time audio and video stream is the audio and video stream received by the second anchor client from the second live broadcast platform server in real time; the live broadcast playback video of the second anchor client is obtained; wherein, the live broadcast playback video of the second anchor client is obtained after the second anchor client receives and stores the audio and video stream; if there is no abnormality in the playback video of the live broadcast, then according to the audio and video file obtained from the second live broadcast platform server, determine the audio frame information and the video frame information corresponding to the start time and the end time; wherein, the audio and video file is the file obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server; according to the audio frame information and the video frame information, obtain a video in a preset format; if the video in the preset format is synchronized with the video screen of the live broadcast playback video, parse the video in the preset format. The present application saves the audio and video files on the second live broadcast platform server, extracts the audio frame information and the video frame information from the audio and video files according to the abnormal event information of the real-time audio and video stream playback with microphones, obtains the video in a preset format according to the audio frame information and the video frame information, and parses the video in the preset format. The cause of the abnormal event of the real-time audio and video stream playback with microphones is automatically and quickly determined without human intervention, thereby improving the efficiency of determining the cause of the abnormal event of the audio and video stream playback.

[0018] For better understanding and implementation, the technical solution of the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of an application scenario of the method for processing audio and video streams for cross-platform live broadcast and microphone connection provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a method for processing audio and video streams for cross-platform live broadcast and microphone connection provided by an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of an audio and video stream processing device for cross-platform live broadcast and microphone connection provided by an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0024] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" / "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0026] Those skilled in the art will appreciate that the "client", "terminal" and "terminal device" used in this application include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0027] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially a computer device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0028] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, each server should be logically divided. In physical space, these servers can be independent of each other but can be called through an interface, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0029] See also Figure 1 , Figure 1 A schematic diagram of an application scenario of the cross-platform live broadcast interaction method provided in an embodiment of the present application, wherein the application scenario includes the first live broadcast platform server 101 provided in an embodiment of the present application and the first anchor client 102 and the first audience client 103 logged in on the first live broadcast platform server 101, wherein the first anchor client 102 and the first audience client 103 interact with each other through the first live broadcast platform server 101; and the second live broadcast platform server 201 and the second anchor client 202 and the second audience client 203 logged in on the second live broadcast platform server 201, wherein the second anchor client 202 and the second audience client 203 interact with each other through the second live broadcast platform server 201.

[0030] Among them, the first anchor client 102 and the second anchor client 202 refer to the end that sends the live broadcast video, which is usually the client used by the anchor (ie, the live broadcast anchor user) in the live broadcast.

[0031] The first viewer client 103 and the second viewer client 203 refer to an end for receiving and viewing a live network video, and are generally the clients used by viewers (ie, live viewer users) who view the video in a live network video.

[0032] The hardware pointed to by the client is essentially a computer device, specifically, it can be a computer device of the type of a smart phone, a smart interactive tablet, a personal computer, etc. Each client can access the Internet through a well-known network access method and establish a data communication link with the server.

[0033] The first live broadcast platform server 101 and the second live broadcast platform server 201 may be business servers, each of which includes or is connected to the audio data server, video stream server and other servers providing related support for the first live broadcast platform and the second live broadcast platform, thereby forming a logically related service cluster to serve related terminal devices, such as Figure 1 Provides services to the various clients shown in .

[0034] In the embodiment of the present application, the host client and the audience client of the same live broadcast platform can join the live broadcast room (i.e., live broadcast channel) created by the corresponding live broadcast platform server. The above-mentioned live broadcast room refers to a chat room realized by Internet technology, which usually has audio and video broadcast control functions. The host user broadcasts live in the live broadcast room through the host client, and the audience of the audience client can log in to the live broadcast platform server to enter the live broadcast room to watch the live broadcast.

[0035] In the live broadcast room, the host and the audience can interact through known online interactive methods such as voice, video, and text. Generally, the host user performs programs for the audience in the form of audio and video streams, and can also generate behaviors during the interaction. Of course, the application form of the live broadcast room is not limited to online entertainment, but can also be extended to other related scenarios, such as: video conferencing scenarios, product promotion and sales scenarios, and any other scenarios that require similar interactions.

[0036] Specifically, the process of watching live broadcasts by viewers is as follows: viewers can click to access the live broadcast application installed on the viewer client, and choose to enter any live broadcast room, triggering the viewer client to load the live broadcast room interface for the viewer, and conduct various online interactions.

[0037] The first live broadcast platform server 101 and the second live broadcast platform server 201 are two independent network live broadcast platforms, such as the server of the first network live broadcast platform and the second network live broadcast platform.

[0038] The first live broadcast platform server 101 establishes a network connection with the second live broadcast platform server 201, and performs data exchange through the network. The first live broadcast platform server 101 creates a first virtual anchor client 104 and a first virtual anchor account corresponding to the first virtual anchor client 104 locally. The first virtual anchor client 104 is implemented by the first live broadcast platform server 101 through computational simulation, and the corresponding first virtual anchor account is an anchor account that can be logged in to the first live broadcast platform server 101.

[0039] When implementing cross-platform live broadcasting by anchors, the first live broadcast platform server 101 obtains the live broadcast media stream uploaded by the first anchor client 102 logged in on the platform. In addition, the first live broadcast platform server 101 obtains the second anchor account corresponding to the second anchor client 202 logged in by the second live broadcast platform server 201, and establishes a binding relationship between the second anchor account and the first virtual anchor account. The first live broadcast platform server 101 obtains the live broadcast media stream uploaded by the second anchor client 202 from the second live broadcast platform server 201 through the binding relationship, and maps the live broadcast media stream uploaded by the second anchor client 202 into the media stream of the first virtual anchor client 104 created locally, that is, the second live broadcast media stream, so that the second anchor client 202 logged in on the second live broadcast platform server 201 can be broadcast live in the first live broadcast platform server 101 as the first virtual anchor client 104. The live media stream of the first virtual anchor client 104 can be set by the first live platform server 101 to be viewable / searchable individually or not viewable / searchable individually. In this embodiment, the first live platform server 101 sets the live media stream of the first virtual anchor client 104 to be not viewable / searchable individually, that is, the live media stream of the first virtual anchor client 104 can only be used for microphone interaction.

[0040] When the live broadcast interaction is turned on, the first live broadcast platform server 101 creates a live broadcast room, adds the first virtual anchor client 104 and the first anchor client 102 to the live broadcast room, so that the first live broadcast media stream corresponding to the first anchor client 102 and the second live broadcast media stream corresponding to the first virtual anchor client 104 are both displayed in the live broadcast room.

[0041] In one embodiment, the first live broadcast platform server 101 obtains the first live broadcast media stream corresponding to the first anchor account and the second live broadcast media stream corresponding to the first virtual anchor account, mixes the first live broadcast media stream and the second live broadcast media stream to obtain a mixed live broadcast media stream, and sends the mixed live broadcast media stream to the client that has joined the live broadcast room with microphones. The client that joins the live broadcast room with microphones created by the first live broadcast platform server 101 includes the first live broadcast client 102 and the first audience client 103 that log in to the first live broadcast platform.

[0042] The second live broadcast platform server 201 implements cross-platform live broadcast in at least two ways:

[0043] The first method is: the second live broadcast platform server 201 also creates a second virtual anchor client 204 and a second virtual anchor account corresponding to the second virtual anchor client 204 locally. The second virtual anchor client 204 is implemented by the second live broadcast platform server 201 through computational simulation, and the corresponding second virtual anchor account is an anchor account that can be logged in by the second live broadcast platform server 201. The second live broadcast platform server 201 obtains the first anchor account corresponding to the first anchor client 102 logged in by the first live broadcast platform server 101, and establishes a binding relationship between the first anchor account and the second virtual anchor account. The second live broadcast platform server 201 obtains the live broadcast media stream uploaded by the first anchor client 102 from the first live broadcast platform server 101 through the binding relationship, and maps the live broadcast media stream uploaded by the first anchor client 102, that is, the first live broadcast media stream, into the media stream of the second virtual anchor client 204 created locally, so that the first anchor client 102 logged in to the first live broadcast platform server 101 can perform live broadcast, microphone connection, etc. in the second live broadcast platform server 201 as the second virtual anchor client 204. The second virtual anchor client 204 performs live broadcast and microphone connection in the second live broadcast platform server 201, which is similar to the first virtual anchor client 104 performing live broadcast and microphone connection in the first live broadcast platform server 101, and will not be described in detail here.

[0044] The second method is: the second live broadcast platform server 201 directly obtains the mixed live broadcast media stream formed by mixing the first live broadcast media stream and the second live broadcast media stream from the first live broadcast platform server 101 through the network connection with the first live broadcast platform server 101, and sends the mixed live broadcast media stream to the client that has joined the live broadcast room with microphones created locally by the second live broadcast platform server 201. The client that joins the live broadcast room with microphones created by the second live broadcast platform server 201 includes the second live broadcast client 202 and the second audience client 203 that log in to the second live broadcast platform.

[0045] In the existing technology, anchors can only communicate with each other through voice or video on the same live broadcast platform, and the communication mode is relatively simple, which affects the user's interactive experience during the live broadcast. For example, during the live broadcast, anchors on the self-developed live broadcast platform can only communicate with anchors on the self-developed platform, but cannot communicate with anchors on other third-party live broadcast platforms. In addition, during the communication on the same live broadcast platform, the audio and video stream playback of the live broadcast has problems such as asynchrony, freeze, and frame loss.

[0046] In the prior art, the test of abnormal audio and video stream playback often requires the use of human subjective judgment and manual operation, resulting in low efficiency in determining the cause of the abnormal audio and video stream playback.

[0047] To this end, an embodiment of the present application provides an audio and video stream processing method for cross-platform live broadcast and microphone connection.

[0048] See also Figure 2 , Figure 2 A flow chart of a method for processing audio and video streams of a cross-platform live broadcast and microphone connection is provided for an embodiment of the present application. The method is applied to a cross-platform live broadcast and microphone connection system. The cross-platform live broadcast and microphone connection system includes a first live broadcast platform server and a second live broadcast platform server. The first live broadcast platform server and the second live broadcast platform server are connected in communication. The first live broadcast platform server pushes the audio and video stream of the first anchor client to the second live broadcast platform server, and the second live broadcast platform server sends the audio and video stream to the second anchor client, including the following steps:

[0049] S10: Obtain the real-time abnormal event information of the audio and video stream playback of the second anchor client during the live broadcast process; wherein, the real-time abnormal event information of the audio and video stream playback of the second anchor client includes the start time and end time of the real-time abnormal event of the audio and video stream playback of the second anchor client; wherein, the real-time audio and video stream is the audio and video stream received in real time by the second anchor client from the second live broadcast platform server.

[0050] The audio and video stream is the live media stream uploaded by the first anchor client to the first live platform server, including an audio stream and a video stream. Specifically, the first anchor client collects the anchor's audio data, encodes the audio data, and obtains an audio stream. The first anchor client collects the anchor's video data, encodes the video data, and obtains a video stream. The audio stream includes several audio frames, and the video stream includes several video frames. The first anchor client stamps each audio frame and each video frame with a collection timestamp.

[0051] During the live broadcast with microphones, the process of the second live broadcast platform server downloading the audio and video stream to the second anchor client includes: the second live broadcast platform server is built with a first SDK and a second SDK, the first SDK is used to decode the audio and video stream, obtain the decoded audio frame and video frame, the first SDK sends the decoded audio frame and video frame to the second SDK, the second SDK re-encodes the decoded audio frame and video frame according to the preset encoding protocol, obtains the encoded audio and video stream, and then sends the encoded audio and video stream to the second anchor client. Among them, the first SDK can be an SDK that is compatible with the first anchor client, and is used to decode the audio and video stream pushed by the first live broadcast platform server. The second SDK can be an SDK that is compatible with the second anchor client, and is used to re-encode the audio and video stream decoded by the first SDK for playback by the second anchor client.

[0052] After receiving the audio and video stream sent by the second live broadcast platform server, the second anchor client needs to play the audio and video stream in real time. In order to ensure low latency and smooth playback of the connected audio and video stream, the second anchor client will preset a small cache space. Specifically, the second live broadcast platform server sends the audio video stream to the cache space of the second anchor client. The cache space can cache a certain number of audio frames and video frames. After the cache space is filled with audio frames and video frames, the audio frames and video frames are played frame by frame in the order of arrival time.

[0053] Among them, the abnormal events of real-time audio and video stream playback with microphones include the occurrence of audio and video stream playback asynchronization, jamming and frame loss when the audio and video stream is played during the microphone live broadcast process. The reasons for the asynchronization of audio and video stream playback include the inconsistency of the acquisition timestamps of the audio frame and the video frame, and the inconsistency of the time intervals of the audio frame and the video frame arriving at the cache space of the second anchor client. For example, the cache space can cache 10 audio frames and 10 video frames. The difference between the time when a certain video frame arrives at the cache space and the time when the previous video frame arrives at the cache space is greater than the sum of the time intervals of 10 video frames, and the time intervals of each audio frame arriving at the cache space are the same. When the second anchor client plays the audio frame, the video frame corresponding to a certain audio frame will be displayed lagging, resulting in asynchronization of audio and video playback. The reasons for the jamming of audio and video stream playback include the interruption of audio and video stream, the loss of audio and video frames, the disorder of the acquisition timestamps of audio and video frames, and the inconsistency of the time intervals of the audio and video frames arriving at the cache space of the second anchor client. The reasons for the frame loss of audio and video streams include the loss of audio and video frames when the audio and video frames arrive at the second live broadcast platform server and the loss of audio and video frames when the audio and video frames arrive at the cache space of the second anchor client.

[0054] In an embodiment of the present application, the second anchor client can detect whether there is an abnormality in the playback of the real-time connected microphone audio and video stream. If a playback abnormality is detected, the real-time connected microphone audio and video stream playback abnormality event information is generated. Specifically, when the second anchor client plays the real-time connected microphone audio and video stream, it will perform timed or periodic sampling on the real-time connected microphone audio and video stream to generate sampling log data. Among them, the sampling log data includes the acquisition timestamp of each audio frame or video frame in the real-time connected microphone audio and video stream. When the playback of the real-time connected microphone audio and video stream is detected to be abnormal, a moment can be selected within a preset time period before the abnormal playback moment as the start time of the real-time connected microphone audio and video stream playback abnormality event, and a moment can be selected within a preset time period after the abnormal playback moment as the end time of the real-time connected microphone audio and video stream playback abnormality event. For example, if the abnormal playback moment is 12:00:00, 11:58:00 can be used as the start time and 12:03:00 as the end time.

[0055] S20: Obtain the live broadcast playback video of the second anchor client; wherein the live broadcast playback video is a video obtained after the second anchor client receives and stores the audio and video stream.

[0056] In the embodiment of the present application, the second anchor client is preset with an offline cache space, which can store a large number of audio frames and video frames. After the audio and video streams sent by the second live broadcast platform server arrive at the offline cache space, the second anchor client aligns and synthesizes each audio frame and each video frame frame by frame according to the time sequence of arrival at the offline cache space, obtains the live broadcast playback video with microphone connection, and obtains the live broadcast playback video with microphone connection from the second anchor client.

[0057] In an optional embodiment, a real-time audio and video service network RTN may be provided between SDK2 and the second anchor client, and the audio and video stream sent by SDK2 may be recorded through the audio and video service network RTN to obtain the live broadcast playback video with microphones connected, and the live broadcast playback video with microphones connected may be obtained from the audio and video service network RTN.

[0058] S30: If there is no abnormality in the playback of the live broadcast video, determine a number of audio frame information and a number of video frame information corresponding to the start time and the end time according to the audio and video file obtained from the second live broadcast platform server; wherein the audio and video file is a file obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server.

[0059] The audio and video file is a file saved after the second live broadcast platform server decodes the audio and video stream. The audio and video stream includes audio and video frames, private coding protocols of audio and video frames, and private data of audio and video frames. The private coding protocols include but are not limited to H264 and H265 coding protocols, and the private data includes but is not limited to the acquisition timestamp of the audio and video frames, the audio and video frame coding type, the frame header information of the audio and video frames, the frame data length of the audio and video frames, and the frame data content.

[0060] In an embodiment of the present application, if the audio and video playback in the live broadcast playback video is not synchronized, it means that the acquisition timestamps of the audio frame and the video frame are inconsistent, and it is necessary to further determine the link that produces the inconsistent acquisition timestamps of the audio frame and the video frame. It may be that the first host client stamps the acquisition timestamp for each audio frame and each video frame, and the inconsistency may occur, or it may be that the second SDK of the second live broadcast platform server rebuilds the acquisition timestamp for each audio frame and each video frame. If the audio and video playback in the live broadcast playback video is synchronized, it means that the acquisition timestamps of the audio frame and the video frame are consistent. It is necessary to further determine the link that produces the inconsistent time intervals of the audio frame and the video frame arriving at the cache space of the second host client. It may be that the time intervals of the audio frame and the video frame arriving at the first SDK are inconsistent, or it may be that the time intervals of the audio frame and the video frame arriving at the first SDK are consistent. In the process of the second SDK sending the audio frame and the video frame to the second host client, due to the time-consuming processing or link transmission, the time intervals of the audio and video frames to the cache space of the second host client are different.

[0061] If the playback anomaly is a freeze in the audio and video playback, it is necessary to determine whether the cause of the playback anomaly is an audio and video disconnection, audio and video frame loss, disordered audio and video acquisition timestamps, and inconsistent time intervals for the audio and video frames to arrive at the small cache space of the second anchor client. If there is a freeze in the live broadcast playback video playback, use the existing technology to troubleshoot the cause from the audio and video disconnection, audio and video frame loss, and disordered audio and video acquisition timestamps. If there is no freeze in the live broadcast playback video playback, it means that the time intervals for the audio and video frames to arrive at the cache space of the second anchor client are inconsistent. It is necessary to further determine the link that causes the inconsistent time intervals for the audio and video frames to arrive at the cache space of the second anchor client. It may be that the time intervals for the audio and video frames to arrive at the first SDK are inconsistent, or it may be that the time intervals for the audio and video frames to arrive at the first SDK are consistent. In the process of the second SDK sending audio frames and video frames to the second anchor client, due to the time-consuming processing or link transmission, the time intervals for the audio and video frames to the cache space of the second anchor client are different.

[0062] If the playback abnormality is frame loss in audio and video, it is necessary to determine whether the cause of the playback abnormality is the real-time network fluctuation during the connection process of the second anchor client, frame loss when the audio and video frames arrive at the second live broadcast platform server, or frame loss when the audio and video frames are sent from the second live broadcast platform server to the second anchor client.

[0063] The audio frame information and video frame information corresponding to the duration between the start time and the end time can be intercepted from the audio and video file, the audio frame information includes each audio frame in the duration and the time when each audio frame actually arrives at the second live broadcast platform server, and the video frame information includes each video frame in the duration and the time when each video frame actually arrives at the second live broadcast platform server. Specifically, the time when the audio frame actually arrives at the second live broadcast platform server can be the time when the audio frame is decoded, and the time when the video frame actually arrives at the second live broadcast platform server can be the time when the video frame is decoded.

[0064] S40: obtaining a video in a preset format according to a plurality of audio frame information and a plurality of video frame information; if the video in the preset format is synchronized with the video screen of the live broadcast playback video with microphone connection, parsing the video in the preset format.

[0065] The video in the preset format may be a video that can be opened by a general playback tool or playback software, including but not limited to MP4, AVI, and WMV.

[0066] In the embodiment of the present application, the audio frame information and video frame information corresponding to the duration between the start time and the end time can be synthesized into MP4. Compare whether the video images of the corresponding duration in the MP4 and the live broadcast playback video are consistent. If they are consistent, parse the MP4 to obtain the audio frame information and the video frame information. According to the decoding timestamps of the audio frames and the video frames in the audio frame information and the video frame information, the link where the playback abnormality occurs can be determined.

[0067] If there is inconsistency, it means that there are frame losses in the video images of the corresponding length in the live broadcast playback video, and the cause of the frame losses needs to be further investigated. Specifically, according to the audio frame information and the acquisition timestamps of the audio frames and video frames in the video frame information, if the time interval between the acquisition timestamps of adjacent audio frames and adjacent video frames is fixed, it means that there is no frame loss when the audio and video frames arrive at the second live broadcast platform server, and frame loss occurs when the audio and video frames are sent from the second live broadcast platform server to the second anchor client. If the time interval between the acquisition timestamps of adjacent audio frames or adjacent video frames is not fixed, it means that there is frame loss when the audio and video frames arrive at the second live broadcast platform server.

[0068] By applying the embodiment of the present application, the real-time abnormal event information of the playback of the microphone-connected audio and video stream of the second anchor client during the microphone-connected live broadcast process is obtained; wherein the real-time abnormal event information of the playback of the microphone-connected audio and video stream includes the start time and the end time of the abnormal event of the playback of the real-time microphone-connected audio and video stream; wherein the real-time microphone-connected audio and video stream is the audio and video stream received by the second anchor client from the second live broadcast platform server in real time; the microphone-connected live broadcast playback video of the second anchor client is obtained; wherein the microphone-connected live broadcast playback video is the video obtained after the second anchor client receives and stores the audio and video stream; if there is no abnormality in the playback of the microphone-connected live broadcast video, then according to the audio and video file obtained from the second live broadcast platform server, a number of audio frame information and a number of video frame information corresponding to the start time and the end time are determined; wherein the audio and video file is the file obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server; according to the number of audio frame information and the number of video frame information, a video in a preset format is obtained; if the video in the preset format is synchronized with the video screen of the microphone-connected live broadcast playback video, the video in the preset format is parsed. The present application saves the audio and video files on the second live broadcast platform server, extracts the audio frame information and the video frame information from the audio and video files according to the abnormal event information of the real-time audio and video stream playback with microphones, obtains the video in a preset format according to the audio frame information and the video frame information, and parses the video in the preset format. The cause of the abnormal event of the real-time audio and video stream playback with microphones is automatically and quickly determined without human intervention, thereby improving the efficiency of determining the cause of the abnormal event of the audio and video stream playback.

[0069] In one embodiment, the audio and video file includes an audio file and a video file. Before the step of determining a plurality of audio frame information and a plurality of video frame information corresponding to the start time and the end time in the audio and video file obtained from the second live broadcast platform server in step S30, steps S301 to S302 are further included, which are specifically as follows:

[0070] S301: The second live broadcast platform server decodes the audio and video streams to obtain a number of audio frames, a number of video frames, an acquisition timestamp of each audio frame, and an acquisition timestamp of each video frame, and generates a decoding timestamp of each audio frame and a decoding timestamp of each video frame.

[0071] In an embodiment of the present application, after the first SDK of the second live broadcast platform server receives the audio and video stream pushed by the first live broadcast platform server, it decodes the audio and video stream to obtain a number of audio frames, a number of video frames, the acquisition timestamp of each audio frame, the acquisition timestamp of each video frame, the encoding type of each audio frame, the frame header information, the frame data length and the frame data content, the encoding type, the frame header information, the frame data length and the frame data content of each video frame, and generates a frame number and a decoding timestamp for each audio frame and video frame. Among them, the frame number is used to uniquely identify the audio and video frame, the decoding timestamp of each audio frame can be regarded as the actual time when each audio frame arrives at the second live broadcast platform server, and the decoding timestamp of each video frame can be regarded as the actual time when each video frame arrives at the second live broadcast platform server.

[0072] S302: The second live broadcast platform server saves a number of audio frames, the acquisition timestamp of each audio frame, and the decoding timestamp of each audio frame as an audio file, and saves a number of video frames, the acquisition timestamp of each video frame, and the decoding timestamp of each video frame as a video file.

[0073] In an embodiment of the present application, the audio file can be written in the order of the acquisition timestamp, decoding timestamp, frame sequence number, encoding type, frame header information, frame data length and frame data content of each audio frame. The video file can be written in the order of the acquisition timestamp, decoding timestamp, frame sequence number, encoding type, frame header information, frame data length and frame data content of each video frame. The second live broadcast platform server saves the audio file and the video file in the storage space. By decoding the audio and video streams, the audio and video files can be automatically and quickly obtained.

[0074] In one embodiment, the step of determining a plurality of audio frame information and a plurality of video frame information corresponding to the start time and the end time in the audio and video file obtained from the second live broadcast platform server in step S30 includes steps S311 to S312, which are specifically as follows:

[0075] S311: matching a first audio frame whose time indicated by a capture timestamp is consistent with a start time from the audio file, matching a second audio frame whose time indicated by a capture timestamp is consistent with an end time, and taking the first audio frame, the second audio frame, and all audio frames between the first audio frame and the second audio frame as a number of audio frames corresponding to a time range from the start time to the end time; obtaining a capture timestamp and a decoding timestamp of a number of audio frames from the audio file, and determining the number of audio frames and the capture timestamp and the decoding timestamp of each audio frame as a number of audio frame information;

[0076] S312: Match the first video frame whose moment indicated by the acquisition timestamp is consistent with the start time from the video file, match the second video frame whose moment indicated by the acquisition timestamp is consistent with the end time, and use the first video frame, the second video frame, and all video frames between the first video frame and the second video frame as a number of video frames corresponding to the time range from the start time to the end time; obtain the acquisition timestamps and decoding timestamps of the several video frames from the video file, and determine the several video frames and the acquisition timestamp and decoding timestamp of each video frame as a number of video frame information.

[0077] In an embodiment of the present application, audio and video files can be obtained from the storage space of the second live broadcast platform server, and audio files and video files within the time range between the start time and the end time can be extracted from the audio and video files. Specifically, by matching the start time, the end time and the acquisition timestamp in the audio and video files, the first audio frame and the first video frame corresponding to the start time, and the second audio frame and the second video frame corresponding to the end time are determined. After determining the first audio frame and the second audio frame, all audio frames between the first audio frame and the second audio frame are traversed from the audio file according to the frame sequence number, and all video frames between the first video frame and the second video frame are traversed from the video file. Then, the acquisition timestamp and decoding timestamp corresponding to each audio frame and video frame are obtained.

[0078] By matching the start time and the end time with the acquisition timestamp in the audio and video files respectively, a number of audio frame information and a number of video frame information corresponding to the start time and the end time can be automatically and quickly obtained.

[0079] In one embodiment, the step of obtaining a video in a preset format according to a plurality of audio frame information and a plurality of video frame information in step S40 includes steps S401 to S403, which are specifically as follows:

[0080] S401: inserting the acquisition timestamp and the decoding timestamp of each audio frame into the supplementary enhancement information of the corresponding audio frame to obtain a number of target audio frames;

[0081] S402: inserting the acquisition timestamp and the decoding timestamp of each video frame into the supplementary enhancement information of the corresponding video frame to obtain a plurality of target video frames;

[0082] S403: synthesize a plurality of target audio frames and a plurality of target video frames into a video in a preset format.

[0083] Among them, Supplemental Enhancement Information (SEI) provides a method for adding information to audio bitstreams and video bitstreams.

[0084] In the embodiment of the present application, the preset video format is mp4, and the supplementary enhancement information is used as a field in the frame header of the audio frame and the video frame, and the ffmpeg interface is called in sequence to write each target audio frame and each target video frame into mp4. By writing the acquisition timestamp and the decoding timestamp into the supplementary enhancement information, the acquisition timestamp and the decoding timestamp can be saved in mp4, which is convenient for determining the cause of the subsequent abnormal event of the real-time audio and video stream playback.

[0085] In one embodiment, after step S40, steps S411 to S414 are included, which are specifically as follows:

[0086] S411: Obtain a decoding timestamp of each video frame and a decoding timestamp of each audio frame; wherein each video frame and each audio frame are obtained by parsing a video in a preset format;

[0087] S412: Calculate a first time difference between moments indicated by decoding timestamps of adjacent audio frames, and calculate a second time difference between moments indicated by decoding timestamps of adjacent video frames;

[0088] S413: If at least one first time difference is greater than a preset threshold, and / or at least one second time difference is greater than a preset threshold, determining that the cause of the abnormal playback event of the real-time microphone-connected audio and video stream is that the time when the audio stream is transmitted from the first live broadcast platform server to the second live broadcast platform server is not synchronized;

[0089] S414: If the first time difference is less than or equal to the preset threshold, and the second time difference is less than or equal to the preset threshold, it is determined that the cause of the abnormal playback event of the real-time microphone audio and video stream is that the time when the audio stream is sent from the second live broadcast platform server to the second anchor client is not synchronized.

[0090] In the embodiment of the present application, a multimedia analysis tool can be used to parse a video in a preset format. Since the video in the preset format carries supplementary enhancement information, the decoding timestamps of the audio and video frames can be obtained by parsing the video in the preset format. By calculating the time intervals between the decoding timestamps of adjacent audio frames and adjacent video frames, the cause of the abnormal playback event of the real-time microphone audio and video stream can be determined.

[0091] Specifically, if the first time difference is less than or equal to the preset threshold, and the second time difference is less than or equal to the preset threshold, it means that the time intervals of each audio frame and each video frame transmitted from the first live broadcast platform server to the first SDK of the second live broadcast platform server are regular, and the time intervals of each audio frame and each video frame sent from the second SDK of the second live broadcast platform server to the cache space of the second anchor client are no longer regular. It may be that in the process of the second SDK sending audio frames and video frames to the second anchor client, due to the time-consuming processing or link transmission, the time intervals of the audio and video frames to the cache space of the second anchor client are different. If at least one first time difference is greater than the preset threshold, and / or at least one second time difference is greater than the preset threshold, it means that the time intervals of each audio frame and each video frame transmitted from the first live broadcast platform server to the first SDK of the second live broadcast platform server are irregular. It may be that there are network fluctuations in the process of the first live broadcast platform server transmitting the audio and video streams to the second live broadcast platform server.

[0092] By comparing the time interval between the decoding timestamps of adjacent audio frames or adjacent video frames with a preset threshold, the cause of the abnormal playback event of the real-time co-hosted audio and video stream can be automatically and quickly determined.

[0093] See also Figure 3 , is a schematic diagram of the structure of a cross-platform live broadcast and microphone audio and video stream processing device provided by an embodiment of the present application. The device can be implemented as all or part of a server through software, hardware, or a combination of both. The device 5 includes:

[0094] The event information acquisition module 51 is used to obtain the abnormal event information of the real-time audio and video stream playback of the second anchor client during the live broadcast process; wherein the abnormal event information of the real-time audio and video stream playback of the second anchor client includes the start time and end time of the abnormal event of the real-time audio and video stream playback of the second anchor client; wherein the real-time audio and video stream of the second anchor client is the audio and video stream received in real time by the second anchor client from the second live broadcast platform server;

[0095] The playback video acquisition module 52 is used to acquire the live broadcast playback video of the second anchor client; wherein the live broadcast playback video of the second anchor client is a video obtained after receiving and storing the audio and video stream;

[0096] The frame information determination module 53 is used to determine a number of audio frame information and a number of video frame information corresponding to the start time and the end time according to the audio and video files obtained from the second live broadcast platform server if there is no abnormality in the playback video of the live broadcast with microphones connected. The audio and video files are files obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server.

[0097] The video parsing module 54 is used to obtain a video in a preset format according to a plurality of audio frame information and a plurality of video frame information; if the video in the preset format is synchronized with the video screen of the live broadcast playback video with microphone connection, parse the video in the preset format.

[0098] It should be noted that the audio and video stream processing device for cross-platform live broadcast and microphone connection provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the audio and video stream processing method for cross-platform live broadcast and microphone connection. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio and video stream processing device for cross-platform live broadcast and microphone connection provided in the above embodiment and the audio and video stream processing method for cross-platform live broadcast and microphone connection belong to the same concept. The implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0099] See also Figure 4 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the computer device 21 may include: a processor 210, a memory 211, and a computer program 212 stored in the memory 211 and executable on the processor 210, for example: an audio and video stream processing program for cross-platform live broadcast and microphone connection; when the processor 210 executes the computer program 212, the steps in the above-mentioned embodiments are implemented.

[0100] Among them, the processor 210 may include one or more processing cores. The processor 210 uses various interfaces and lines to connect various parts in the computer device 21, and executes various functions and processes data of the computer device 21 by running or executing instructions, programs, code sets or instruction sets stored in the memory 211, and calling the data in the memory 211. Optionally, the processor 210 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programble Logic Array, PLA). The processor 210 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 210, and it can be implemented by a single chip.

[0101] Among them, the memory 211 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 211 includes a non-transitory computer-readable storage medium. The memory 211 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 211 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 211 may also be optionally at least one storage device located away from the aforementioned processor 210.

[0102] The embodiment of the present application also provides a computer storage medium, which can store multiple instructions. The instructions are suitable for being loaded by a processor and executing the method steps of the above embodiment. The specific execution process can be found in the specific description of the above embodiment, which will not be repeated here.

[0103] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0104] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0105] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0106] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc.

[0110] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

Claims

1. A method for processing audio and video streams of a cross-platform live broadcast and microphone connection, applied to a cross-platform live broadcast and microphone connection system, wherein the cross-platform live broadcast and microphone connection system comprises a first live broadcast platform server and a second live broadcast platform server, wherein the first live broadcast platform server and the second live broadcast platform server are connected in communication, wherein the first live broadcast platform server pushes the audio and video stream of the first anchor client to the second live broadcast platform server, and the second live broadcast platform server sends the audio and video stream to the second anchor client; characterized in that: The steps include: Acquire the real-time audio and video stream playback abnormal event information of the second anchor client during the live broadcast process; wherein the real-time audio and video stream playback abnormal event information includes the start time and end time of the real-time audio and video stream playback abnormal event; wherein the real-time audio and video stream is the audio and video stream received in real time by the second anchor client from the second live broadcast platform server; Obtaining a live broadcast playback video of the second anchor client; wherein the live broadcast playback video is a video obtained after the second anchor client receives and stores the audio and video stream; If there is no abnormality in the playback of the live broadcast with microphones, determine a number of audio frame information and a number of video frame information corresponding to the start time and the end time according to the audio and video files obtained from the second live broadcast platform server; wherein the audio and video files are files obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server; A video in a preset format is obtained according to the plurality of audio frame information and the plurality of video frame information; if the video in the preset format is synchronized with the video screen of the live broadcast playback video with microphones connected, the video in the preset format is parsed.

2. The method for processing audio and video streams for cross-platform live broadcast and microphone connection according to claim 1, characterized in that: The audio and video files include audio files and video files; Before the step of determining a plurality of audio frame information and a plurality of video frame information corresponding to the start time and the end time according to the audio and video file obtained from the second live broadcast platform server, the step includes: The second live broadcast platform server decodes the audio and video streams to obtain a plurality of audio frames, a plurality of video frames, an acquisition timestamp of each of the audio frames and an acquisition timestamp of each of the video frames, and generates a decoding timestamp of each of the audio frames and a decoding timestamp of each of the video frames; The second live broadcast platform server saves the plurality of audio frames, the acquisition timestamp of each audio frame, and the decoding timestamp of each audio frame as the audio file, and saves the plurality of video frames, the acquisition timestamp of each video frame, and the decoding timestamp of each video frame as the video file.

3. The method for processing audio and video streams for cross-platform live broadcast and microphone connection according to claim 2, characterized in that: The step of determining a plurality of audio frame information and a plurality of video frame information corresponding to the start time and the end time according to the audio and video file obtained from the second live broadcast platform server includes: Matching a first audio frame whose time indicated by a collection timestamp is consistent with the start time from the audio file, matching a second audio frame whose time indicated by a collection timestamp is consistent with the end time, taking the first audio frame, the second audio frame, and all audio frames between the first audio frame and the second audio frame as a number of audio frames corresponding to a time range from the start time to the end time; obtaining collection timestamps and decoding timestamps of the several audio frames from the audio file, and determining the several audio frames and the collection timestamp and decoding timestamp of each of the audio frames as the several audio frame information; Match the first video frame whose moment indicated by the acquisition timestamp is consistent with the start time from the video file, match the second video frame whose moment indicated by the acquisition timestamp is consistent with the end time, and use the first video frame, the second video frame, and all video frames between the first video frame and the second video frame as a number of video frames corresponding to the time range from the start time to the end time; obtain the acquisition timestamps and decoding timestamps of the several video frames from the video file, and determine the several video frames and the acquisition timestamp and decoding timestamp of each of the video frames as the several video frame information.

4. The method for processing audio and video streams for cross-platform live broadcast and microphone connection according to claim 2, characterized in that: The step of obtaining a video in a preset format according to the plurality of audio frame information and the plurality of video frame information comprises: Inserting the acquisition timestamp and the decoding timestamp of each audio frame into the supplementary enhancement information of the corresponding audio frame to obtain a plurality of target audio frames; Inserting the acquisition timestamp and the decoding timestamp of each of the video frames into the supplementary enhancement information of the corresponding video frame to obtain a plurality of target video frames; The plurality of target audio frames and the plurality of target video frames are synthesized into a video in a preset format.

5. The method for processing audio and video streams of cross-platform live broadcast and microphone connection according to claim 2, characterized in that: After the step of parsing the video in the preset format, the method further comprises: Obtaining a decoding timestamp of each video frame and a decoding timestamp of each audio frame; wherein each of the video frames and each of the audio frames are obtained by parsing the video in the preset format; Calculating a first time difference between moments indicated by decoding timestamps of adjacent audio frames, and calculating a second time difference between moments indicated by decoding timestamps of adjacent video frames; If at least one of the first time differences is greater than a preset threshold, and / or at least one of the second time differences is greater than the preset threshold, it is determined that the cause of the abnormal playback event of the real-time microphone-connected audio and video stream is that the time when the audio stream is transmitted from the first live broadcast platform server to the second live broadcast platform server is not synchronized; If the first time difference is less than or equal to the preset threshold, and the second time difference is less than or equal to the preset threshold, it is determined that the cause of the abnormal playback event of the real-time microphone audio and video stream is that the time when the audio stream is sent from the second live broadcast platform server to the second anchor client is not synchronized.

6. A cross-platform live broadcast and microphone audio and video stream processing device, characterized in that: include: An event information acquisition module is used to acquire abnormal event information of real-time audio and video stream playback of the second anchor client during the live broadcast of the live broadcast; wherein the abnormal event information of real-time audio and video stream playback of the second anchor client includes the start time and end time of the abnormal event of real-time audio and video stream playback of the second anchor client; wherein the real-time audio and video stream of the second anchor client is the audio and video stream received in real time by the second anchor client from the second live broadcast platform server; A playback video acquisition module, used to acquire the live broadcast playback video of the second anchor client; wherein the live broadcast playback video is a video obtained after the second anchor client receives and stores the audio and video stream; A frame information determination module, configured to determine, if the playback video of the live broadcast with microphones is normal, a number of audio frame information and a number of video frame information corresponding to the start time and the end time according to the audio and video files obtained from the second live broadcast platform server; wherein the audio and video files are files obtained by the second live broadcast platform server receiving and decoding the audio and video stream pushed by the first live broadcast platform server; The video parsing module is used to obtain a video in a preset format according to a plurality of audio frame information and a plurality of video frame information; if the video in the preset format is synchronized with the video screen of the live broadcast playback video with microphone connection, parse the video in the preset format.

7. An electronic device comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images