Synchronous playing method and apparatus for multiple channels of audio / videos, and electronic device and reservoir medium

Abstract

Provided in the present application are a synchronous playing method and system for multiple channels of audio / videos, and an electronic device and a reservoir medium. The method comprises: acquiring an audio / video synchronous playing request, wherein the audio / video synchronous playing request includes an identifier of an audio / video collection device; on the basis of the identifier of the audio / video collection device, pulling multiple channels of audio / video data from a server side; and calling a StreamaxPlayer library to respectively decode and render the multiple channels of audio / video data according to a playing time, so as to synchronously play, at the playing time, data content which is in each channel of audio / video data and the timestamp of which is consistent with the playing time. On the basis of the method, decoding and rendering of each channel of audio / video data can be controlled according to a time reference at a browser side, such that data content which is in channels of audio / video data and which is generated at the same time is played at the same time, thereby ensuring the synchronization and consistency of playing of multiple channels of audio / video data.

Description

Multi-channel synchronous playback method and device for audio and video, electronic equipment and storage medium Technical Field

[0001] The present application relates to the field of audio and video playback technology, and in particular to a method, system, electronic device, and storage medium for multi-channel synchronous playback of audio and video. Background Art

[0002] With the development of network technology, the demand for playing audio and video on the WEB side is increasing. Existing in-vehicle video surveillance systems usually use multiple monitoring devices to achieve comprehensive monitoring. In order to conveniently view events that occurred before and after a certain time, when viewing the monitoring screen, users need to synchronously play videos of multiple channels through the WEB side to observe the images captured by multiple monitoring devices at the same time. In order to ensure the synchronization of multi-channel audio and video playback, the existing synchronous playback method basically ensures that multiple links push audio and video data at the same time to the client for playback through the server. This method can only ensure that the audio and video data are synchronized when pushed out by the server. In actual situations, the audio and video data played out are prone to time differences due to network fluctuations, decoding and rendering speeds, etc., making it difficult to ensure multi-channel synchronous playback and unable to achieve time consistency of multiple playback windows. Technical issues

[0003] In view of this, the embodiments of the present application provide a method, system, electronic device and storage medium for multi-channel synchronous playback of audio and video, which can control the decoding and rendering of audio and video data of each channel according to the time base on the browser side to ensure the synchronous consistency of the playback of multi-channel audio and video data. Technical Solutions

[0004] A first aspect of an embodiment of the present application provides a method for multi-channel synchronous playback of audio and video, which is applied on a browser side and includes: obtaining an audio and video synchronous playback request, wherein the audio and video synchronous playback request includes an identifier of an audio and video acquisition device; pulling multiple channels of audio and video data from a server side according to the identifier of the audio and video acquisition device; calling a StreamaxPlayer library to decode and render the audio and video data of the multiple channels respectively according to the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

[0005] In one possible implementation, the step of calling the StreamaxPlayer library to decode the multiple channels of audio and video data according to the playback time, so as to synchronously play the data content in each channel of audio and video data whose timestamp is consistent with the playback time at the playback time, includes: calling the StreamaxPlayer.wasm library to decapsulate the audio and video data of each channel to obtain the timestamp information of the audio and video data of each channel; calling the StreamaxPlayer.js library to decode and render the data content in the audio and video data of each channel whose timestamp is consistent with the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

[0006] In one possible implementation, before the step of calling the StreamaxPlayer.js library to decode and render the data content in the audio and video data of each channel whose timestamp is consistent with the playback time, it also includes: determining a reference channel according to the channel identifier, and using the start time of loading the audio and video data of the reference channel as the start playback time of the multi-channel synchronous playback of audio and video, and the start playback time is the start time of the browser end decoding and rendering the audio and video data of each channel; detecting whether other channels are loaded with audio and video data, if there are channels in the other channels that are not loaded with audio and video data, adjusting the start playback time of the multi-channel synchronous playback of audio and video according to the preset delay value, and the other channels are all channels configured by the browser end except the reference channel.

[0007] In one possible implementation, the step of calling the StreamaxPlayer.js library to decode and render the data content in the audio and video data of each channel with a timestamp consistent with the playback time, so as to synchronously play the data content in the audio and video data of each channel with a timestamp consistent with the playback time at the playback time, also includes: detecting whether all channels have loaded audio and video data at the playback time, if there is any channel that has no audio and video data loaded at the playback time, then ignoring the synchronous playback of audio and video data for the arbitrary channel at the playback time, the arbitrary channel being any one or more channels among all the channels.

[0008] A second aspect of an embodiment of the present application provides a multi-channel synchronous playback method for audio and video, which is applied on a server side and includes: receiving multi-channel audio and video data uploaded by an audio and video acquisition device; unpacking the multi-channel audio and video data to obtain original audio and video data of multiple channels; for the original audio and video data of each channel, adding extended data to the I-frame audio and video data of the original audio and video data, and encapsulating the added extended data with the I-frame audio and video data to obtain audio and video data of multiple channels; using multiple links to transmit the audio and video data of the multiple channels to a browser side, wherein the multiple links correspond one-to-one to the audio and video data of the multiple channels.

[0009] In one possible implementation, the step of adding extended data to the I-frame audio and video data of the original audio and video data includes: for each I-frame audio and video data in the original audio and video data, adding an extension unit to the I-frame audio and video data, and adding the PTS timestamp and UTC timestamp corresponding to the I-frame audio and video data in the extension unit.

[0010] A third aspect of an embodiment of the present application provides a multi-channel synchronous playback device for audio and video, and the method is applied on the browser side, including: an acquisition module, used to obtain an audio and video synchronous playback request, wherein the audio and video synchronous playback request contains an identifier of an audio and video acquisition device; a pulling module, used to pull multiple channels of audio and video data from a server side according to the identifier of the audio and video acquisition device; a playback module, used to call a StreamaxPlayer library to decode and render the audio and video data of the multiple channels respectively according to the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

[0011] The fourth aspect of the embodiments of the present application provides a multi-channel synchronous playback device for audio and video, and the method is applied on the server side, including: a receiving module for receiving multi-channel audio and video data uploaded by an audio and video acquisition device; an unpacking module for unpacking the multi-channel audio and video data to obtain original audio and video data of multiple channels; an adding module for adding extended data to the I-frame audio and video data of the original audio and video data for each channel, and encapsulating the added extended data with the I-frame audio and video data to obtain audio and video data of multiple channels; a transmission module for using multiple links to transmit the audio and video data of the multiple channels to the browser side, wherein the multiple links correspond one-to-one to the audio and video data of the multiple channels.

[0012] The fifth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in any one of the first aspect or the second aspect are implemented.

[0013] A sixth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method described in any one of the first aspect or the second aspect.

[0014] A seventh aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device implements the steps of the method described in any one of the first aspect or the second aspect when executing the computer program product. Beneficial effects

[0015] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0016] This application obtains a request for synchronous audio and video playback, and according to the identifier of the audio and video acquisition device contained in the request, pulls the audio and video data of multiple channels from the server side, and then decodes and renders the audio and video data of the multiple channels according to the playback time by calling the StreamaxPlayer library, so as to synchronously play the data content in the audio and video data of each channel with the same timestamp and playback time at the playback time. Based on this method, it is possible to control the decoding and rendering of the audio and video data of each channel according to the time reference on the browser side, and simultaneously play the data content generated at the same time in the audio and video data of each channel, thereby ensuring the synchronization consistency of the playback of multi-channel audio and video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] FIG1 is a flowchart illustrating a method for implementing a multi-channel synchronous playback method of audio and video according to an embodiment of the present application;

[0019] FIG2 is a flowchart illustrating a method for synchronously playing audio and video in a multi-channel synchronous playback method of audio and video provided in an embodiment of the present application;

[0020] FIG3 is a flowchart illustrating a method for determining a start playback time in a multi-channel synchronous playback method for audio and video provided in an embodiment of the present application;

[0021] FIG4 is a flowchart illustrating another method for implementing a multi-channel synchronous playback method of audio and video according to an embodiment of the present application;

[0022] FIG5 is a basic structural block diagram of a multi-channel synchronous playback device for audio and video provided in an embodiment of the present application;

[0023] FIG6 is a basic structural block diagram of another apparatus for synchronous multi-channel playback of audio and video provided in an embodiment of the present application;

[0024] FIG7 is a basic structural block diagram of an electronic device provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0025] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0027] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0029] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. "Multiple" means "two or more."

[0031] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0032] In some embodiments of the present application, please refer to Figure 1, which is a method implementation flow chart of a multi-channel synchronous playback method for audio and video provided in an embodiment of the present application, which may specifically include steps S11 to S13.

[0033] S11: Obtain an audio and video synchronous playback request, where the audio and video synchronous playback request includes an identifier of an audio and video acquisition device.

[0034] In this embodiment, the browser serves as the client for audio and video playback. Users can input audio and video playback operations on the browser's interactive page. Based on the user's input, the browser receives a request for synchronized audio and video playback. In this embodiment, the request for synchronized audio and video playback includes an identifier for an audio and video capture device, which is used to locate the audio and video capture device. The audio and video capture device can be a microphone, camera, or other device. The identifier for the audio and video capture device can be a unique identifier representing the audio and video capture device, such as an address or device number.

[0035] S12: Pulling multiple channels of audio and video data from the server according to the identifier of the audio and video acquisition device.

[0036] In this embodiment, after receiving a request for synchronized audio and video playback, the browser forwards the request to the server. Based on the identification of the audio and video capture device, the server is triggered to establish a TCP (Transmission Control Protocol) communication link with the audio and video capture device. This triggers the server to send audio and video signaling to the corresponding audio and video capture device, causing the audio and video capture device to upload the real-time audio and video data to the server. The server then transmits the audio and video data from the multiple channels collected by the audio and video capture device to the browser via multiple links, with each link transmitting audio and video data for one channel.

[0037] S13: calling the StreamaxPlayer library to decode and render the audio and video data of the multiple channels respectively according to the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

[0038] In this embodiment, on the browser side, the audio and video data of the obtained multiple channels are time-synchronized, and then the audio and video are decoded and played using the MSE method. Among them, MSE (Media Source Extensions) refers to a new Web interface supported by browsers such as Chrome, Safari, and Edge, which allows JavaScript to play media streams. A StreamaxPlayer library is built in the browser side, and plug-in-free audio and video playback on the browser side can be achieved through the StreamaxPlayer library. In this embodiment, after the browser side obtains the audio and video data of multiple channels, it decapsulates the audio and video data of each channel by calling the StreamaxPlayer library, obtains the timestamp information of the audio and video data of each channel, and then uses the timestamp of the audio and video data of one of the channels as a reference, based on the playback timeline, and controls the decoding and rendering of the audio and video data of the remaining channels according to the playback time, so that the audio and video data of each channel are time-synchronized during playback. For example, at the playback time, the data content in the audio and video data of each channel with a timestamp consistent with the playback time can be decoded, and the data content generated at the same time in the audio and video data of each channel can be played at the same time, so that the audio and video data of each channel can be controlled on the browser side to be decoded and played instantly according to the time frame.

[0039] From the above, it can be seen that in the multi-channel synchronous playback method of audio and video provided by the embodiment of the present application, after obtaining the audio and video synchronous playback request, the browser side pulls the audio and video data of multiple channels from the server side according to the identifier of the audio and video acquisition device contained in the audio and video synchronous playback request, and then decodes and renders the audio and video data of the multiple channels according to the playback time by calling the StreamaxPlayer library, so as to synchronously play the data content in the audio and video data of each channel with the same timestamp and playback time at the playback time. Based on this method, it is possible to control the decoding and rendering of the audio and video data of each channel according to the time reference on the browser side, and simultaneously play the data content generated at the same time in the audio and video data of each channel, thereby ensuring the synchronization consistency of the playback of multi-channel audio and video data.

[0040] In some embodiments of the present application, please refer to Figure 2, which is a flowchart of a method for synchronously playing audio and video in the multi-channel synchronous playback method of audio and video provided in the embodiment of the present application. As shown in Figure 2, it can specifically include steps S21 to S22.

[0041] S21: calling the StreamaxPlayer.wasm library to decapsulate the audio and video data of each channel and obtain the timestamp information of the audio and video data of each channel;

[0042] S22: Call the StreamaxPlayer.js library to decode and render the data content in the audio and video data of each channel whose timestamp is consistent with the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

[0043] In this embodiment, the StreamaxPlayer library includes two parts: the StreamaxPlayer.wasm library and the StreamaxPlayer.js library. Among them, the StreamaxPlayer.wasm library is specifically compiled using WebAssembly technology. The browser side can decapsulate the audio and video data of each channel by calling the StreamaxPlayer.wasm library, parse out the SEI extension data of the I frame in the audio and video data of each channel, and obtain the timestamp information of the audio and video data recorded in the SEI extension data. The StreamaxPlayer.js library is written in JavaScript. The browser side can decode and render the data content whose timestamp is consistent with the playback time in the audio and video data of each channel by calling the StreamaxPlayer.js library at the playback time, thereby synchronously playing the data content whose timestamp is consistent with the playback time in the audio and video data of each channel at the playback time. Based on this embodiment, the StreamaxPlayer.js library can be used to control the remaining channels based on the timestamp of one channel, so as to control the decoding and rendering time of the audio and video data of each channel according to the playback time, thereby achieving the synchronous playback of the audio and video data of each channel, realizing hardware decoding, and supporting the browser to accelerate the decoding and rendering process through the graphics card.

[0044] In a specific embodiment, the StreamaxPlayer.wasm library includes three software modules: a video processing module, an FFmpeg processing module, and an SEI parsing module. The C / C++ programs corresponding to the three software modules are compiled into the StreamaxPlayer.wasm library using WebAssembly technology to implement business logic such as audio and video frame decapsulation and FMP4 file encapsulation. In this embodiment, the C / C++ program can be compiled into LLVM bytecode through WebAssembly technology. LLVM bytecode is a coding format that can only be understood by computers and has the characteristics of high security and fast running speed. In this embodiment, the StreamaxPlayer.wasm library specifically decapsulates the audio and video data through the audio and video processing module, parses the I frame SEI extension data in the audio and video data through the SEI parsing module, and obtains the UTC timestamp information of the audio and video data; and converts the format of the audio and video data through the FFmpeg processing module and encapsulates it into audio and video data blocks in FMP4 format. FMP4 (Fragmented MP4) is a video and audio streaming format and an extension of the MPEG-4-Part12 standard. It can divide audio and video data into several fragments. Each fragment is an MP4 file containing media data, metadata, and index information.

[0045] In a specific embodiment, the StreamaxPlayer.js library includes three hardware modules: an audio and video streaming module, an MSE decoding module, and a playback management module. The three hardware modules are written in JavaScript. JavaScript is an object-based and event-driven, relatively secure, and directly interpreted scripting language. In this embodiment, the StreamaxPlayer.js library specifically pulls the audio and video data of each channel from the server through the audio and video streaming module, and inputs the pulled audio and video data into the StreamaxPlayer.wasm library for data processing. The MSE decoding module receives the audio and video data blocks encapsulated in the FMP4 format output by the StreamaxPlayer.wasm library, and performs hardware decoding on these FMP4 format audio and video data blocks. A virtual playback timeline is provided in the playback management module. The virtual playback timeline is used to indicate the time node that currently needs to be played. Each channel calls back the time point of the current audio and video data frame in real time, so that the virtual playback timeline controls whether decoding and rendering are allowed. The playback management module controls the MSE decoding module to decode and render the audio and video data of each channel according to the playback time displayed on the virtual playback timeline, thereby achieving synchronous playback of audio and video data of multiple channels.

[0046] Based on the above embodiment, hardware decoding and rendering can be implemented on the browser side by calling the StreamaxPlayer library. When the browser detects that the CPU occupancy rate is high, the browser is supported to accelerate the decoding and rendering process through the graphics card.

[0047] In some embodiments of the present application, please refer to Figure 3, which is a flowchart of a method for determining the start playback time in the multi-channel synchronous playback method of audio and video provided in the embodiment of the present application. As shown in Figure 3, it can specifically include steps S31 to S32.

[0048] S31: Determine a reference channel according to the channel identifier, and use the start time of loading the audio and video data of the reference channel as the start time of multi-channel synchronous playback of audio and video, where the start time is the start time of decoding and rendering the audio and video data of each channel by the browser end;

[0049] S32: Detect whether other channels are loaded with audio and video data. If there are channels in the other channels that are not loaded with audio and video data, adjust the start playback time of the multi-channel synchronous playback of the audio and video according to the preset delay value. The other channels are all channels configured on the browser side except the reference channel.

[0050] In this embodiment, on the browser side, when the audio and video data of multiple channels are controlled by the playback management module in the StreamaxPlayer.js library to be decoded and rendered according to the playback time, one of the channels can be determined as the reference channel based on the multiple channels controlled by the playback management module. Exemplarily, each of the multiple channels has a corresponding channel number, which can be regarded as a channel identifier. In this embodiment, the channel with channel number 1 can be determined as the reference channel based on the channel identifier. When the StreamaxPlayer.js library pulls the audio and video data through the audio and video streaming module, the audio and video data is loaded into the browser side from each channel. In this embodiment, the starting time of the audio and video data loaded into the reference channel is used as the starting playback time of the multi-channel synchronous playback of audio and video. It can be understood that the starting playback time is the starting time for the browser side to decode and render the audio and video data of each channel. The starting playback time is displayed on the virtual playback timeline of the playback management module, so that the playback management module controls the audio and video data of each channel to be decoded and rendered at the starting playback time. Furthermore, after using the start time of audio and video data loaded into the reference channel as the start time of audio and video multi-channel synchronous playback, it is also possible to detect whether other channels are loaded with audio and video data. If any of the other channels do not have audio and video data loaded, the start time of audio and video multi-channel synchronous playback is adjusted according to a preset delay value. It is understood that the other channels are all channels configured on the browser side except the reference channel. For example, assuming that the browser side is configured with four physical channels, numbered 1, 2, 3, and 4, where physical channel numbered 1 is the reference channel, upon detecting that physical channel numbered 1 has audio and video data loaded, the other three physical channels (i.e., physical channels numbered 2, 3, and 4) are detected to see if they have audio and video data loaded. If any channels do not have audio and video data loaded, assuming that physical channel numbered 2 does not have audio and video data loaded, a delay value can be set to adjust the start time of audio and video multi-channel synchronous playback according to the preset delay value to wait for physical channel numbered 2 to load audio and video data.

[0051] In some embodiments of the present application, when calling the StreamaxPlayer.js library to decode and render the audio and video data of multiple channels according to the playback time, it is detected whether all channels have loaded audio and video data at the playback time. If any channel does not have loaded audio and video data at the playback time, the synchronous playback of the audio and video data of any channel is ignored at the playback time. It can be understood that any channel refers to any one or more channels among all channels. In other words, if it is detected at the current playback time that one or more of the multiple channels do not have loaded audio and video data, the synchronous playback of the audio and video data of the one or more channels can be ignored at the current playback time.

[0052] In some embodiments of the present application, synchronized playback of audio and video data can also be achieved on the browser side through a plug-in. Specifically, a proprietary protocol can be used to multiplex multi-channel data, transmit the multi-channel audio and video data to the browser side through a single link, and then a pre-built synchronized playback plug-in can be used to parse the multi-channel audio and video data by channel and synchronize the playback according to the playback time.

[0053] In some embodiments of the present application, please refer to Figure 4, which is a method implementation flowchart of another method for multi-channel synchronous playback of audio and video provided by an embodiment of the present application, which may specifically include steps S41 to S44.

[0054] S41: receiving multi-channel audio and video data uploaded by an audio and video acquisition device;

[0055] S42: Unpacking the multi-channel audio and video data to obtain original audio and video data of multiple channels;

[0056] S43: For each channel of original audio and video data, add extended data to the I-frame audio and video data of the original audio and video data, and encapsulate the added extended data with the I-frame audio and video data to obtain audio and video data of multiple channels;

[0057] S44: Use multiple links to transmit the audio and video data of the multiple channels to the browser side, wherein the multiple links correspond to the audio and video data of the multiple channels one by one.

[0058] In the present embodiment, at the server end, after receiving the audio and video synchronous play request from the browser end, the server end can pronounce video signaling under the TCP communication link set up between the server end and the video acquisition device, so that the video acquisition device will upload the multi-channel audio and video data collected in real time to the server end according to the audio and video signaling. In the present embodiment, information such as channel information, server end IP and media port are carried in the audio and video signaling. Specifically, after receiving the audio and video signaling issued by the server end, the video acquisition device can set up the TCP media link between the video acquisition device and the server end by the server end IP and media port information carried in the audio and video signaling. Meanwhile, the video acquisition device uses corresponding microphone and camera to collect and obtain multi-channel audio and video data in real time according to the channel information, and then multi-channel audio and video data is carried out H264 or H265 coding compression, merged and uploaded to the server end again. After the media port of the server end receives the multi-channel audio and video data, the multi-channel audio and video data can be further unpacked, stored and restored H264 / H265 video frames and original audio frames according to the channel, thereby obtaining the original audio and video data of multiple channels. For the original audio and video data of each channel, extended data can be added to the I-frame audio and video data of the original audio and video data, and the added extended data and its corresponding I-frame audio and video data are encapsulated in accordance with the FLV standard format to obtain the audio and video data of each channel. Among them, I frame refers to an intra-frame coded frame (Intra Picture), which can be decoded into a complete frame of picture. Furthermore, for the audio and video data of multiple channels, multiple links can be used to transmit the audio and video data of multiple channels to the browser side. It can be understood that the multiple links correspond one-to-one to the audio and video data of multiple channels. In this embodiment, the server side can specifically distribute the audio and video data of each channel to the browser side through different HTTP-FLV links. HTTP-FLV (Flash Video Over Http) refers to a transmission protocol for FLV audio and video data based on HTTP.

[0059] In some embodiments of the present application, on the server side, when adding extended data to the I-frame audio and video data of the original audio and video data, specifically for each I-frame audio and video data in the original audio and video data, an extension unit can be added to the I-frame audio and video data, and the PTS timestamp and UTC timestamp corresponding to the I-frame audio and video data can be added to the extension unit. The PTS timestamp is represented as a display timestamp, which is used to tell the player the time to display this frame of data. The UTC timestamp is represented as a playback timestamp, which is used to tell the player the time to play this frame of data. Adding PTS timestamps and UTC timestamps to the original audio and video data through the extension unit will not destroy the original frame structure of H264 / H265, and standard streaming protocols such as HTTP-FLV, HLS, RTMP, etc. can still be used between the browser and the server, which has good versatility and scalability.

[0060] It can be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0061] In some specific embodiments, please refer to Figure 5, which is a basic structural block diagram of a multi-channel synchronous playback device for audio and video provided in an embodiment of the present application. The various units included in the device in this embodiment are used to execute the various steps in the above-mentioned method embodiment applied on the browser side. Please refer to the relevant description in the above-mentioned method embodiment for details. For the convenience of explanation, only the parts related to this embodiment are shown. As shown in Figure 5, the multi-channel synchronous playback device for audio and video includes an acquisition module 51, a pull module 52 and a playback module 53. Among them, the acquisition module 51 is used to obtain an audio and video synchronous playback request, and the audio and video synchronous playback request contains the identification of the audio and video acquisition device. The pull module 52 is used to pull the audio and video data of multiple channels from the server side according to the identification of the audio and video acquisition device. The playback module 53 is used to call the StreamaxPlayer library to decode and render the audio and video data of the multiple channels according to the playback time, so as to synchronously play the data content in the audio and video data of each channel with a timestamp consistent with the playback time at the playback time.

[0062] It should be understood that the multi-channel synchronous playback device for audio and video in this embodiment corresponds one-to-one to the multi-channel synchronous playback method for audio and video applied on the browser side, and will not be described in detail here.

[0063] In some specific embodiments, please refer to Figure 6, which is a basic block diagram of another multi-channel synchronous audio and video playback device provided in an embodiment of the present application. In this embodiment, the device includes various units configured to execute the various steps of the aforementioned method embodiment applied on the server side. For details, please refer to the relevant descriptions in the aforementioned method embodiment. For ease of illustration, only the portions relevant to this embodiment are shown. As shown in Figure 6, the multi-channel synchronous audio and video playback device includes a receiving module 61, an unpacking module 62, an adding module 63, and a transmission module 64. The receiving module 61 is configured to receive multi-channel audio and video data uploaded by an audio and video acquisition device. The unpacking module 62 is configured to unpack the multi-channel audio and video data to obtain the original audio and video data for multiple channels. The adding module 63 is configured to add extended data to the I-frame audio and video data of each channel's original audio and video data, and to encapsulate the added extended data with the I-frame audio and video data to obtain the audio and video data for multiple channels. The transmission module 64 is configured to transmit the audio and video data for the multiple channels to the browser side using multiple links, wherein the multiple links correspond one-to-one to the audio and video data for the multiple channels.

[0064] It should be understood that the multi-channel synchronous playback device for audio and video in this embodiment corresponds one-to-one to the multi-channel synchronous playback method for audio and video applied on the server side, and will not be described in detail here.

[0065] In some embodiments of the present application, please refer to Figure 7, which is a basic structural block diagram of an electronic device provided in an embodiment of the present application. As shown in Figure 7, the electronic device 7 of this embodiment includes: a processor 71, a memory 72, and a computer program 73 stored in the memory 72 and runnable on the processor 71, such as a program for the multi-channel synchronous playback method of each of the above-mentioned audio and video. When the processor 71 executes the computer program 73, the steps in each embodiment of the multi-channel synchronous playback method of each of the above-mentioned audio and video are implemented. Alternatively, when the processor 71 executes the computer program 73, the functions of each module in the embodiment corresponding to the multi-channel synchronous playback device of the above-mentioned audio and video are implemented. Please refer to the relevant description in the embodiment for details, which will not be repeated here.

[0066] Exemplarily, the computer program 73 can be divided into one or more modules (units), each of which is stored in the memory 72 and executed by the processor 71 to complete the present application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 73 in the electronic device 7. For example, the computer program 73 can be divided into an acquisition module, a pull module, and a playback module, and the functions of each module are as described above. Alternatively, the computer program 73 can be divided into a receiving module, an unpacking module, an adding module, and a transmission module, and the functions of each module are as described above.

[0067] The electronic device may include, but is not limited to, a processor 71 and a memory 72. Those skilled in the art will appreciate that FIG7 is merely an example of the electronic device 7 and does not limit the electronic device 7. The electronic device 7 may include more or fewer components than shown in the figure, or may combine certain components or different components. For example, the electronic device may further include input and output devices, network access devices, buses, and the like.

[0068] The processor 71 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0069] The memory 72 can be an internal storage unit of the electronic device 7, such as a hard drive or memory of the electronic device 7. The memory 72 can also be an external storage device of the electronic device 7, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 7. Furthermore, the memory 72 can include both an internal storage unit of the electronic device 7 and an external storage device. The memory 72 is used to store the computer program and other programs and data required by the electronic device. The memory 72 can also be used to temporarily store data that has been output or is about to be output.

[0070] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0071] The present application also provides a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, the computer-readable storage medium can implement the steps of the above-mentioned method embodiments. In this embodiment, the computer-readable storage medium can be non-volatile or volatile.

[0072] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal can implement the steps of the above-mentioned various method embodiments when executing the computer program product.

[0073] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0074] If the integrated module / unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0075] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0076] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for multi-channel synchronous playback of audio and video, characterized in that: The method is applied on the browser side, and includes: Obtaining an audio and video synchronous playback request, wherein the audio and video synchronous playback request includes an identifier of an audio and video acquisition device; Pulling multiple channels of audio and video data from a server according to the identifier of the audio and video acquisition device; The StreamaxPlayer library is called to decode and render the audio and video data of the multiple channels respectively according to the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

2. The method for multi-channel synchronous playback of audio and video according to claim 1, characterized in that: The step of calling the StreamaxPlayer library to decode the multiple channels of audio and video data according to the playback time, so as to synchronously play the data content in each channel of audio and video data whose timestamp is consistent with the playback time at the playback time, comprises: Call the StreamaxPlayer.wasm library to decapsulate the audio and video data of each channel and obtain the timestamp information of the audio and video data of each channel; The StreamaxPlayer.js library is called to decode and render the data content in the audio and video data of each channel whose timestamp is consistent with the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

3. The method for multi-channel synchronous playback of audio and video according to claim 2, characterized in that: Before the step of calling the StreamaxPlayer.js library to decode and render the data content in the audio and video data of each channel whose timestamp is consistent with the playback time, the step further includes: Determine the reference channel according to the channel identifier, and use the start time of loading the audio and video data of the reference channel as the start time of multi-channel synchronous playback of audio and video, and the start time is the start time of decoding and rendering the audio and video data of each channel by the browser end; Detect whether other channels are loaded with audio and video data. If there are channels in the other channels that are not loaded with audio and video data, adjust the starting playback time of the multi-channel synchronous playback of the audio and video according to the preset delay value. The other channels are all channels configured on the browser side except the reference channel.

4. The method for multi-channel synchronous playback of audio and video according to claim 2 or 3, characterized in that: The step of calling the StreamaxPlayer.js library to decode and render the data content in the audio and video data of each channel whose timestamp is consistent with the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time, further includes: Check whether all channels have loaded audio and video data at the playback time. If any channel has no audio and video data loaded at the playback time, ignore the synchronous playback of audio and video data for the arbitrary channel at the playback time. The arbitrary channel is any one or more channels among all the channels.

5. A method for multi-channel synchronous playback of audio and video, characterized in that: The method is applied on the server side and includes: Receive multi-channel audio and video data uploaded by audio and video acquisition equipment; Unpacking the multi-channel audio and video data to obtain original audio and video data of multiple channels; For the original audio and video data of each channel, adding extended data to the I-frame audio and video data of the original audio and video data, and encapsulating the added extended data with the I-frame audio and video data to obtain audio and video data of multiple channels; A plurality of links are used to transmit the audio and video data of the plurality of channels to the browser end, wherein the plurality of links correspond one-to-one to the audio and video data of the plurality of channels.

6. The method for multi-channel synchronous playback of audio and video according to claim 5, characterized in that: The step of adding extended data to the I-frame audio and video data of the original audio and video data comprises: For each I frame of audio and video data in the original audio and video data, an extension unit is added to the I frame of audio and video data, and a PTS timestamp and a UTC timestamp corresponding to the I frame of audio and video data are added to the extension unit.

7. A multi-channel synchronous playback device for audio and video, characterized in that: The device is applied on the browser side and includes: An acquisition module, used to acquire an audio and video synchronous playback request, wherein the audio and video synchronous playback request includes an identifier of an audio and video acquisition device; A pulling module, used to pull the audio and video data of multiple channels from the server according to the identification of the audio and video acquisition device; The playback module is used to call the StreamaxPlayer library to decode and render the audio and video data of the multiple channels respectively according to the playback time, so as to synchronously play the data content in the audio and video data of each channel whose timestamp is consistent with the playback time at the playback time.

8. A method for multi-channel synchronous playback of audio and video, characterized in that: The device is applied on the server side and includes: A receiving module, used to receive multi-channel audio and video data uploaded by an audio and video acquisition device; An unpacking module, used for unpacking the multi-channel audio and video data to obtain original audio and video data of multiple channels; An adding module is used to add extended data to the I-frame audio and video data of the original audio and video data of each channel, and encapsulate the added extended data with the I-frame audio and video data to obtain audio and video data of multiple channels; The transmission module is used to transmit the audio and video data of the multiple channels to the browser end by using multiple links, wherein the multiple links correspond one-to-one to the audio and video data of the multiple channels.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 or 5 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 or 5 to 6 are implemented.

Images