Live streaming processing method and electronic device

Abstract

The present invention proposes a live streaming processing method and electronic device, which belongs to the field of image communication technology. The method includes: a recording device records audio frames and video frames in real time, and performs target detection on the video frames to generate SEI frames; the recording device encodes the audio frames, video frames and SEI frames into streaming media frames respectively, and pushes them to the media server in real time; after the playback device establishes a connection with the media server, the media server pushes the streaming media frames to the playback device in real time; the playback device receives the streaming media frames, and plays the streaming media frames with the same recording time at the same time to play and display synchronously. In this way, the adjacent and subsequent push of the audio frames, video frames and SEI frames of the video frames recorded at the same time is realized, so that the playback device can obtain the video frames, audio frames and SEI frames at the same time with almost no delay, reduce or even eliminate the delay of synchronous playback, and greatly improve the real-time and synchronization of the display of video and target detection information in the live broadcast scene.

Description

Technical Field

[0001] The present invention relates to the field of image communication technology, and in particular to a live streaming processing method and electronic equipment. Background Art

[0002] With the increase of network bandwidth and the advancement of video encoding and decoding technology, live streaming has been widely used in entertainment, education, security monitoring and other fields. With the support of live streaming technology, video content is transmitted to user terminals around the world in real time, allowing users to watch and participate in live streaming in real time.

[0003] In current online video playback, in order to increase the fun and information richness of the video, target detection is usually performed on the video, for example, detecting information such as character emotions and movements, object types, etc., and displaying the target detection information when playing the video. However, currently, the synchronous display of video and target detection information can only be achieved in pre-recorded video files. In live broadcast scenarios, the real-time and synchronization of video and target detection information cannot be guaranteed. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a live streaming processing method and electronic device, which greatly improves the real-time and synchronization of the display of video and target detection information in a live streaming scene.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a live stream processing method, which is applied to a live stream processing system, wherein the live stream processing system includes a media server and a recording device and a playback device that are communicatively connected to the media server, and the method includes:

[0007] The recording device performs real-time recording to obtain an audio frame and a video frame at each recording moment, performs target detection on the video frame, and generates an SEI frame according to the detection result;

[0008] The recording device encodes the audio frame, the video frame and the SEI frame into streaming media frames respectively, and pushes the streaming media frames to the media server in real time;

[0009] When a live broadcast event is detected to be triggered, the playback device initiates a pull request to the media server to establish a connection with the media server;

[0010] After the connection is established, the media server pushes the streaming media frames to the playback device in real time;

[0011] The playback device receives the streaming media frames, plays the streaming media frames with the same recording time simultaneously, plays the video frames and the audio frames recorded simultaneously simultaneously, and draws the target detection information corresponding to the SEI frame on the played video frames.

[0012] Optionally, the playback device includes a network cache queue, a decoding cache queue and a playback cache queue, and a browser is deployed on the playback device;

[0013] The step of the playback device receiving the streaming media frame comprises:

[0014] The browser parses the received streaming media frame, organizes the parsing result into a parsed data frame according to a preset structure, and puts the parsed data frame into the network cache queue; wherein the parsing result includes a play time, and the play time is consistent with the recording time of the streaming media frame;

[0015] The browser cyclically reads the parsed data frames in the network cache queue, and puts the parsed data frames into the decoding cache queue if the parsed data frames read meet the delay limit;

[0016] The browser cyclically decodes the parsed data frames in the decoding cache queue to obtain the streaming media frames, and stores each of the streaming media frames into the playback cache queue in the order of the playback time.

[0017] Optionally, the browser cyclically reads the parsed data frames in the network cache queue, and when the parsed data frames read meet the delay limit, puts the parsed data frames into the decoding cache queue, comprising:

[0018] The browser selects the parsed data frame with the highest sorting from the network cache queue as the first target frame, and calculates the network delay of the first target frame;

[0019] Determining whether the network delay of the first target frame exceeds a preset maximum delay;

[0020] If yes, the parsed data frames whose playing time is before the current time are taken as frames to be cleared, and the frames to be cleared in the decoding cache queue are deleted;

[0021] The parsed data frame in the network cache queue whose playing time is the current moment is put into the decoding cache queue, and the step of returning to execute the browser taking the first parsed data frame in the network cache queue as the first target frame is performed.

[0022] Optionally, the parsing result also includes a SEI identifier;

[0023] The browser cyclically decodes the parsed data frames in the decoding cache queue to obtain the streaming media frames, and stores each of the streaming media frames in the playback cache queue in the order of the playback time, including:

[0024] The browser cyclically reads the parsed data frame that is at the front of the decoding cache queue as the second target frame;

[0025] For each of the second target frames, determine whether the SEI identifier of the second target frame is the first identifier; if so, put the second target frame into the play cache queue according to the order of the play time; if not, transfer the second target frame to a preset audio and video decoder; wherein the first identifier indicates a SEI frame;

[0026] Calculating the audio time difference and the video time difference between each of the second target frames input into the audio and video decoder;

[0027] When the audio time difference or the video time difference reaches a preset minimum delay, the second target frame input is decoded in sequence by the audio and video decoder to obtain a streaming media frame;

[0028] The streaming media frames are placed into the playback cache queue in the order of the playback time.

[0029] Optionally, the analysis result also includes a playback type;

[0030] The step of calculating the audio time difference and the video time difference between the second target frames transmitted to the audio and video decoder comprises:

[0031] From the second target frames transmitted to the audio and video decoder, select the second target frames whose playback type is audio as an audio group, and select the second target frames whose playback type is video as a video group;

[0032] Determine the earliest play time and the latest play time from the audio group, and use the time difference between the earliest play time and the latest play time as the audio time difference;

[0033] An earliest play time and a latest play time are determined from the video group, and a time difference between the earliest play time and the latest play time is used as a video time difference.

[0034] Optionally, the step of placing the second target frame into the playback cache queue in the order of the playback time includes:

[0035] Using a binary search method, searching the playback cache queue for a streaming media frame whose playback time is closest to the second target frame as an associated frame;

[0036] When the playback time of the second target frame is after the associated frame, or the playback time of the second target frame and the associated frame is the same, placing the second target frame after the associated frame;

[0037] When the play time of the second target frame is before the associated frame, the second target frame is placed before the associated frame.

[0038] Optionally, the step of simultaneously playing the streaming media frames having the same recording time includes:

[0039] Selecting the streaming media frame at the front of the playback cache queue and the streaming media frame and / or the SEI frame having the same playback time as the streaming media frame as the to-be-played group;

[0040] The video frames in the to-be-broadcast group are displayed in a canvas tag, the audio frames in the to-be-broadcast group are played in an audio tag, and the target detection information in the to-be-broadcast group is drawn on the canvas tag.

[0041] In a second aspect, the present invention provides a live stream processing method, which is applied to a recording device, and the method comprises:

[0042] Record the area within the camera range in real time to obtain audio frames and video frames at each recording moment, perform target detection on the video frames, and generate SEI frames according to the detection results;

[0043] The audio frame, the video frame and the SEI frame are respectively encoded into streaming media frames, and the streaming media frames are pushed to a media server in real time to implement the live stream processing method as described in the first aspect.

[0044] In a third aspect, the present invention provides a live streaming processing method, which is applied to a playback device, wherein the playback device is communicatively connected to a media server, and the method comprises:

[0045] When a live broadcast event is detected to be triggered, a pull request is initiated to the media server to establish a connection with the media server;

[0046] After the connection is established, the streaming media frames pushed by the media server are received, the streaming media frames with the same recording time are played simultaneously, so as to simultaneously play the video frames and the audio frames recorded at the same time, and draw the target detection information in the corresponding SEI frame on the played video frames.

[0047] In a fourth aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the live stream processing method as described in the second aspect or the third aspect.

[0048] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the live stream processing method as described in the second aspect or the third aspect is implemented.

[0049] The live streaming processing method and electronic device provided by the embodiment of the present invention include: a recording device performs real-time recording to obtain audio frames and video frames at each recording moment, and performs target detection on the video frames to generate SEI frames; the recording device encodes the audio frames, video frames and SEI frames into streaming media frames respectively, and pushes the streaming media frames to a media server in real time; when a live broadcast event is detected to be triggered, the playback device initiates a pull request to the media server to establish a connection with the media server; after the connection is established, the media server pushes the streaming media frames to the playback device in real time; the playback device receives the streaming media frames, and plays the streaming media frames with the same recording moment at the same time, so as to play the video frames and audio frames recorded at the same time, and draws the target detection information in the corresponding SEI frame on the played video frames.

[0050] In this way, the recording device completes the target detection of the video frame while recording, and pushes the audio frame, video frame and SEI frame of the video frame recorded simultaneously to the media server, and the media server pushes them to the connected playback device in real time, so as to realize the adjacent and subsequent push of the audio frame, video frame and SEI frame of the video frame recorded simultaneously. Therefore, the playback device can obtain the video frame, audio frame and SEI frame at the same time with almost no delay, so as to be able to simultaneously display the target detection information in the video frame and the SEI frame of the drawn video. Thereby, the synchronous playback delay caused by target detection of the playback device is reduced or even eliminated, which greatly improves the real-time and synchronization of the display of video and target detection information in the live broadcast scene.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A schematic structural diagram of a live streaming processing system provided by an embodiment of the present invention is shown.

[0054] Figure 2 One of the flow charts of the live streaming processing method provided by an embodiment of the present invention is shown.

[0055] Figure 3 Shows Figure 2 One of the flowcharts of some sub-steps of step 19.

[0056] Figure 4 Shows Figure 3 A flowchart of some sub-steps of step 192.

[0057] Figure 5 Shows Figure 3 A flowchart of some sub-steps of step 193.

[0058] Figure 6 Shows Figure 5 A flowchart of some sub-steps of step 1933.

[0059] Figure 7 Shows Figure 5 A flowchart of some sub-steps of step 1935.

[0060] Figure 8 A block diagram of an electronic device provided by an embodiment of the present invention is shown.

[0061] Fig. 9 The second flowchart of the live streaming processing method provided by the embodiment of the present invention is shown.

[0062] Fig.10 The third flowchart of the live streaming processing method provided by the embodiment of the present invention is shown.

[0063] Explanation of reference numerals: 10 - live streaming processing system; 110 - media server; 120 - recording device; 130 - playing device; 20 - electronic device; 210 - memory; 220 - processor; 230 - communication module. DETAILED DESCRIPTION

[0064] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0065] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0066] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0067] The live streaming processing method provided by the embodiment of the present invention can be applied to Figure 1 In the live streaming processing system 10 shown, the live streaming processing system 10 includes a media server 110 and a recording device 120 and a playback device 130 that are communicatively connected to the media server 110 .

[0068] Among them, one or more browsers are installed on the playback device 130 and the recording device 120 (the browsers in the present invention are all app video browsers with live broadcast function).

[0069] The recording device 120 can be connected to the media server 110 through a network, so that the live broadcast personnel can start the live broadcast through the live broadcast function of a browser of the recording device 120. After the viewing user clicks or selects a live broadcast event in the browser of the playback device 130, the live broadcast event is triggered, and the playback device 130 initiates a pull request to the media server 110 corresponding to the live broadcast event, so as to establish a communication connection with the media server 110 for playing the live broadcast event.

[0070] For example, the javascript script in the browser uses a protocol such as the wss protocol or the https protocol to send a pull request to the media server 110 corresponding to the Uniform Resource Locator (URL) of the live event, and the media server 110 establishes a connection with the browser based on the pull request.

[0071] The recording device 120 and the playback device 130 include, but are not limited to: a personal computer, a laptop computer, a tablet computer, a smart TV, a camera, a wearable editing device, etc.

[0072] Please refer to Figure 2 , the embodiment of the present invention provides an application to the above Figure 1 The live stream processing method of the live stream processing system 10 shown includes the following steps 11 to 19.

[0073] Step 11: The recording device performs real-time recording to obtain audio frames and video frames at each recording moment, performs target detection on the video frames, and generates SEI frames according to the detection results.

[0074] Step 13: The recording device encodes the audio frames, video frames and SEI frames into streaming media frames respectively, and pushes the streaming media frames to the media server in real time.

[0075] Step 15: When it is detected that the live broadcast event is triggered, the playback device initiates a pull request to the media server to establish a connection with the media server.

[0076] Step 17: After the connection is established, the media server pushes the streaming media frames to the playback device.

[0077] Step 19, the playback device receives streaming media frames, plays streaming media frames with the same recording time simultaneously, plays the simultaneously recorded video frames and audio frames simultaneously, and draws the target detection information in the corresponding SEI frame on the played video frames.

[0078] For example, in combination Figure 1 As shown, the live broadcaster starts the live broadcast through the live broadcast function of a browser on the recording device 120. The recording device 120 records the content that the live broadcaster wants to broadcast in real time, obtains the audio frame and video frame at each recording moment, performs target detection on the video frame, and generates an SEI frame according to the detection result.

[0079] For example, the recording device 120 can use any target detection model to perform target detection on each captured YUV format video frame image, and obtain target detection information including the target frame position, target name, target type recognition rate (i.e., confidence or probability), etc. The frame position is the position coordinates of the four points of the rectangular frame, which can be in the form of (0,0) in the upper left corner of the image and (100,100) in the lower right corner. The SEI frame can be in the form of a json string, for example, the character content can be: [{"xmin":50, "xmax":60, "ymin":40, "ymax":40, "class": "person", "score":0.8},{...},{...},...]. Among them, xmin, xmax, ymin and ymax respectively represent the coordinate values ​​of the frame position, class represents the target name of the target type, and score represents the recognition rate of the target type.

[0080] Next, the recording device 120 encodes the audio frame, the video frame and the SEI frame into a streaming media frame, and pushes the streaming media frame to the media server 110 in real time. When pushing the streaming media frame, the SEI frame of the video frame follows the video frame. In order to increase the speed of subsequent push transmission, the media server 110 can encode the received streaming media frame according to the FLV encoding rule to obtain a streaming media frame with a smaller file size.

[0081] After the viewing user clicks or selects a live event in the browser of the playback device, the live playback event is triggered, and the playback device initiates a pull request to the media server corresponding to the live event. For example, the JavaScript script in the browser uses protocols such as the wss protocol or the https protocol to send a pull request to the media server corresponding to the Uniform Resource Locator (URL) of the live event, so as to establish a communication connection with the media server for playing the live event.

[0082] After the browser of the playback device establishes a connection with the media server, the media server pushes the encoded streaming media frames received from the recording device to the playback device in real time. The playback device receives the streaming media frames pushed by the media server, and plays the streaming media frames with the same recording time at the same time, so as to play the video frames and audio frames recorded at the same time, and draw the target detection information in the corresponding SEI frame on the played video frame.

[0083] In step 11 to step 19 of the live streaming processing method provided by the present invention, the recording device completes the target detection of the video frame while recording, and pushes the audio frame, video frame and SEI frame of the video frame recorded at the same time to the media server, and the media server pushes them to the connected playback device in real time, so as to realize the adjacent and subsequent push of the audio frame, video frame and SEI frame of the video frame recorded at the same time. Therefore, the playback device can obtain the video frame, audio frame and SEI frame at the same time with almost no delay, so as to be able to simultaneously display the target detection information in the video frame and the SEI frame of the drawn video. Thereby, the synchronous playback delay caused by the target detection of the playback device is reduced or even eliminated, which greatly improves the real-time and synchronization of the display of the video and target detection information in the live broadcast scene.

[0084] In step 19, the playback device may directly decode and play after receiving the streaming media frame and obtaining all the streaming media frames at the same recording time, or may receive according to a preset rule and then play, which is not limited in this embodiment.

[0085] In order to reduce the delay of live broadcast and improve the fluency of live broadcast, the idea of ​​removing the streaming media frames with high delay is introduced in the process of receiving the streaming media frames in step 19. For example, the playback device 130 is pre-set with a network cache queue, a decoding cache queue and a playback cache queue, and a browser is deployed on the playback device 130. Figure 3 , the process of the playback device 130 receiving the streaming media frame in step 19 may include steps 191 to 193.

[0086] Step 191, the browser parses the received streaming media frame, organizes the parsing result into parsed data frames according to a preset structure, and puts the parsed data frames into a network cache queue.

[0087] The parsing result includes the playback time, which is consistent with the recording time of the streaming media frame, that is, the playback time is the recording time.

[0088] Step 192: The browser cyclically reads the parsed data frames in the network cache queue, and puts the parsed data frames into the decoding cache queue when the parsed data frames read meet the delay limit.

[0089] Step 193, the browser cyclically decodes the parsed data frames in the decoding cache queue to obtain streaming media frames, and stores each streaming media frame in the playback cache queue in the order of playback time.

[0090] In step 191, the received streaming media frame is byte data, and may be byte data after the streaming media frame is encoded according to the FLV encoding rule. Therefore, the browser may parse the byte data (i.e., the received streaming media frame) one by one according to the FLV encoding rule to obtain a parsing result, which includes the playback time (in milliseconds), the playback type (i.e., audio or video), whether it is an SEI frame, and the data within the frame.

[0091] Furthermore, according to the structure of the predefined structure, each streaming media frame is organized into a structure containing the above parsing results to obtain a parsed data frame, and then the parsed data frame is put into the network cache queue. The structure may be in the form of: {"ts":1000,"type":"video","is_sei":true,"data":[...]}, ts indicates the playback time, type indicates the playback type, is_sei indicates whether it is a SEI frame, and data indicates the data within the frame.

[0092] It should be noted that the intra-frame data is the content data of the streaming media frame (such as audio data, video data or object detection data). In addition, since the SEI frame is composed of the object detection information of the video frame, and the object detection information needs to be displayed synchronously with the video frame, the playback type of the SEI frame is video.

[0093] In step 192, the setting of the determination method of whether the delay limit is met can be flexibly selected. For example, the interval between the recording time and the current time can be calculated, and the parsed data frames whose interval is longer than the preset time do not meet the delay limit, or the parsed data frames whose interval is longer than the preset time and some parsed data frames after the parsed data frames do not meet the delay limit. Moreover, the above method is only an example, and its specific implementation is not limited.

[0094] In order to improve the fluency and real-time performance of live broadcast, when the interval length of the parsed data frames is longer than the preset length in step 192, all the parsed data frames recorded before the current time do not meet the delay limit. For example, in the network cache queue, the parsed data frames are arranged in order from the earliest to the latest recording time, referring to Figure 4 , step 192 may include steps 1921 to 1925.

[0095] Step 1921: The browser selects the parsed data frame with the highest sorting order from the network cache queue as the first target frame, and calculates the network delay of the first target frame.

[0096] The network delay of the first target frame is the time difference between the current time and the playback time of the first target frame.

[0097] Step 1922, determine whether the network delay of the first target frame exceeds the preset maximum delay. If yes, execute steps 1923 and 1924 in sequence, if no, execute step 1925.

[0098] Step 1923: The parsed data frames whose playback time is before the current time are regarded as frames to be cleared, and each frame to be cleared in the decoding cache queue is deleted.

[0099] Step 1924: put the parsed data frame in the network cache queue whose playback time is the current moment into the decoding cache queue. After step 1924, the process returns to step 1921.

[0100] Step 1925: put the first target frame into the decoding buffer queue. After step 1925, the process returns to step 1921.

[0101] Among them, the value of the maximum delay can be flexibly set according to the live broadcast performance requirements or different live broadcast scenarios. For example, it can be 3 seconds, 4 seconds, or any other arbitrary value, which is not limited in this embodiment.

[0102] Through the above steps 1921 to 1925, the browser script continuously detects whether there is a parsed data frame in the network cache queue, cyclically reads the first parsed data frame in the network cache queue, calculates the network delay of the parsed data frame, and if the network delay exceeds the pre-set maximum delay, clears the current network cache queue (that is, clears the parsed data frames in the network cache queue whose playback time is before the current time), directly reads the latest data frame, and puts the read data frame into the decoding cache queue.

[0103] Taking the pre-set maximum delay of 3 seconds as an example, if the playback time of the read parsed data frame is the 5th second, but the current moment is the 10th second, the data of the 10th second should be played. At this time, the pre-set maximum network delay limit of 3 seconds has been exceeded. The parsed data frames with playback time before the current moment in the network cache queue are cleared, and the latest parsed data frame is directly read and put into the decoding cache queue.

[0104] In this way, after a large delay occurs, it can be ensured that the playback device plays the latest data next, which greatly improves the smoothness and real-time performance of the live broadcast.

[0105] While the browser cyclically reads the parsed data frames in the network cache queue and puts them into the decoding cache queue, the browser also continuously decodes the parsed data frames in the decoding cache queue and puts them into the playback cache queue by executing step 193. The decoding method can be flexibly set, for example, the parsed data frames in the decoding cache queue are continuously decoded in the order of the playback time, or they can be decoded according to a preset decoding rule, which is not limited in this embodiment.

[0106] In order to reduce the jamming phenomenon during live broadcast and further improve the fluency of live broadcast, in step 193 of this embodiment, the decoding is performed only when the number of playable continuous frames reaches a preset number. For example, the decoded data frames in the decoding cache queue are sorted in order from the beginning to the end according to the playback time. Figure 5 , step 193 may include sub-steps 1931 to 1935.

[0107] Step 1931: The browser cyclically reads the parsed data frame with the highest sorting order from the decoding cache queue as the second target frame.

[0108] Step 1932, for each second target frame, determine whether the SEI identifier of the second target frame is the first identifier. If so, put the second target frame into the playback cache queue in the order of playback time. If not, pass the second target frame to the preset audio and video decoder.

[0109] The first identifier indicates a SEI frame. In addition, the setting of the SEI identifier can be flexibly selected, for example, the SEI identifier can be "1" and "0", "1" indicates that it is a SEI frame, and "0" indicates that it is not a SEI frame. The SEI identifier can also be "true" and "false", "true" indicates that it is a SEI frame, and "false" indicates that it is not a SEI frame, which is not limited in this embodiment.

[0110] Step 1933, calculating the audio time difference and video time difference between each second target frame input to the audio and video decoder.

[0111] Step 1934, when the audio time difference or the video time difference reaches the preset minimum delay, the incoming second target frames are decoded in sequence by the audio and video decoders to obtain streaming media frames.

[0112] Step 1935, put the streaming media frames into the playback cache queue in the order of playback time.

[0113] The audio time difference in step 1933 refers to the play time difference between the audio frames in the second target frame, and the video time difference refers to the play time difference between the video frames in the second target frame. Figure 6, the calculation of the audio time difference and the video time difference may include the following steps 331 to 333.

[0114] Step 331 : From the second target frames input to the audio and video decoder, select the second target frames whose playback type is audio as the audio group, and select the second target frames whose playback type is video as the video group.

[0115] Step 332, determining the earliest play time and the latest play time from the audio group, and using the time difference between the earliest play time and the latest play time as the audio time difference.

[0116] Step 333: determine the earliest play time and the latest play time from the video group, and use the time difference between the earliest play time and the latest play time as the video time difference.

[0117] The above-mentioned audio and video decoder can be any decoder, for example, it can be an audio and video decoder applied for by webcodecs API, and the audio and video decoder can be composed of an independent audio decoder and video decoder, which is not limited in this embodiment.

[0118] In actual live broadcast scenarios, when the network quality is poor, the playback device receives streaming media frames slowly, making the speed of parsing data frames stored in the decoding cache queue slower than the speed of playing data frames. As a result, it is very easy for the decoding cache queue to have little or no data, but there are no data frames in the playback cache queue, causing the live broadcast to be stuck or not smooth. Or when the network is congested or the quality is poor, it is very easy for decoded data frames with a later playback time to arrive before decoded data frames with an earlier playback time, causing playback disorder.

[0119] By executing steps 1931 to 1935, the decoded data frames are sorted in order of playback time, decoding is performed only when the time difference between consecutive decoded data frames to be decoded reaches the shortest delay, and the frames are placed in the playback cache queue in order of playback time, so that decoding and playback strictly follow the permission of the playback time, thereby improving the situation where the live broadcast is not smooth and disordered due to network fluctuations.

[0120] In addition, the browser script continuously detects whether there are parsed data frames in the decoding cache queue, and cyclically reads the first parsed data frame in the decoding cache queue. When the first parsed data frame is an SEI frame, it does not need to be decoded and is directly stored in the playback cache queue. When the first parsed data frame is not an SEI frame, it is passed to the audio and video decoder applied for through the webcodecs API, and the total cache time difference (audio time difference or video time difference) in the decoding cache queue must reach the preset minimum delay before the decoding operation is performed. Therefore, when the speed of pulling data caused by short-term network congestion is slower than the decoding process, the cached data in the decoding cache queue can support a certain amount of time, which can offset the impact of network congestion and improve the jamming situation.

[0121] In addition, the playback device can detect the amount of data in the decoding buffer queue through the browser. When the amount of data in the decoding buffer queue is detected to be empty, the decoding is stopped until the data time difference (i.e., audio time difference or video time difference) in the decoding buffer queue reaches the preset minimum delay before continuing to perform decoding. This is to try to make the amount of data in the decoding buffer queue able to support a certain amount of time, improve the performance of hedging the impact of network congestion, and further maintain the smoothness of playback.

[0122] In the process of putting the second target frame into the play cache queue in the above step 1932, the placement position of the second target frame can be searched by traversing one by one, or by searching according to a preset rule, which is not limited in this embodiment.

[0123] In order to find the exact position as soon as possible when putting the data into the play cache queue. In the process of putting the second target frame into the play cache queue in step 1932, refer to Figure 7 , the following steps 351 to 353 can be performed.

[0124] Step 351 , using a binary search method, searches for a streaming media frame whose playback time is closest to the second target frame from the playback cache queue as an associated frame.

[0125] Step 352: When the playback time of the second target frame is after the associated frame, or the playback time of the second target frame and the associated frame is the same, place the second target frame after the associated frame.

[0126] Step 353: When the playing time of the second target frame is before the associated frame, the second target frame is placed before the associated frame.

[0127] The process of putting the streaming media frame into the playback cache queue in step 1935 is similar to the above steps 351 to 353, and the second target frame is replaced with the streaming media frame, which will not be repeated in this embodiment.

[0128] For each streaming media frame put into the play buffer queue, any method can be used to play it in step 19, for example, it can be played directly, or it can be played after being sorted according to a preset rule, which is not limited in this embodiment.

[0129] In order to improve the synchronization of video frames, audio frames and SEI frames at the same recording time, refer to Figure 7 The process of simultaneously playing the streaming media frames having the same recording time in step 19 may include steps 194 to 195.

[0130] Step 194 : Select the streaming media frame at the front of the queue and the streaming media frame and / or SEI frame with the same play time as the streaming media frame from the play buffer queue as the to-be-played group.

[0131] Step 195: display the video frames in the to-be-played group in the canvas tag, play the audio frames in the to-be-played group in the audio tag, and draw the target detection information in the to-be-played group on the canvas tag.

[0132] The canvas tag is the canvas tag, and the audio tags are the AudioContext and audio tags. If there is an SEI frame in the to-be-played group, the data in the SEI frame is read in a string format and converted into a javascript object to obtain the position coordinates, target name, recognition rate and other information of the target detection frame, and a graphic frame and text are drawn in the canvas tag of the video display using webGL. The text indicates the target name (i.e., classification) and recognition rate of the object in the frame.

[0133] Please refer to Figure 8 , is a block diagram of an electronic device 20, which may be Figure 8 The recording device 120 or the playback device 130 in the live streaming processing method shown. The electronic device 20 includes a memory 210, a processor 220 and a communication module 230. The memory 210, the processor 220 and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0134] The memory 210 is used to store programs or data and can be, but not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable read-only memory, an electrically erasable read-only memory, and the like.

[0135] The processor 220 is used to read / write data or programs stored in the memory 210 and execute corresponding functions.

[0136] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through a network, and to send and receive data through the network.

[0137] It should be understood that Figure 8 The structure shown is only a schematic diagram of the structure of the electronic device 20. The electronic device 20 may also include Figure 8 More or fewer components as shown, or with Figure 8 Different configurations shown. Figure 8 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0138] Based on the same aspect as the above live stream processing method, the present invention also provides an Figure 1 The live stream processing method of the recording device 120 of the live stream processing system 10 shown in FIG. Fig. 9 , including steps 21 to 23, and the recording device 120 can be an electronic device 20 as shown in the figure, and the processor 220 of the electronic device 20 executes the computer program stored in the memory 210 to implement the following steps 21 to 23.

[0139] Step 21, recording the area within the camera range in real time, obtaining audio frames and video frames at each recording moment, performing target detection on the video frames, and generating SEI frames according to the detection results.

[0140] Step 23: Encode the audio frames, video frames and SEI frames into streaming media frames respectively, and push the streaming media frames to the media server in real time.

[0141] Through the above steps 21 to 23, the live stream processing method applied to the live stream processing system 10 as provided above is assisted in being implemented.

[0142] Similarly, based on the same aspect as the above live stream processing method, the present invention also provides an application Figure 1 The live stream processing method of the playback device 130 of the live stream processing system 10 shown in FIG. Fig.10 , including steps 31 to 33, and the recording device 120 can be an electronic device 20 as shown in the figure, and the processor 220 of the electronic device 20 executes the computer program stored in the memory 210 to implement the following steps 31 to 33.

[0143] Step 31, when it is detected that the live broadcast event is triggered, a pull request is initiated to the media server to establish a connection with the media server.

[0144] Step 33, after the connection is established, the streaming media frames pushed by the media server are received, the streaming media frames with the same recording time are played simultaneously, so as to simultaneously play the video frames and audio frames recorded simultaneously, and the target detection information in the corresponding SEI frame is drawn on the played video frame.

[0145] Moreover, the specific implementation and effect of the above step 31 can refer to the description of step 15 above, and the specific implementation and effect of the above step 33 can refer to the description of step 19 above, which will not be repeated in this embodiment. Through the above steps 31 to 33, the live stream processing method applied to the live stream processing system 10 as provided above is assisted to be implemented.

[0146] Thus, through the above steps 21 to 23, and with the assistance of steps 31 to 33, in the live streaming processing system 10, the recording device 120 completes the target detection of the video frame while recording, and pushes the audio frame, video frame and SEI frame of the video frame recorded at the same time to the media server 110, and the media server 110 pushes them to the connected playback device 130 in real time, so as to realize the adjacent and subsequent push of the audio frame, video frame and SEI frame of the video frame recorded at the same time. Therefore, the playback device 130 can obtain the video frame, audio frame and SEI frame at the same time almost without delay, so as to simultaneously display the target detection information in the video frame and the SEI frame of the drawn video. Thus, the synchronous playback delay caused by the target detection of the playback device 130 is reduced or even eliminated, which greatly improves the real-time and synchronization of the video and target detection information display in the live scene.

[0147] The embodiment of the present invention further provides a live stream processing device. When the live stream processing device is applied to the recording device 120, it includes a recording detection module and an encoding push module.

[0148] The recording detection module is used to record the area within the camera range in real time, obtain the audio frame and video frame at each recording moment, perform target detection on the video frame, and generate SEI frame according to the detection result.

[0149] The encoding push module is used to encode the audio frame, the video frame and the SEI frame into the streaming media frame respectively, and push the streaming media frame to the media server 110 in real time, so as to realize the live streaming processing method applied to the live streaming processing system 10 as provided above.

[0150] When the live streaming processing apparatus is applied to the playback device 130, it includes a connection module and a receiving and playback module.

[0151] The connection module is used to initiate a pull request to the media server 110 to establish a connection with the media server 110 when detecting that a live broadcast event is triggered.

[0152] The receiving and playing module is used to receive the streaming media frames pushed by the media server 110 after establishing the connection, play the streaming media frames with the same recording time simultaneously, play the video frames and audio frames recorded simultaneously, and draw the target detection information in the corresponding SEI frame on the played video frame.

[0153] For the specific implementation and effect of the live stream processing device, please refer to the description of the implementation of the live stream processing method above. For example, for the specific implementation and effect of the recording detection module, please refer to the description of the relevant content of step 21 above, for the specific implementation and effect of the connection module, please refer to the description of the relevant content of step 31 above, and for the specific implementation and effect of the receiving and playing module, please refer to the description of the relevant content of step 33 above, which will not be repeated here.

[0154] In addition, each module in the above-mentioned live stream processing device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or can be stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to the above modules to implement the live stream processing method provided above.

[0155] Based on the same inventive concept as the above-mentioned live stream processing method, an embodiment of the present invention also provides an electronic device 20, including a processor 220 and a memory 210, the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the live stream processing method proposed in the embodiment of the present invention.

[0156] The embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the live streaming processing method proposed in the embodiment of the present invention is implemented.

[0157] The embodiment of the present invention further provides a computer program product, which includes computer instructions, wherein when the computer instructions are executed by the processor 220, the live streaming processing method proposed in the embodiment of the present invention is implemented.

[0158] In summary, the live streaming processing method and electronic device provided by the embodiments of the present invention have at least the following beneficial effects:

[0159] (1) Real-time: The target detection information can be generated and transmitted in real time during the live broadcast and played synchronously with the video, ensuring the timeliness and accuracy of the information;

[0160] (2) Integrated storage and transmission: Video data and target detection information are stored and transmitted in an integrated manner. Users only need to download and process one data stream, which simplifies the usage process and improves the user experience.

[0161] (3) Convenient and fast: Through optimized technical means, the synchronous playback of video and target detection information can be easily achieved on the web side without the need for additional operations or downloading multiple files, which improves the convenience of the system and user experience.

[0162] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0164] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0165] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A live stream processing method, characterized in that: Applied to a live streaming processing system, the live streaming processing system includes a media server and a recording device and a playback device that are communicatively connected to the media server, the method includes: The recording device performs real-time recording to obtain an audio frame and a video frame at each recording moment, performs target detection on the video frame, and generates an SEI frame according to the detection result; The recording device encodes the audio frame, the video frame and the SEI frame into streaming media frames respectively, and pushes the streaming media frames to the media server in real time; When a live broadcast event is detected to be triggered, the playback device initiates a pull request to the media server to establish a connection with the media server; After the connection is established, the media server pushes the streaming media frames to the playback device in real time; The playback device receives the streaming media frames, plays the streaming media frames with the same recording time simultaneously, plays the video frames and audio frames recorded simultaneously simultaneously, and draws the target detection information corresponding to the SEI frame on the played video frames; The playback device includes a network cache queue, a decoding cache queue and a playback cache queue, and a browser is deployed on the playback device; The step of the playback device receiving the streaming media frame comprises: The browser parses the received streaming media frame, organizes the parsing result into a parsed data frame according to a preset structure, and puts the parsed data frame into the network cache queue; wherein the parsing result includes a play time, and the play time is consistent with the recording time of the streaming media frame; The browser cyclically reads the parsed data frames in the network cache queue, and puts the parsed data frames into the decoding cache queue if the parsed data frames read meet the delay limit; The browser cyclically decodes the parsed data frames in the decoding cache queue to obtain the streaming media frames, and stores each of the streaming media frames into the playback cache queue in the order of the playback time; The browser cyclically reads the parsed data frames in the network cache queue, and puts the parsed data frames into the decoding cache queue when the parsed data frames read meet the delay limit, including: The browser selects the parsed data frame with the highest sorting from the network cache queue as the first target frame, and calculates the network delay of the first target frame; Determining whether the network delay of the first target frame exceeds a preset maximum delay; If yes, the parsed data frames whose playing time is before the current time are taken as frames to be cleared, and the frames to be cleared in the decoding cache queue are deleted; The parsed data frame in the network cache queue whose playing time is the current moment is put into the decoding cache queue, and the step of returning to execute the browser taking the first parsed data frame in the network cache queue as the first target frame is performed.

2. The live streaming processing method according to claim 1, characterized in that: The analysis result also includes a SEI identifier; The browser cyclically decodes the parsed data frames in the decoding cache queue to obtain the streaming media frames, and stores each of the streaming media frames in the playback cache queue in the order of the playback time, including: The browser cyclically reads the parsed data frame that is at the front of the decoding cache queue as the second target frame; For each of the second target frames, determine whether the SEI identifier of the second target frame is the first identifier; if so, put the second target frame into the play cache queue according to the order of the play time; if not, transfer the second target frame to a preset audio and video decoder; wherein the first identifier indicates a SEI frame; Calculating the audio time difference and the video time difference between each of the second target frames input into the audio and video decoder; When the audio time difference or the video time difference reaches a preset minimum delay, the second target frame input is decoded in sequence by the audio and video decoder to obtain a streaming media frame; The streaming media frames are placed into the playback cache queue in the order of the playback time.

3. The live streaming processing method according to claim 2, characterized in that: The analysis result also includes the playback type; The step of calculating the audio time difference and the video time difference between the second target frames transmitted to the audio and video decoder comprises: From the second target frames transmitted to the audio and video decoder, select the second target frames whose playback type is audio as an audio group, and select the second target frames whose playback type is video as a video group; Determine the earliest play time and the latest play time from the audio group, and use the time difference between the earliest play time and the latest play time as the audio time difference; An earliest play time and a latest play time are determined from the video group, and a time difference between the earliest play time and the latest play time is used as a video time difference.

4. The live streaming processing method according to claim 2, characterized in that: The step of placing the second target frame into the playback cache queue in the order of the playback time comprises: Using a binary search method, searching the playback cache queue for a streaming media frame whose playback time is closest to the second target frame as an associated frame; When the playback time of the second target frame is after the associated frame, or the playback time of the second target frame and the associated frame is the same, placing the second target frame after the associated frame; When the play time of the second target frame is before the associated frame, the second target frame is placed before the associated frame.

5. The live streaming processing method according to claim 1, characterized in that: The step of simultaneously playing the streaming media frames having the same recording time comprises: Selecting the streaming media frame at the front of the playback cache queue and the streaming media frame and / or the SEI frame having the same playback time as the streaming media frame as the to-be-played group; The video frames in the to-be-broadcast group are displayed in a canvas tag, the audio frames in the to-be-broadcast group are played in an audio tag, and the target detection information in the to-be-broadcast group is drawn on the canvas tag.

6. A live stream processing method, characterized in that: Applied to a recording device, the method comprises: Record the area within the camera range in real time to obtain audio frames and video frames at each recording moment, perform target detection on the video frames, and generate SEI frames according to the detection results; The audio frame, the video frame and the SEI frame are respectively encoded into streaming media frames, and the streaming media frames are pushed to a media server in real time to implement the live streaming processing method as described in any one of claims 1 to 5.

7. A live stream processing method, characterized in that: Applied to a playback device, the playback device is communicatively connected to a media server, and the method comprises: When a live broadcast event is detected to be triggered, a pull request is initiated to the media server to establish a connection with the media server; After establishing the connection, the streaming media frames pushed by the media server are received, the streaming media frames with the same recording time are played simultaneously, so as to simultaneously play the video frames and audio frames recorded simultaneously, and the target detection information in the corresponding SEI frame is drawn on the played video frame to implement the live stream processing method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the live stream processing method as described in claim 6 or 7.

Images