Player-based teaching live broadcast PPT alignment method and system

Abstract

The invention provides a method and system for carrying out teaching live broadcast PPT alignment based on a player, and relates to the technical field of signal processing. A server rear end converts a video live broadcast signal into an m3u8 playlist, carries out video stream transmission in a ts format video fragment form, and carries out definition according to time zone time; the PPT recording signal is subjected to image recognition processing and converted into PPT image data, time is defined and generated through time zone time, and corresponding PPT data is obtained and locally cached at the front end of the browser; the front end of the browser continuously receives a video stream through an HLS protocol, plays the video stream in the browser, monitors an hls.js library to obtain TS fragment information corresponding to the currently played ts format video fragment, inquires PPT data corresponding to time zone time in a local cache, plays corresponding PPT image data, and completes teaching live broadcast PPT alignment.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a method and system for aligning PPT in live teaching broadcast based on a player. Background Art

[0002] With the popularization of online education, online live teaching accounts for an increasingly larger proportion in online teaching scenarios. However, in the currently popular online live teaching scenarios, the content described in the video and the online handouts (PPT) are often out of sync, which gives users a very poor user experience and makes it difficult to achieve ideal learning results.

[0003] This situation is mainly limited by the network infrastructure and network bandwidth in the user's area. Users may experience playback freezes during online live learning. The freezes will cause the content the user watches on the client to lag behind the content pushed by the server.

[0004] If the PPT is pushed to the client for display based on the playback progress of the server, the worse the network environment is or the longer the live broadcast lasts, the more serious the asynchrony between the live broadcast image and the PPT will be. In addition, due to the server's response to the request or the CDN's back-to-source processing, there will be a certain delay in the live video viewed by different users, resulting in a certain delay even if the user refreshes the page. Due to the superposition of delays, the asynchrony between the live broadcast image and the PPT will occur again soon.

[0005] Therefore, the present invention provides a method and system for aligning a live teaching PPT based on a player to alleviate the situation where the picture and the PPT cannot be aligned during the user's viewing process. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and system for aligning PPT for live teaching based on a player. Based on the HLS protocol, the alignment of video images and online handouts (PPT) is realized on the front end of the browser, thereby solving the problem that the video and PPT cannot be aligned due to network environment, backend source return, deployment and other reasons.

[0007] The purpose of the present invention is achieved through the following technical solutions: On the one hand, the present invention provides a method for aligning a teaching live PPT based on a player, and the teaching live PPT alignment is performed through the following steps: Step 1: The live broadcast collection unit collects live broadcast data during the teaching live broadcast process and uploads it to the server backend; wherein the live broadcast data includes video live broadcast signal and PPT recording signal; Step 2: The server backend receives live streaming data; the live video signal is converted into several m3u8 playlists according to the HLS protocol; the PPT recorded signal is processed by image recognition and converted into several PPT image data; Step 3: The server backend uses the principle of live stream pts increment to check the timestamps of the PPT recording signal and the live video signal, and keeps the timestamp increment through timestamp correction; Step 4: The server backend performs live teaching video broadcast; during the live video broadcast process, the live video signal is transmitted as a video stream in the form of several ts format video segments, and the ts format video segments in the m3u8 playlist are defined according to the time zone through the live broadcast feature of the HLS protocol; Step 5: The server backend generates corresponding PPT data; wherein, in the process of generating the PPT data, the generation time of the PPT image data is defined by the time zone time to obtain the corresponding PPT data; Step 6: The browser front end receives the live teaching video; during the video playback process, the browser front end continuously receives the video stream through the HLS protocol and plays the video stream in the browser; the TS segment information corresponding to the TS format video segment currently being played is obtained by monitoring the hls.js library; the TS segment information includes the segment path and definition rules; Step 7: During the process of receiving the live teaching video, the PPT server pushes the PPT data to the browser front end, and the browser front end caches the PPT data locally; Step 8: The browser front end obtains the TS segment information corresponding to the currently playing ts format video segment, finds the ts format video segment through the segment path, queries the corresponding time zone time according to the defined rules, queries the PPT data corresponding to the time zone time in the local cache, plays the corresponding PPT image data, and completes the live teaching PPT alignment.

[0008] As a further solution, in step 3, the timestamp is corrected by a correction algorithm to keep the timestamp incremented; wherein the correction algorithm sets the following rules: Rule 1: Set the jitter range of the transition area and allow the timestamp to have negative values ​​in the transition area; Rule 2: When the time interval between adjacent timestamps is greater than the increasing threshold, the timestamps are adjusted until the time interval reaches the increasing threshold.

[0009] As a further solution, the time zone is set to the regional standard time corresponding to the live broadcast area; wherein, China Standard Time is correspondingly set to Beijing time.

[0010] As a further solution, in step 4, the time zone of the ts format video segment is defined according to the naming rule.

[0011] As a further solution, in step 6, the TS segment information corresponding to the currently playing TS format video segment is obtained by monitoring the Hls.Events.FRAG_CHANGED event of the hls.js library.

[0012] As a further solution, in step 8, when the PPT data corresponding to the time zone time is queried in the local cache, the PPT data is queried and output according to the principle of proximity in time.

[0013] On the other hand, the present invention provides a system for aligning PPT in teaching live broadcast based on a player, including a live broadcast acquisition unit, a server backend and a browser frontend; wherein, Live broadcast acquisition unit: used to collect live broadcast data and push live broadcast data to the server backend, including a live broadcast video acquisition module and a PPT collection module; wherein the live broadcast video acquisition module is used to collect live broadcast video signals, and the PPT collection module is used to collect PPT collection signals; Server backend: used to provide server backend services, including PPT server and video streaming server; wherein, the PPT server is used to provide PPT data, and the video streaming server performs video streaming based on the HLS protocol; Browser front-end: A player is deployed in the browser and the hls.js library is installed; wherein the player is used to play the video stream based on the HLS protocol in the browser and establish a WebSocket communication connection with the PPT server.

[0014] Compared with the related art, the method and system for aligning PPT for live teaching based on a player provided by the present invention has the following advantages: 1. The present invention solves the problem of video content and PPT being out of sync in online live teaching by realizing precise alignment of video images and PPT on the front end of the browser, thereby improving the coherence and logic of the teaching content; ensuring that users can obtain a smooth and synchronous live teaching experience in different network environments, reducing learning interruptions caused by freezes or delays, and enhancing user satisfaction.

[0015] 2. The present invention enhances the system's adaptability to network fluctuations and delays through timestamp proofreading and correction algorithms, and improves the stability and reliability of live broadcasting.

[0016] 3. The present invention reduces the need for complex processing on the server side by implementing alignment on the client browser, thereby reducing the computing and storage costs of the server; the method is easy to implement in the existing live teaching platform, does not require large-scale hardware upgrades or replacements, and is easy to deploy and apply quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the steps of a method for aligning PPT for live teaching based on a player provided by the present invention; Figure 2 A schematic diagram of the system flow for aligning PPT for live teaching based on a player provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0019] See also Figure 1 The present invention provides a method for aligning the teaching live PPT based on a player, and the teaching live PPT is aligned through the following steps: Step 1: The live broadcast collection unit collects live broadcast data during the teaching live broadcast process and uploads it to the server backend; wherein the live broadcast data includes video live broadcast signal and PPT recording signal; Step 2: The server backend receives live streaming data; the live video signal is converted into several m3u8 playlists according to the HLS protocol; the PPT recorded signal is processed by image recognition and converted into several PPT image data; Step 3: The server backend uses the principle of live stream pts increment to check the timestamps of the PPT recording signal and the live video signal, and keeps the timestamp increment through timestamp correction; Step 4: The server backend performs live teaching video broadcast; during the live video broadcast process, the live video signal is transmitted as a video stream in the form of several ts format video segments, and the ts format video segments in the m3u8 playlist are defined according to the time zone through the live broadcast feature of the HLS protocol; Step 5: The server backend generates corresponding PPT data; wherein, in the process of generating the PPT data, the generation time of the PPT image data is defined by the time zone time to obtain the corresponding PPT data; Step 6: The browser front end receives the live teaching video; during the video playback process, the browser front end continuously receives the video stream through the HLS protocol and plays the video stream in the browser; the TS segment information corresponding to the TS format video segment currently being played is obtained by monitoring the hls.js library; the TS segment information includes the segment path and definition rules; Step 7: During the process of receiving the live teaching video, the PPT server pushes the PPT data to the browser front end, and the browser front end caches the PPT data locally; Step 8: The browser front end obtains the TS segment information corresponding to the currently playing ts format video segment, finds the ts format video segment through the segment path, queries the corresponding time zone time according to the defined rules, queries the PPT data corresponding to the time zone time in the local cache, plays the corresponding PPT image data, and completes the live teaching PPT alignment.

[0020] It should be noted that most of the content of the live video signal in the teaching scene is the classroom teaching panorama or the teacher's lens; during the live teaching process, there will also be another live signal to record the PPT video (corresponding to the PPT recording signal), and during the live broadcast process, we can process the PPT recording signal into pictures one by one by recognizing the live broadcast information of the PPT; like Figure 2 As shown, in the process of receiving live data push stream at the server backend, we use the principle of increasing pts of live stream through the server backend, and check the timestamps of PPT video signal and classroom video signal to ensure the timestamp increase and correction of PPT signal and video signal; in a specific embodiment, the correction algorithm is set as: According to rule 1: set the original jump area [-inf,0)+(500,inf) to (-inf,-250)+(250,inf), that is, as long as the timestamp jitter is within 250ms; According to Rule 2: If the timestamp is incremented by more than 10ms, the video signal modulation is modified to 10ms. The continuity of video signal transmission and data consistency between the video signal and the PPT signal are guaranteed through video stream timestamp correction.

[0021] In the process of implementing live video broadcast, we use the characteristics of HLS protocol live broadcast to define the TS of the fragments in the m3u8 playlist according to Beijing time; if the time when the current live broadcast TS generates fragments is Beijing time 2022-10-1000:00:00, then we name the TS as 1665331200.ts (that is, the timestamp value of this time). In order to improve accuracy, the time of file generation can be reduced to 1 second or 2 seconds; the content of the m3u8 playlist generated by HLS is as follows: #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:316576 #EXT-X-TARGETDURATION:2 #EXT-X-DISCONTINUITY #EXTINF:1.968,nodesc d45922a0a2fdb9bdd42332c443418bcd / 1667548890096.ts #EXT-X-DISCONTINUITY #EXTINF:1.946,nodesc d45922a0a2fdb9bdd42332c443418bcd / 1667548902122.ts.

[0022] Among them, m3u8 playlist is usually used for streaming media transmission and is part of HTTP Live Streaming (HLS protocol) technology; HTTP Live Streaming is a streaming media transmission protocol developed by Apple for real-time transmission of audio and video content on the Internet; its main feature is to divide the entire video stream into short ts segments and use m3u8 playlist files to indicate the order and position of these segments.

[0023] The m3u8 playlist refers to an M3U file in UTF-8 encoding format. It is actually a text file containing multiple URLs, each of which points to an audio or video segment. These segments can be segmented, so that the streaming media can be transmitted on demand, thus achieving adaptive bitrate.

[0024] When starting a streaming session, the client will download an m3u8 playlist file containing metadata to find available media streams; HLS only requests basic HTTP messages. Unlike the Real-time Transport Protocol (RTP), HLS can pass through any firewall or proxy server that allows HTTP data to pass; it is also easy to use a content distribution network to transmit media streams; in this process, we adjust the defined timestamps according to needs, so as to use the background marking time information as a benchmark to provide a basic basis for subsequent PPT alignment.

[0025] In the process of generating PPT data, we use the server's Beijing time to define the time when the PPT material is generated. For example, if the time when the current recognized content is generated is 2022-10-1000:00:00 Beijing time, then we name the picture 1665331200.jpg (that is, the timestamp value of this time).

[0026] In the hls.js library, the Hls.Events.FRAG_CHANGED event is triggered when the fragment matching the current video position changes. Specifically, this event provides information about the changes in the media fragment being played, which is very useful for tracking specific fragment changes in the media stream; when the player switches from one fragment to another, this event is triggered, allowing developers or applications to respond to or handle such changes accordingly; the event data object contains the currently playing fragment object and its corresponding ID name.

[0027] Therefore, when the browser player is broadcasting live on the Internet through the HLS protocol, the browser front-end can obtain the information of the TS video fragment currently being played by listening to the Hls.Events.FRAG_CHANGED event of the hls.js library, that is, obtain the fragment path (parse the path of the currently playing fragment object) and the naming rule (parse the ID name) and other contents; Among them, the hls.js library is a JavaScript library used to implement video streaming playback based on the HTTP a Live Streaming (HLS) protocol on web pages; it provides a simple and easy-to-use interface that enables developers to easily embed HLS video streams into web pages and play them through HTML5 video tags.

[0028] HLS is a streaming protocol originally developed by Apple for smooth video playback on iOS and macOS devices. It splits the entire video file into small, continuous TS (TranspontSteam) segments and uses M3U8 playlist files to instruct the player to load and play these segments. The hls.js library implements the function of playing HLS video streams in various browsers by parsing M3U8 files and loading TS segments.

[0029] WebSocket is a protocol for full-duplex communication on a single TCP connection, allowing the server to actively push data to the client. In WebSocket, the browser and the server only need to complete a handshake to directly create a persistent connection between the two and perform two-way data transmission; during the live broadcast, the browser and the PPT service establish WebSocket communication, so that the PPT server can push the converted image information of the video signal to the browser, and treat the browser cache as a browser front-end data container; The browser front end obtains the TS segment information and uses the timestamp of the current TS segment as a reference to search for the matching PPT in the browser memory and output it nearby according to the playback content, thereby achieving alignment between the video screen and the PPT handouts. Based on the user's actual usage environment and the actual playback progress seen by the user, the playback progress information is collected in real time and aligned with the online handout (PPT) to improve the user's viewing experience during the learning process. This overcomes the problem of video and PPT being unable to align due to network environment, backend source return, deployment, and other reasons.

[0030] In summary, the present invention takes the background marking time information as the benchmark and the browser front-end dynamic alignment as the main implementation method, which can effectively solve the problem of alignment between live video screen and PPT handouts; after testing, the error between the video signal and PPT screen based on this scheme can be guaranteed to be within 3s; and the scheme has a low technical threshold, is simple to implement, and is easy to implement.

[0031] The above embodiments only express preferred implementation modes, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A method for aligning PPT for live teaching based on a player, characterized in that: Perform the following steps to align the live teaching PPT: Step 1: The live broadcast collection unit collects live broadcast data during the teaching live broadcast process and uploads it to the server backend; wherein the live broadcast data includes video live broadcast signal and PPT recording signal; Step 2: The server backend receives live streaming data; the live video signal is converted into several m3u8 playlists according to the HLS protocol; the PPT recorded signal is processed by image recognition and converted into several PPT image data; Step 3: The server backend uses the principle of live stream pts increment to check the timestamps of the PPT recording signal and the live video signal, and keeps the timestamp increment through timestamp correction; Step 4: The server backend performs live teaching video broadcast; during the live video broadcast process, the live video signal is transmitted as a video stream in the form of several ts format video segments, and the ts format video segments in the m3u8 playlist are defined according to the time zone through the live broadcast feature of the HLS protocol; Step 5: The server backend generates corresponding PPT data; wherein, in the process of generating the PPT data, the generation time of the PPT image data is defined by the time zone time to obtain the corresponding PPT data; Step 6: The browser front end receives the live teaching video; during the video playback process, the browser front end continuously receives the video stream through the HLS protocol and plays the video stream in the browser; the TS segment information corresponding to the TS format video segment currently being played is obtained by monitoring the hls.js library; the TS segment information includes the segment path and definition rules; Step 7: During the process of receiving the live teaching video, the PPT server pushes the PPT data to the browser front end, and the browser front end caches the PPT data locally; Step 8: The browser front end obtains the TS segment information corresponding to the currently playing ts format video segment, finds the ts format video segment through the segment path, queries the corresponding time zone time according to the defined rules, queries the PPT data corresponding to the time zone time in the local cache, plays the corresponding PPT image data, and completes the live teaching PPT alignment.

2. According to a method for aligning PPT in live teaching based on a player according to claim 1, it is characterized in that: In step 3, the timestamp is corrected by a correction algorithm to keep the timestamp incremented; wherein the correction algorithm sets the following rules: Rule 1: Set the jitter range of the transition area and allow the timestamp to have negative values ​​in the transition area; Rule 2: When the time interval between adjacent timestamps is greater than the increasing threshold, the timestamps are adjusted until the time interval reaches the increasing threshold.

3. According to a method for aligning PPT in live teaching based on a player according to claim 1, it is characterized in that: The time zone is set to the regional standard time corresponding to the live broadcast area; wherein, China Standard Time is correspondingly set to Beijing time.

4. According to a method for aligning PPT in live teaching based on a player according to claim 1, it is characterized in that: In step 4, the time zone of the ts format video segment is defined according to the naming rules.

5. According to a method for aligning PPT in live teaching based on a player according to claim 1, it is characterized in that: In step 6, the TS segment information corresponding to the currently playing TS format video segment is obtained by monitoring the Hls.Events.FRAG_CHANGED event of the hls.js library.

6. The method for aligning PPT in live teaching based on a player according to claim 1, characterized in that: In step 8, when the PPT data corresponding to the time zone is queried in the local cache, the PPT data is queried and output according to the principle of proximity in time.

7. A system for aligning PPT in live teaching based on a player, characterized in that: It includes live broadcast collection unit, server backend and browser frontend; among them, Live broadcast acquisition unit: used to collect live broadcast data and push live broadcast data to the server backend, including a live broadcast video acquisition module and a PPT collection module; wherein the live broadcast video acquisition module is used to collect live broadcast video signals, and the PPT collection module is used to collect PPT collection signals; Server backend: used to provide server backend services, including PPT server and video streaming server; wherein, the PPT server is used to provide PPT data, and the video streaming server performs video streaming based on the HLS protocol; Browser front-end: A player is deployed in the browser and the hls.js library is installed; wherein the player is used to play the video stream based on the HLS protocol in the browser and establish a WebSocket communication connection with the PPT server.

Images