An interactive recording and live streaming method and system based on B / S architecture
By separating the HTML content layer and canvas drawing layer in a B/S architecture, user operations and audio data are recorded, solving the problems of high bandwidth, high cost, and large recording files in existing technologies. This achieves low-latency, high-definition interactive recording and live streaming, supporting multi-user interaction and cross-platform operation.
Patent Information
- Application Number
- CN202611343927.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing interactive online learning systems require high bandwidth, are costly, have high latency, lack interactivity, and produce large recording files, making them difficult to meet the needs of large-scale use cases.
It adopts an interactive recording and live streaming method based on B/S architecture. By separating the HTML content layer and the canvas drawing layer, it records user operations and audio data, establishes timeline synchronization, generates structured operation data packages, and supports cross-platform operation and multimedia resource display.
It achieves low latency, low bandwidth, and high definition recording and live streaming, supports multi-user interaction, has small recording file size, runs across platforms, requires no plugins, and features an interactive playback page with separation of content display and drawing operations.
Smart Images

Figure CN123123518A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an interactive recording and live streaming method and system based on a B / S architecture, and more particularly to a recording and live streaming technology that separates operation records, whiteboard writing processes, and displayed content for processing, achieving precise synchronization of operation and sound via a timeline, and supporting interactive content during playback / live streaming. This invention features small file size, low latency, high definition, and cross-platform operation, supports real-time interaction and secondary editing, and can be widely used in fields such as educational informatization, online education, remote conferencing, and live teaching. Background Technology
[0002] With the rapid development of digital technology, online learning has become increasingly popular. Existing interactive online learning systems typically use video streaming, which requires high bandwidth, has high latency, places high demands on servers, and is costly and difficult to maintain. Furthermore, traditional video streaming lacks true interactivity, making communication and feedback difficult and impacting learning effectiveness.
[0003] Existing technologies also include some solutions based on operation recording rather than video streaming. Reference 1 (CN103646573A) discloses a method for generating a dedicated format file for a panoramic teaching system. It acquires real-time audio via a microphone and simultaneously captures actions on the current system interface via a mouse, keyboard, writing pad, or electronic whiteboard, recording them chronologically. Reference 2 (CN104156079A) discloses a learning trajectory recording method, which associates learning actions to generate a learning trajectory and sends it to a server. Reference 3 (CN105261252A) discloses a real-time action rendering method based on a panoramic learning platform system, combining action position coordinates, action feature codes, and narration audio to form a binary data stream for transmission or storage. Reference 4 (CN105405325A) discloses a network teaching method that collects image data, application data, and audio data for synchronous display and real-time recording. Reference 5 (CN105306861A) discloses a network teaching recording method and system that can record user operations using a multimedia whiteboard, speech, and interactions with other users.
[0004] The aforementioned existing technologies have the following drawbacks: they require specific hardware or software support, increasing the difficulty and barrier to entry for use; they can only record simple whiteboard writing and audio recording, and cannot fully process or display multimedia resources; they can only achieve simple one-way recording based on whiteboard writing, with weak interactivity, and during playback or live streaming, the process is usually rendered as a visual representation, preventing viewers from interacting with the content; the recorded files are relatively large, easily causing stuttering and high latency, placing high demands on servers, and making it difficult to meet the needs of large-scale use cases. In particular, the binary data stream transmission method used in the aforementioned existing technologies (such as Reference 3) uses unstructured data, which is not conducive to subsequent data acquisition, analysis, and application.
[0005] Therefore, there is an urgent need for a new recording and live streaming technology that can achieve low bandwidth, low latency, small file size, high definition, support flexible interaction and multiple resource displays, reduce dependence on dedicated hardware, support cross-platform operation, and support both live streaming and on-demand playback. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides an interactive recording and live streaming method and system based on a B / S architecture.
[0007] (I) Technical Solution
[0008] 1. Interactive recording and playback method
[0009] This invention provides an interactive recording and playback method based on a B / S architecture, comprising the following steps:
[0010] Content construction steps: Construct a content display area, which includes an underlying HTML content layer and a canvas drawing layer covering the HTML content layer. The HTML content layer is used to carry content data, and the canvas drawing layer is used for drawing operations; the display content and drawing operations are spatially separated.
[0011] The HTML content layer is used to carry and display content data such as text, images, audio, video, tables, and template elements. The canvas drawing layer is used for drawing, and its drawing API encapsulates visualization tools for writing text, drawing lines, graphics, charts, and animations. The combination of the HTML content layer and the canvas drawing layer allows the entire page to extend vertically, displaying and writing any content without requiring page turning or page skipping.
[0012] Synchronous recording steps: The user's drawing operation data is collected through the canvas drawing layer; the user's editing operation data is collected by listening to user operation events on the HTML content layer; and audio data is recorded simultaneously. A timeline based on time progress is established, the current time node is obtained, and the drawing operation data, the editing operation data, and the user's identity identifier are bound to the current time node to generate a structured operation data packet. The structured operation data packet and the audio data are then sent to the server respectively.
[0013] The content data refers to the static resource content carried in the HTML content layer, including text, images, audio, video, tables, and template elements. This content data, as part of the recording data, is stored in association with the structured operation data packet and the audio data, enabling the complete reconstruction of the HTML content layer during playback without relying on any external video files.
[0014] Storage Steps: The structured operation data packet, the content data of the HTML content layer, and the audio data are associated and stored together to generate recording data containing the structured operation data packet, the content data, and the audio data. The structured operation data packet and the audio data are mapped together via the timeline to synchronize operations with the audio during playback. "Associated storage" refers to establishing a mapping relationship between the structured operation data packet and the audio data, enabling the location and restoration of the corresponding operation data based on time points during playback. For example, associated storage may involve storing the structured operation data packet and the audio data together using a common file identifier.
[0015] Playback steps: The recorded playback data is parsed, and based on the content data within the recorded playback data, the HTML content layer is reconstructed on the user's browser. The drawing operations on the canvas drawing layer are then synchronously restored according to the order and type of the data. The HTML content layer remains interactive during playback. This interaction includes user actions and responses to interactive elements within the HTML content layer. During playback, when the playback progress reaches the position recorded in the structured operation data packet, the corresponding drawing instructions are executed on the canvas drawing layer. The user can control playback, fast forward, pause, stop, volume, and jump to the next playback position.
[0016] 2. Interactive Live Streaming Methods
[0017] This invention provides an interactive live streaming method based on a B / S architecture, comprising the following steps:
[0018] Content construction steps: Construct a content display area, which includes an HTML content layer at the bottom and a canvas drawing layer covering the HTML content layer. The HTML content layer is used to carry content data, and the canvas drawing layer is used for drawing operations. The display content and drawing operations are separated in space.
[0019] Synchronous data collection steps: User drawing operation data is collected through the canvas drawing layer; user editing operation data is collected by listening to user operation events on the HTML content layer; and audio data is collected simultaneously. A timeline based on time progress is established, the current time node is obtained, and the drawing operation data, the editing operation data, and the user's identity identifier are bound to the current time node to generate a structured operation data package.
[0020] Real-time transmission step: The structured operation data packet and the audio data are sent to one or more remote clients in real time.
[0021] Synchronous rendering steps: The remote client, based on the timeline and the order and type of the data, and in conjunction with the content data of the HTML content layer, reconstructs the HTML content layer and synchronously restores the drawing operations on the canvas drawing layer, while simultaneously playing the audio data. The HTML content layer remains interactive during rendering, and the interaction includes user operations and responses to interactive elements within the HTML content layer. The content data can be pre-loaded or shared by the remote client, or it can be sent to the remote client along with the structured operation data packet.
[0022] 3. Interactive recording and live streaming system
[0023] This invention also provides an interactive recording and live streaming system based on a B / S architecture, comprising:
[0024] The content module is used to construct a content display area, which includes an underlying HTML content layer and a canvas drawing layer overlaid on the HTML content layer. The HTML content layer carries content data, and the canvas drawing layer is used for drawing operations. The acquisition module is used to acquire user drawing operation data through the canvas drawing layer, acquire user editing operation data by listening to user operation events on the HTML content layer, and simultaneously acquire audio data. It establishes a timeline based on time progress, obtains the current time node, and binds the drawing operation data, the editing operation data, and the user's identity identifier to the current time node to generate a structured operation data packet. The processing module... The system is used to establish a mapping relationship between the structured operation data packet, the content data of the HTML content layer, and the audio data through the timeline for associated storage to generate recording data containing the structured operation data packet, the content data, and the audio data, and / or to send the structured operation data packet and the audio data to one or more remote clients in real time; the rendering module is used to parse the recording data or receive the structured operation data packet, reconstruct the HTML content layer on the user's browser based on the content data in the recording data, synchronously restore the drawing operations on the canvas drawing layer according to the order and type of the data, and play the audio data, wherein the HTML content layer remains interactive during playback or rendering.
[0025] The method described in this invention can be implemented based on browser technology, with all control logic implemented by JavaScript. It can be encapsulated as a standard plugin for other web applications to call, and the backend can be encapsulated as an API for other platforms to call. Alternatively, the computer program implementing the method can be stored in a computer-readable storage medium and executed by a processor.
[0026] 4. Preferred Further Solutions
[0027] The present invention may also include one or more of the following preferred further embodiments:
[0028] (1) The height and width of the canvas drawing layer are the same as the height and width of the HTML content layer, respectively. It is fixedly positioned to completely cover the HTML content layer. When using text-based tools, the pointer-events style of the canvas drawing layer is set to none to allow the HTML content layer to be manipulated. When using drawing tools, the pointer-events style is restored to auto to allow the canvas drawing layer to receive drawing operations. In this way, the two layers are superimposed on each other without interfering with each other.
[0029] (2) The timeline is based on the time progress of the audio data, and the current time node is the current playback duration of the audio data, in milliseconds. During the recording process, the system records the presenter's voice as the main audio data throughout, and the time nodes bound to all structured operation data packets are based on this voice duration. Using voice as the timeline, precise alignment of drawing operations, editing operations, interactive operations, and audio is achieved, and multi-channel content input is supported.
[0030] (3) The synchronous recording step or synchronous acquisition step further includes collecting user interaction event data, including answering events, questionnaire events, live streaming events, screen sharing events, raising hand events or asking questions events; binding the interaction event data with the user identity identifier and the current time node, and generating the structured operation data package together.
[0031] (4) The drawing operation data includes handwriting data and geometric data; the handwriting data includes at least coordinate position, line width, and color; the geometric data includes at least coordinate position. The coordinate position is a relative coordinate position relative to the content display area.
[0032] (5) In the playback step, the user can control playback, fast forward, pause, stop, volume and jump to the playback position, and can individually control whether the voice and operation of one or more users are played, including the presenter and himself.
[0033] (6) In the playback step, the HTML content layer is completely reconstructed on the user's browser. The interactive elements within the HTML content layer remain operable, allowing the user to edit the playback content to create private notes. These private notes are only visible to the current user. Since playback completely restores the HTML content layer to the user's browser, rather than rendering it as an inoperable video screen, the interactive elements on the page—including buttons, forms, template elements, and labels—maintain their original interactivity during playback. This is a key feature that distinguishes this invention from traditional video playback. The viewer's secondary editing data is stored in a directory named after the user's ID, accessible only to the current user.
[0034] In addition, the present invention may also include one or more of the following preferred further embodiments:
[0035] (7) The HTML content layer is used to carry text, images, audio, video, tables, or template elements; resources with path attributes only reference their paths, and the user chooses whether to load the resources during playback. The advantages of referencing paths are: first, the user can selectively decide whether to load these resources during playback, especially in scenarios with poor network conditions or where data saving is required; second, it can reduce the size of the recorded file; third, audio, video, and other resources themselves can hardly be compressed further, and referencing paths will not increase the size of the final recorded file. Moreover, these resources only occupy storage space once and can be called in different recording scenarios as materials, which saves storage space to a great extent and improves resource utilization.
[0036] (8) The template elements are preset HTML-based combinations. Each template element has predefined styles, layouts and attributes, as well as special effects, animation effects and interactive functions. The template elements are implemented by transparent PNG images with CSS and JS, and are built into the subsystem. By dragging the template elements, HTML elements with preset effects can be quickly created. By double-clicking, the text content of the template elements can be modified. By clicking to select the template elements, an attribute setting panel will pop up, allowing users to further customize the styles, layouts and attributes.
[0037] (9) The structured operation data packet is structured data in JSON format. Each operation data packet includes at least the tool type, mouse status code, mouse event data, user identity identifier and current time node.
[0038] (10) The drawing operation data is collected at a preset sampling frequency, which is approximately once every 300 milliseconds. During playback, a Bézier curve is used to connect adjacent coordinate points to achieve a smoothing effect. According to browser characteristics, 1000 events are triggered per second when drawing on the canvas. If each event is processed and recorded, it will seriously affect performance and generate a large amount of redundant data. It is more reasonable to record once every 300 milliseconds. During playback, for every three adjacent coordinate points, the smoothing curve segment is calculated using the Bézier curve formula, starting from the middle point and combining the adjacent points before and after. This process is repeated until all coordinate points are connected.
[0039] (11) The height of the container containing the HTML content layer automatically adjusts with the content and can extend vertically. A vertical scroll bar appears when it exceeds the visible area. Compared with traditional paginated courseware or whiteboard systems, the HTML content layer of this invention adopts a vertically infinite extension design. The page height automatically adjusts with the content and never turns pages. All teaching content is presented completely in the same continuous space. The presenter does not need to worry about the whiteboard being interrupted by page turning. The whiteboard is completely preserved, and the viewer can scroll up at any time to review the previous content, maintaining the continuity, integrity, and traceability of the teaching content.
[0040] (12) The user identity identifier includes a user identifier and / or a user role, used to distinguish the drawing operations, editing operations and interactive operations of multiple users. Binding the operation data to the user identity identifier and time node can: firstly, support the input of multiple content; secondly, support multi-party interactive data during live broadcast without worrying about data corruption; and thirdly, allow for individual control over whether to display and play the voice and operations of a specific user during playback.
[0041] (13) During the live stream, viewers can review previously displayed content by scrolling the vertical scroll bar. In live stream mode, users can watch the live stream by scrolling the vertical scroll bar, just like browsing a webpage, providing a better user experience.
[0042] (14) The storage step stores the structured operation data packet and the audio data in separate directories as temporary files, named with timestamps accurate to milliseconds. Specifically, during the first storage, a folder named with a timestamp is created as the root directory, and a json folder and an audio folder are created in the root directory; in recording mode, the received structured operation data packet is named with a timestamp accurate to milliseconds and stored in the json folder under the root directory with the json suffix, and the received audio data is named with a timestamp accurate to milliseconds and stored in the audio folder under the root directory with the mp3 suffix; when recording is interrupted, recording can continue from the last recording position; when the recording mode ends, the files in the json folder under the root directory are first merged into a json file in order, and the files in the audio folder are merged into an mp3 file in order, and saved to the root directory. Then, the user information, recording information and root directory information are written to the database, and finally the json folder and audio folder are deleted. The above design achieves three objectives: first, recording can be resumed in case of interruption; second, files can be merged and temporary files can be deleted to save space when recording ends; and third, it facilitates distributed storage and reduces database operations, requiring only the corresponding data to be written to the database at the beginning and end of recording.
[0043] (15) The method further includes a display step: generating a complete image of the page content at the end of the recorded demonstration, with floating icons for key points, difficulties, common mistakes, and annotations marked during the demonstration. Users can click on the icons to view the corresponding explanatory content or jump to the playback position of the explanation marked; and it supports switching to pure content mode to view all the marked and annotated content in text form. Among them, the key points are marked with a red fluorescent pen, and the area drawn by the pen is marked as key points; the difficulties are marked with a green fluorescent pen, and the area drawn by the pen is marked as difficult; the common mistakes are marked with a blue fluorescent pen, and the area drawn by the pen is marked as common mistakes; the annotations are displayed in the form of auto-incrementing numerical sequences at the position where they need to be displayed, and are used to write notes, questions, or discussions.
[0044] (16) The method also includes a speaker pattern: the presenter’s speech information is not broadcast to the viewers.
[0045] (17) During the live broadcast, the presenter sets the room name and password to create a live broadcast room and obtains the room number; the viewer enters the live broadcast room through the room number and password.
[0046] (II) Beneficial Effects
[0047] Compared with the prior art, the present invention has the following advantages:
[0048] 1. Small file size and low latency. Unlike video streaming technology, this invention only records user-triggered operation data and audio data. Based on a 45-minute lesson, traditional recording data is around 1.5GB, while the final recording file generated by this invention is only about 10MB to 15MB, with latency down to the millisecond level.
[0049] 2. Precise synchronization between operation and audio. This invention establishes a timeline based on time progress, binding all operation data to the current time node, thus achieving precise alignment between drawing operations, editing operations, interactive operations, and audio.
[0050] 3. Interactive playback page. This invention fully restores the HTML content layer to the user's device during playback or live streaming, keeping interactive elements on the page operational. Users can still interact with page elements even after pausing, a fundamental difference from traditional video playback. It also supports secondary editing, allowing viewers to create their own private notes based on the recorded content.
[0051] 4. Separation of content display and drawing operations. This invention achieves physical separation of static content display and dynamic drawing operations in space through a two-layer architecture of HTML content layer and canvas drawing layer, so that the two do not interfere with each other.
[0052] 5. Supports two-way interaction between presenters and viewers. Presenters can initiate interactive operations such as answering questions, surveys, connecting with viewers via voice chat, and sharing screens. Viewers can initiate interactive requests such as raising their hands, asking questions, connecting with viewers via voice chat, or sharing screens. Interaction data is bound to the current time point and stored together, and is synchronously restored according to the same timeline during playback.
[0053] 6. Supports multiple content streams and multiple users. By binding user identity identifiers with time nodes, it supports multiple user inputs, and during playback, it allows for individual control over whether to display and play a specific user's voice and actions.
[0054] 7. Cross-platform operation, no installation required. This invention is based on a B / S architecture, is not limited by the terminal, and does not require the installation of any plugins.
[0055] 8. The content in the display area is not paginated and can be extended infinitely vertically without erasing the blackboard, allowing for a complete display of the content. Users can view the content horizontally (by the progress bar) or vertically (by the page's vertical scroll bar), and can review previous and subsequent content using the vertical scroll bar while the content is playing or paused.
[0056] 9. The content is based on native HTML, so there are no clarity issues. It can be edited online without relying on third-party software. The content can be searched and supports indexing. There is zero learning cost. High-quality content display pages with rich animation effects and interactive functions can be quickly built using template elements. Attached Figure Description
[0057] Figure 1 This is a schematic diagram of the recording and broadcasting system demonstration end, where 1 is the HTML content layer, 2 is the canvas drawing layer, 3 is the screen visible area, 4 is the vertical scroll bar, 5 is the height of the page scrolling off the screen, 6 is the screen visible height, 7 is the toolbar, 8 is the sound recording progress bar, 9 is the template element attribute setting panel, and 10 is the floating layer.
[0058] Figure 2 This is a schematic diagram of the process of collecting whiteboard data. In the diagram, 1, 2, and 3 represent coordinate positions, 4 is the height of the page scrolling off the screen, 5 is the vertical height of point 1 from the top left corner of the screen, 6 is the horizontal width of point 1 from the top left corner of the screen, 8 is the canvas drawing layer, 9 is the sound recording progress bar, 10 is the vertical scroll bar, and 11 is the visible area of the screen.
[0059] Figure 3 This is a flowchart illustrating the entire recording and broadcasting system's operation. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0061] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0062] I. Construction of Content Display Area (Two-Tier Architecture)
[0063] like Figure 1 As shown, the content display area of this invention consists of two parts: 1 in the figure is the HTML content layer, and 2 in the figure is the canvas drawing layer.
[0064] The HTML content layer is located at the bottom layer and is used to carry and display content data such as text, images, audio, video, tables, and template elements. The height of the container containing the HTML content layer automatically adjusts with the content, can extend vertically, and displays a vertical scrollbar when it exceeds the visible area.
[0065] The canvas drawing layer overlays the HTML content layer, using fixed positioning. Its height and width are the same as the HTML content layer, completely covering it. When using text-based tools, the canvas's `pointer-events` style is set to `none`, allowing the user to manipulate the underlying HTML content. When using drawing tools, this style is restored to `auto`, enabling the canvas to receive drawing operations. Through these settings, the two layers overlap without interfering with each other.
[0066] The canvas drawing layer encapsulates visualization tools on top of its drawing API, used for writing text, drawing lines, graphics, charts, and animations. The combination of the HTML content layer and the canvas drawing layer allows the entire page to extend vertically, displaying and writing any content without requiring page turns or page breaks.
[0067] In the HTML content layer, resources with path attributes (such as audio and video) only reference their paths. Referencing paths has three main advantages: First, users can selectively decide whether to load these resources during playback, especially in scenarios with poor network conditions or where data saving is required; second, it reduces the size of the recorded file; and third, audio and video resources are virtually impossible to further compress. Directly inserting them into the recorded file would undoubtedly increase the file size, while referencing them not only avoids increasing the final recorded file size but also ensures that these resources only occupy storage space once and can be used as source material in different recording scenarios, significantly saving storage space and improving resource utilization.
[0068] This system includes built-in template elements. Template elements are pre-defined HTML-based combinations, each with predefined styles, layouts, and attributes, as well as effects, animations, and interactive features. These effects, animations, and interactive features are implemented using transparent PNG images combined with CSS and JavaScript. Users can quickly create HTML-based pages with animations, effects, and interactivity by dragging and dropping templates. Double-clicking a template element allows modification of its text content; clicking on a selected element brings up a property settings panel where the style and attributes of each element can be customized to user needs, such as font color, size, animation method, interaction method, border style, shadow style, and transparency. This simplifies the presentation content creation process, eliminating the need for traditional software like Word, PDF, or PowerPoint. Furthermore, because it's based on CSS and JavaScript, each template element requires very little HTML, so even with extensive use, the recording file size remains unaffected.
[0069] II. Specific Implementation of the Recording Process
[0070] like Figure 2 As shown, the process of obtaining the blackboard notes is as follows:
[0071] Step 1: The user selects a drawing tool, and the recording function captures the mouse press status code, the tool's identifier, the user's identifier, and the mouse release status code.
[0072] Step 2: The user clicks on the whiteboard area to obtain the starting point (e.g., ...). Figure 2 As shown in point 1, the recording function captures the mouse press status code, the horizontal distance of the starting point relative to the left edge of the browser, the vertical distance of the starting point relative to the top edge of the browser, the user's identity identifier, and the current voice duration.
[0073] Step 3: The user holds down the mouse button and moves the mouse to draw the path (e.g., ...). Figure 2As shown in points 2 and 3), the recording function captures data of each coordinate point in the path: the status code of the mouse being pressed and moved, the horizontal distance of the coordinate point relative to the left edge of the browser, the vertical distance of the coordinate point relative to the top edge of the browser (including the height that has scrolled off the screen), the user's identity identifier, and the current voice duration.
[0074] Step 4: When the mouse is released, the recording function captures the mouse release status code, user identity, and current voice duration.
[0075] The coordinate positions mentioned above are relative coordinate positions, used to characterize the relative position of the drawing point with respect to the content display area.
[0076] To optimize performance, the recording sampling frequency is approximately once every 300 milliseconds. Based on browser characteristics, drawing on the canvas triggers 1000 events per second. Processing and recording every single event would severely impact performance and generate a lot of redundant data. Through repeated experiments, triggering events approximately once every 300 milliseconds is deemed more reasonable. During playback, to achieve a smooth effect, Bézier curves are used to connect every three coordinate points: for every three adjacent points, starting from the middle point and combining the preceding and following points, a smooth curve segment is calculated using the formula for Bézier curves, connecting the two adjacent points. This method is repeated to process adjacent three coordinate points sequentially until all points are connected.
[0077] The data captured by the recording function is assembled into JSON format. Each data entry contains values representing each tool, mouse status codes, data generated by mouse events, user identification, and voice recording duration. During the recording process, the system records the presenter's voice as the primary audio data. All time points bound to structured operation data packets are based on this voice duration, measured in milliseconds; that is, the timeline is based on voice.
[0078] During recording, the content data of the HTML content layer (i.e., static resource content, including text, images, audio, video, tables, and template elements) is collected and saved. This content data is stored as part of the recorded playback data, along with the structured operation data packet and the audio data. In this way, the HTML content layer can be completely reconstructed based on the content data in the recorded playback data during playback.
[0079] In addition to drawing operations, the system also collects user editing operations on HTML content. For example, when performing cursor operations in the HTML content editor, whether clicking anywhere or selecting a piece of text, the system records the specific position of the cursor in the HTML content editor; if a piece of text is selected, the recording module obtains the currently selected tool type and the starting position of the text in which paragraph, line, and element, including an array generated across paragraphs, lines, and tags; all operations involving adding, modifying, and deleting text are performed under the premise of text selection. The recording module combines the collected data and the length of the audio file into a JSON and sends it to the server at regular intervals. The server stores the received data and broadcasts it to all viewing users. After receiving the data, the user performs the corresponding selection operation.
[0080] For mouse actions that browsers do not allow to be recreated programmatically, corresponding methods can be written to trigger them. For example, each element can be bound to an ID. When an element is clicked, the recording module records the event tool as a selection tool, the event type as click, and the event ID as the element ID and user ID. This data, along with the recording duration, is then bound to a JSON file and sent to the server. Upon receiving the data, the viewer calls the execution method, triggering the click event corresponding to the ID, thus recreating the action. The presenter's mouse trajectory, generated as JSON data with coordinates and recording duration, is broadcast to the viewer. The user's client uses a layer with a mouse cursor shape, set to a fixed position, and adjusts the left and top coordinates based on the parsed JSON data to simulate the mouse movement trajectory.
[0081] The purpose of this design is to provide the following advantages: First, it supports multi-party data interaction during live streaming without worrying about data corruption; second, if a recording is unexpectedly interrupted for some reason, it can resume recording from the last recording position; third, it is easy to merge the data when recording ends; fourth, it can quickly locate the starting point when performing fast-forward during playback. When the fast-forward position is at the end of an operation (such as a line segment) without a starting point, it is necessary to find the starting point data of the operation and start drawing from the starting point data; fifth, during playback, it is possible to individually control whether to enable the voice and operation of one or more users.
[0082] III. Playback based on the speech timeline
[0083] During playback, the playback module parses the recorded data and first reconstructs the HTML content layer based on the content data in the recorded data.
[0084] The playback engine is driven by the aforementioned voice data. When the playback progress reaches the voice duration position recorded in the structured operation data packet, it executes the corresponding drawing instructions on the canvas drawing layer. Since all operation data is bound to the current time node and uses voice as a unified timeline, precise alignment of drawing operations, interactive operations, and audio is achieved.
[0085] During playback, users can control playback, fast forward, pause, stop, and volume, and can also control whether each user's voice and operation are played.
[0086] Because playback restores the entire HTML content layer to the user's browser, rather than rendering it as an uninterrupted video, interactive elements on the page—including buttons, forms, template elements, and labels—retain their original interactivity during playback. Playback progress only controls the timing of the restoration of drawing operations on the canvas layer and the playback progress of audio data; it does not affect the interactivity of the HTML content layer itself. Viewers can still manipulate page elements during playback. Viewers can further edit the recorded content to create their own unique notes. This edited data is stored in a directory named after the user's ID and is accessible only to the current user. This is a key feature that distinguishes this invention from traditional video playback.
[0087] IV. Processing of Multi-User Interaction Data
[0088] During recording or live streaming, presenters can initiate interactive operations such as answering questions, conducting surveys, connecting with viewers, and sharing screens. Viewers can initiate interactive requests such as raising their hands, asking questions, connecting with viewers, or sharing screens. Furthermore, during live streaming, presenters can also lock the viewer's interface, microphone, camera, keyboard, or mouse. The locking operation event is bound to the user's identity identifier and the current time point, and together they generate the structured operation data packet.
[0089] The aforementioned interactive event data is bound to the user's identity and the current time point, and a structured operation data packet is generated and stored or sent together. During playback, the interactive events and drawing operations are synchronously restored along the same timeline. During live streaming, multi-party interactive data is supported, and there is no need to worry about data corruption.
[0090] V. Storage of Recorded Broadcast Data
[0091] The storage module is used to receive, process, and store recorded data. The recorded data, including operation data, interaction data, audio data, and editing data, is initially stored as temporary files in corresponding directories. This serves the following purposes: first, recording can be resumed in case of interruption; second, files can be merged and temporary files can be deleted to save space when recording ends; and third, it facilitates distributed storage and reduces database operations, as the corresponding data only needs to be written to the database at the beginning and end of recording for on-demand use.
[0092] Specifically, each recording generates a unique overall recording directory containing data on presenters and participants, which is aggregated together. Upon initial storage, a folder named after a timestamp is created as the root directory, and within this root directory, two folders, "json" and "audio". During recording mode, received structured operation data packets are named with timestamps accurate to milliseconds and stored in the "json" folder under the root directory with the "json" extension. Received audio data is named with timestamps accurate to milliseconds and stored in the "audio" folder under the root directory with the "mp3" extension. When recording mode ends, the files in the "json" folder are merged into a single "json" file, and the files in the "audio" folder are merged into a single "mp3" file, both saved to the root directory. Then, user information, recording information, and root directory information are written to the database. Finally, the "json" and "audio" folders are deleted.
[0093] Viewers' secondary edits during playback are stored in a directory named after their user ID. When a user finishes editing, the final data is merged together. Since viewer-edited data is only visible to them, it doesn't need to be written to the database. During playback, the program checks if a directory named after the user ID exists; if it does, it retrieves data with a fixed format and name; otherwise, it ignores it. The database needs to store very little data; the essential data is the presenter ID, course title, and course path.
[0094] VI. Specific Implementation of the Live Streaming Process
[0095] During the live stream, the presenter sets a room name and password, creates a live stream room, and obtains a room number; viewers enter the live stream room using the room number and password.
[0096] During the live stream, real-time data from presenters and participants is sent to the server and broadcast to all users. Upon receiving the data, the system parses and plays the audio, simultaneously reconstructs the operations based on the data's order and type, and renders and displays it in real-time on the browser. The HTML content layer data can be pre-loaded or shared by the remote client (e.g., pre-published HTML content loaded when viewers enter the live stream), or it can be sent to the remote client along with the structured operation data packet. Specifically, the remote client reconstructs the HTML content layer based on the timeline and the data's order and type, simultaneously reconstructing the drawing operations on the canvas layer and playing the audio data. The HTML content layer remains interactive during rendering, enabling interactive live streaming functionality.
[0097] During a live stream, viewers can scroll down a vertical scroll bar to review previously covered content, providing a better user experience.
[0098] VII. Comparison with existing technologies
[0099] This invention differs from video stream recording technology by recording only user-triggered operation data, thus significantly reducing file size.
[0100] Compared with existing operation recording-based technical solutions, the main differences of this invention are: First, existing solutions mainly record simple whiteboard paths, lacking a two-layer separation architecture of HTML content layer and canvas drawing layer, failing to achieve spatial separation between displayed content and drawing operations, and unable to fully process or display multimedia resources; Second, existing solutions mainly achieve one-way whiteboard demonstration and recording, rendering the process as a visual representation during playback or live streaming, lacking interactivity, and viewers cannot interact with page elements or perform secondary editing during playback or live streaming; Third, existing solutions fail to achieve precise synchronization of multi-user operations and audio using the current time node as a unified timeline, lacking a user identity binding mechanism. This invention, through a two-layer architecture, timeline synchronization, user identity binding, and playback reconstruction of the HTML content layer while maintaining interactivity, effectively supports real-time multi-user interaction and interactive playback while achieving low latency and small file size. Industrial applicability
[0101] This invention is based on a B / S architecture and can be widely applied in fields such as educational informatization, online education, remote conferencing, and live teaching.
[0102] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. An interactive recording and playback method based on a B / S architecture, characterized in that, include: Content construction steps: Construct a content display area, which includes an HTML content layer at the bottom and a canvas drawing layer covering the HTML content layer. The HTML content layer is used to carry content data, and the canvas drawing layer is used for drawing operations. The display content and drawing operations are separated in space. Synchronous recording steps: Collect user drawing operation data through the canvas drawing layer; collect user editing operation data by listening to user operation events on the HTML content layer; and record audio data simultaneously. Establish a timeline based on time progress, obtain the current time node, bind the drawing operation data, the editing operation data, and the user identity identifier to the current time node, and generate a structured operation data packet. Send the structured operation data packet and the audio data to the server respectively. Storage steps: The structured operation data packet, the content data of the HTML content layer, and the audio data are associated and stored to generate recording data containing the structured operation data packet, the content data, and the audio data. The structured operation data packet and the audio data are mapped through the time axis to achieve synchronization between operation and audio during playback. Playback steps: parse the recorded data, reconstruct the HTML content layer on the user's browser based on the content data in the recorded data, and synchronously restore the drawing operations on the canvas drawing layer according to the order and type of the data; the HTML content layer remains interactive during the playback process, and the interaction includes the user's operation and response to interactive elements in the HTML content layer.
2. An interactive live streaming method based on a B / S architecture, characterized in that, include: Content construction steps: Construct a content display area, which includes an HTML content layer at the bottom and a canvas drawing layer covering the HTML content layer. The HTML content layer is used to carry content data, and the canvas drawing layer is used for drawing operations. The display content and drawing operations are separated in space. Synchronous data collection steps: Collect user drawing operation data through the canvas drawing layer, collect user editing operation data of content by listening to user operation events on the HTML content layer, and collect audio data at the same time; establish a timeline based on time progress, obtain the current time node, bind the drawing operation data, the editing operation data and user identity identifier with the current time node, and generate a structured operation data package; Real-time transmission step: The structured operation data packet and the audio data are sent to one or more remote clients in real time; Synchronous rendering steps: The remote client reconstructs the HTML content layer and synchronously restores the drawing operations on the canvas drawing layer based on the timeline and the order and type of the data, combined with the content data of the HTML content layer, while playing the audio data; the HTML content layer remains interactive during the rendering process, and the interaction includes the user's operation and response to interactive elements in the HTML content layer.
3. An interactive recording and live streaming system based on a B / S architecture, characterized in that, include: The content module is used to construct a content display area, which includes an HTML content layer at the bottom and a canvas drawing layer covering the HTML content layer. The HTML content layer is used to carry content data, and the canvas drawing layer is used for drawing operations. The acquisition module is used to acquire user drawing operation data through the canvas drawing layer, acquire user editing operation data of content by listening to user operation events on the HTML content layer, and acquire audio data; establish a timeline based on time progress, obtain the current time node, bind the drawing operation data, the editing operation data and user identity identifier to the current time node, and generate a structured operation data packet; The processing module is used to associate and store the structured operation data packet, the content data of the HTML content layer and the audio data through the time axis to generate recording data containing the structured operation data packet, the content data and the audio data, and / or send the structured operation data packet and the audio data to one or more remote clients in real time; The rendering module is used to parse the recorded playback data or receive the structured operation data packet, reconstruct the HTML content layer on the user's browser based on the content data in the recorded playback data, synchronously restore the drawing operations on the canvas drawing layer according to the order and type of the data, and play the audio data; the HTML content layer remains interactive during playback or rendering.
4. The method according to claim 1 or 2, characterized in that, The height and width of the canvas drawing layer are the same as the height and width of the HTML content layer, and it is fixedly positioned to completely cover the HTML content layer. When using text tools, the pointer-events style of the canvas drawing layer is set to none so that the HTML content layer can be manipulated. When using drawing tools, the pointer-events style is restored to auto so that the canvas drawing layer can receive drawing operations.
5. The method according to claim 1 or 2, characterized in that, The timeline is based on the time progress of the audio data, and the current time node is the current playback duration of the audio data, in milliseconds.
6. The method according to claim 1 or 2, characterized in that, The synchronous recording or synchronous acquisition step further includes: acquiring user interaction event data, including answering events, questionnaire events, live streaming events, screen sharing events, raising hand events, or asking questions events; binding the interaction event data with the user identity identifier and the current time node, and generating the structured operation data package together.
7. The method according to claim 1 or 2, characterized in that, The drawing operation data includes handwriting data and geometric data; the handwriting data includes at least coordinate position, line width, and color; the geometric data includes at least coordinate position; the coordinate position is a relative coordinate position relative to the content display area.
8. The method according to claim 1, characterized in that, During the playback process, users can control playback, fast forward, pause, stop, volume, and jump to the playback position. They can also individually control whether the voice and actions of one or more users are played.
9. The method according to claim 1, characterized in that, In the playback step, the HTML content layer is completely reconstructed on the user's browser. The interactive elements in the HTML content layer remain operable. Users can overlay secondary editing data during playback to form private notes. The secondary editing data is independent of the original recording data and is only visible to the current user.
10. The method according to claim 1 or 2, characterized in that, The height of the container containing the HTML content layer automatically adjusts with the content, can extend vertically, and a vertical scrollbar appears when it exceeds the visible area.
Citation Information
Patent Citations
Dedicated format file generation method of panorama mode teaching system
CN103646573A
Learning path recording method and system
CN104156079A
Panoramic learning platform system-based real-time action rendering method
CN105261252A
Online teaching recording and playing method and system
CN105306861A
Network teaching method and system
CN105405325A
