Live streaming method, terminal device and system

Abstract

The embodiment of the application provides a live broadcast method, a terminal device and a system, the method comprises the following steps: collecting trajectory messages and audio signals according to a preset sampling period, obtaining trajectory data according to the trajectory messages, and processing the audio signals to obtain audio data; transmitting the trajectory data and the audio data to a second terminal through a first channel and a second channel respectively, the second terminal analyzes the trajectory data to obtain to-be-displayed trajectory messages, processes the audio data to obtain to-be-played audio signals, and makes the second terminal play the to-be-played audio signals and display the to-be-displayed trajectory messages, thereby improving the effect of remote teaching.

Description

Technical Field

[0001] This application relates to the field of live streaming technology. More specifically, it relates to a live streaming method, terminal device, and system. Background Technology

[0002] In remote training scenarios, instructors play courseware on their computers and transmit it synchronously to students' computers via the network, enabling students to learn remotely online.

[0003] In the current remote teaching process, teachers annotate and explain on the touch screen of the computer on the teaching end. However, the specific content of the annotations cannot be shared on the computer on the listening end. Students can only watch the teacher's explanation process through video, which affects the effectiveness of remote teaching. Summary of the Invention

[0004] The exemplary embodiments of this application provide a live streaming method, terminal device, and system, which solves the problem in the prior art that audio and annotation content cannot be synchronously shared on the listening computer, thereby improving the effectiveness of remote teaching.

[0005] In a first aspect, embodiments of this application provide a live streaming method, applied to a first terminal, comprising:

[0006] Trajectory messages and audio signals are collected according to a preset sampling period. Trajectory data is obtained based on the trajectory messages, and audio data is obtained by processing the audio signals.

[0007] The trajectory data and the audio data are synchronously transmitted to the second terminal through the first channel and the second channel, respectively, so that the second terminal can parse the trajectory data to obtain the trajectory message to be displayed, process the audio data to obtain the audio signal to be played, and then play the audio signal to be played and display the trajectory message to be displayed.

[0008] In one possible design, obtaining trajectory data based on the trajectory message includes:

[0009] The trajectory message is encoded according to a preset data frame format to obtain at least one trajectory frame;

[0010] Trajectory data is obtained by processing all trajectory frames according to a preset message format.

[0011] In one possible design, the acquisition of trajectory messages according to a preset sampling period includes:

[0012] Collect touch data from all touch points within a preset sampling period, and use the touch data from all touch points as trajectory messages. The touch data of each touch point includes at least one of the following: touch number, touch operation type code, touch point coordinate value, touch operation time corresponding to the touch point, and touch parameters. The touch operation type code is any one of the following: write operation type code, erase operation type code, undo operation type code, and back operation type code.

[0013] In one possible design, the preset data frame format is at least one of a write frame format, an erase frame format, an undo frame format, and a rollback frame format, wherein the touch operation type code of the write frame format is a write operation type code, the touch operation type code of the erase frame format is an erase operation type code, the touch operation type code of the undo frame format is an undo operation type code, and the touch operation type code of the rollback frame format is a rollback operation type code.

[0014] Accordingly, encoding the trajectory message according to a preset data frame format to obtain at least one trajectory frame includes:

[0015] The preset data frame format that is consistent with the touch operation type code contained in the touch data of each touch point in the trajectory message is determined as the target frame format corresponding to the touch point, wherein the target frame format is one of the writing frame format, erasing frame format, undo frame format and rollback frame format.

[0016] The touch data of each touch point is encoded according to the target data frame format corresponding to each touch point to obtain the trajectory frame corresponding to each touch point.

[0017] In one possible design, the writing frame format includes a touch number, a writing operation type code, coordinate values, time, note thickness parameters, and note color parameters;

[0018] The erase frame format includes touch number, erase operation type code, coordinate value, time, and erase size parameter;

[0019] The undo frame format includes the touch number, undo operation type code, and time;

[0020] The rollback frame format includes the touch number, rollback operation type code, and time.

[0021] In one possible design, the preset message format includes a total number of messages, a first message body to an Nth message body, wherein each message body includes a message length and message data, and N is an integer greater than 0;

[0022] Accordingly, the step of processing all trajectory frames according to a preset message format to obtain trajectory data includes:

[0023] The total number of messages is determined based on the number of all trajectory frames;

[0024] The message length of the message body corresponding to each trajectory frame is determined based on the data size contained in each trajectory frame, and the message data of the message body corresponding to each trajectory frame is determined based on the data of each trajectory frame.

[0025] The message body corresponding to each trajectory frame is determined based on the message length and message data of the message body corresponding to each trajectory frame. All message bodies are sorted according to the numbers contained in all trajectory frames to obtain the first message body to the Nth message body. The total number of messages and the first message body to the Nth message body are determined as trajectory data according to the preset message format.

[0026] In one possible design, the step of synchronously transmitting the trajectory data and the audio data to the second terminal via the first and second channels, respectively, includes:

[0027] The trajectory data and the audio data are processed using Pulse Code Modulation (PCM) encoding format to obtain a first audio stream and a second audio stream;

[0028] The first audio stream is transmitted to the second terminal via the first channel using a real-time message transmission protocol format, and the second audio stream is transmitted to the second terminal via the second channel using the same real-time message transmission protocol format. This enables the second terminal to parse the first audio stream to obtain the trajectory message to be displayed, and to parse the second audio stream to obtain the audio signal to be played.

[0029] Secondly, embodiments of this application provide a live streaming method applied to a second terminal, comprising:

[0030] The system receives trajectory data and audio data transmitted from the first terminal through the first channel and the second channel, respectively. The trajectory data is obtained by the first terminal based on the trajectory message, and the audio data is obtained by the first terminal processing the audio signal. The trajectory message and the audio signal are collected by the first terminal according to a preset sampling period.

[0031] The trajectory data is parsed to obtain the trajectory message to be displayed, and the audio data is processed to obtain the audio signal to be played.

[0032] Play the audio signal to be played and display the trajectory message to be displayed.

[0033] In one possible design, parsing the trajectory data to obtain the trajectory message to be displayed includes:

[0034] The trajectory data is parsed according to a preset message format to obtain at least one trajectory frame;

[0035] All trajectory frames are parsed according to a preset data frame format to obtain the trajectory message to be displayed, wherein the trajectory message to be displayed contains the touch data of all touch points to be displayed.

[0036] In one possible design, after playing the audio signal to be played and displaying the trajectory message to be displayed, the following is also included:

[0037] The touch trajectory is acquired at preset time intervals, and the touch trajectory is displayed and stored.

[0038] In response to the collected voice signal, the voice signal is sent to the first terminal, causing the first terminal to play the voice signal.

[0039] Thirdly, embodiments of this application provide a terminal device, which includes an audio acquisition device, a touch device, and a first terminal controller;

[0040] The touch device is used to collect trajectory messages according to a preset sampling period;

[0041] The audio acquisition device is used to acquire audio signals according to a preset sampling period;

[0042] The first terminal controller is configured to execute the live streaming method as described in the first aspect and any aspect thereof.

[0043] Fourthly, embodiments of this application provide a terminal device, which includes an audio playback device, a display device, and a second terminal controller;

[0044] The second terminal controller is configured to execute the live streaming method as described in the second aspect and any aspect thereof;

[0045] The audio playback device is used to play the audio signal to be played;

[0046] The display device is used to display the trajectory message to be displayed.

[0047] Fifthly, embodiments of this application provide a live streaming system, which includes a terminal device as described in the third aspect and at least one terminal device as described in the fourth aspect.

[0048] The live streaming method, terminal device, and system provided in this application embodiment collect trajectory messages and audio signals according to a preset sampling period on a first terminal. Trajectory data is obtained from the trajectory messages, and audio data is obtained by processing the audio signals. The trajectory data and audio data are synchronously transmitted to a second terminal via a first and a second channel, respectively. The second terminal parses the trajectory data to obtain the trajectory message to be displayed and processes the audio data to obtain the audio signal to be played. This allows the second terminal to play the audio signal and display the trajectory message to be displayed, enabling the listening computer to display annotations and play the audio signal in real time, thus improving the effectiveness of remote teaching. Attached Figure Description

[0049] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0050] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to an embodiment of the present invention.

[0051] Figure 2 An exemplary block diagram of the configuration of the control device is shown;

[0052] Figure 3 This is a hardware configuration block diagram of the control device provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the software configuration in a display device provided in an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the icon control interface of an application in a display device provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of a live streaming system provided in an embodiment of the present invention;

[0056] Figure 7 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 1 ;

[0057] Figure 8 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 2 ;

[0058] Figure 9 This is a schematic diagram of a preset data frame format provided in an embodiment of the present invention;

[0059] Figure 10 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 3 ;

[0060] Figure 11 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 4 ;

[0061] Figure 12 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 5 ;

[0062] Figure 13 A schematic diagram of the live streaming device provided in an embodiment of the present invention. Figure 1 ;

[0063] Figure 14 A schematic diagram of the live streaming device provided in an embodiment of the present invention. Figure 2 . Detailed Implementation

[0064] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0065] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0066] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0067] In this application, the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement the application in a sequence other than those given in the embodiments illustrated or described herein.

[0068] Furthermore, the terms “including” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0069] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0070] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in an embodiment of the present invention, such as... Figure 1 As shown, a user can operate the display device 100 via a mobile terminal 300 and a control device 200. The control device 200 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 100. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 100. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 100.

[0071] Figure 2 An exemplary block diagram of a control device according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a Wi-Fi chip, a Bluetooth module, NFC, or a replacement module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, buttons, or a replacement module.

[0072] Figure 3 This is a hardware configuration block diagram of a control device provided in an embodiment of the present invention. Figure 3The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0073] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. The user can input user commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0074] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, converting information between its internal form and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include at least one of the following visual interface elements: icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0075] Figure 4 This is a schematic diagram of the software configuration in a display device provided in an embodiment of the present invention, such as... Figure 4As shown, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer. The kernel layer contains at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0076] Figure 5 This is a schematic diagram of the icon control interface display of an application in a display device provided in an embodiment of the present invention, such as... Figure 5 As shown, the application layer contains at least one application whose corresponding icon control can be displayed on the monitor, such as: live TV application icon control, video-on-demand application icon control, media center application icon control, application center icon control, game application icon control, etc. The live TV application can provide live TV through different signal sources. The media center application can provide applications for playing various multimedia content. The application center can provide storage for various applications. The remote teaching application provides users with teaching software that can play courseware and make annotations.

[0077] In existing technologies, in remote training scenarios, instructors play courseware on their computers and transmit it synchronously to students' computers via the network, enabling students to learn remotely online. However, in current remote teaching processes, teachers annotate and explain on the touchscreen of their computers; students' computers cannot share the specific content of these annotations and can only watch the teacher's explanation via video, thus affecting the effectiveness of remote teaching.

[0078] To address the problem that existing methods cannot synchronously share annotations and audio on students' computers, this application provides a live streaming method, terminal device, and system. By setting up a first terminal to encode the collected trajectory messages according to a preset data frame format to obtain trajectory frames, and processing all trajectory frames according to the preset message format to obtain trajectory data, a second terminal is set up to parse the trajectory data to obtain trajectory messages and display them. This enables the synchronous display of annotations and audio on the students' computers, improving the effectiveness of remote teaching.

[0079] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0080] Figure 6 This is a schematic diagram of the structure of the live streaming system provided in an embodiment of the present invention, such as... Figure 6 As shown, the display device includes an image live streaming system comprising a first terminal 6100 and multiple second terminals 6200. The first terminal 6100 includes an audio acquisition device 6101, a touch device 6102, and a first terminal controller 6103. The second terminals 6200 include an audio playback device 6201, a display device 6202, and a second terminal controller 6203. For example, the first terminal 6100 is the display device for the teaching end, and the second terminals 6200 are the display devices for the listening end. Both the first terminal 6100 and the second terminals 6200 provided in this invention employ... Figures 1 to 5 The configuration of the control device, hardware, software, and icon control interface of the display device will not be described in detail here. For example, the second terminal 6200 includes an audio playback device 6201 that uses the SDL open-source player.

[0081] In this embodiment of the invention, the touch device 6102 of the first terminal 6100 acquires trajectory messages according to a preset sampling period, the audio acquisition device 6101 of the first terminal 6100 acquires audio signals according to a preset sampling period, the first terminal controller 6103 obtains trajectory data according to the trajectory messages, processes the audio signals to obtain audio data, and transmits the trajectory data and audio data synchronously to the second terminal controller 6203 through the first channel and the second channel, respectively.

[0082] In this embodiment of the invention, the second terminal controller 6203 obtains the trajectory message to be displayed by parsing the trajectory data, processes the audio data to obtain the audio signal to be played, the touch device 6102 of the second terminal 6200 displays the trajectory message to be displayed, and the audio playback device 6201 of the first terminal 6100 plays the audio signal to be played.

[0083] Figure 7 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 1 The execution entity in this embodiment can be Figure 6 The first terminal controller in the illustrated embodiment. (As shown) Figure 7 As shown, the method includes:

[0084] S701: Collects trajectory messages and audio signals according to a preset sampling period, obtains trajectory data based on the trajectory messages, and processes the audio signals to obtain audio data.

[0085] In this embodiment of the invention, an audio acquisition device is used to acquire audio signals from the teaching environment and resample them into a mono audio signal. Specifically, the audio acquisition device is a microphone. The audio acquisition device sends the acquired mono audio signal to a first terminal controller for storage, and the first terminal controller processes the acquired audio signal to obtain audio data.

[0086] In this embodiment of the invention, touch data from all touch points within a preset sampling period is collected and used as trajectory messages. In this embodiment, the touch device of the first terminal collects touch data generated by all touch operations on the touchscreen according to a preset sampling period. The touch device sends the collected numerical control data of all touch points to the first terminal controller. For example, the preset sampling period is 10 milliseconds. Specifically, the touch data includes at least one of the following: touch number, touch operation type code, touch point coordinate value, touch operation time corresponding to the touch point, and touch parameters; the touch operation type code is any one of the following: write operation type code, erase operation type code, undo operation type code, and back operation type code.

[0087] In this embodiment of the invention, after obtaining the trajectory message, the trajectory message is encoded according to a preset data frame format to obtain at least one trajectory frame, and all trajectory frames are processed according to the preset message format to obtain trajectory data. For example, the preset data frame format is at least one of a write frame format, an erase frame format, an undo frame format, and a rollback frame format. The touch operation type code for the write frame format is a write operation type code, the touch operation type code for the erase frame format is an erase operation type code, the touch operation type code for the undo frame format is an undo operation type code, and the touch operation type code for the rollback frame format is a rollback operation type code. For example, the preset message format includes the total number of messages and a first message body to an Nth message body, where each message body includes a message length and message data, and N is an integer greater than 0. Trajectory data is obtained by processing all trajectory frames according to the preset message format.

[0088] S702: The trajectory data and audio data are synchronously transmitted to the second terminal through the first and second channels respectively, so that the second terminal can parse the trajectory data to obtain the trajectory message to be displayed, process the audio data to obtain the audio signal to be played, and enable the second terminal to play the audio signal to be played and display the trajectory message to be displayed.

[0089] In this embodiment of the invention, after obtaining the trajectory data and audio data, the trajectory data and audio data are processed using Pulse Code Modulation (PCM) to obtain a first audio stream and a second audio stream. The first audio stream is transmitted to the second terminal via the first channel using a Real-Time Messaging Protocol (RTMP) format, and the second audio stream is transmitted to the second terminal via the second channel using the same RTM format, thus achieving synchronous transmission of the first and second audio streams. For example, the first and second channels can be the left and right channels in a stereo system, or any two different channels in a multi-channel system.

[0090] In the real-time example of this invention, the second terminal receives the first audio stream and the second audio stream transmitted by the first terminal through the first channel and the second channel, respectively. Specifically, the left channel is set to transmit the first audio stream and the right channel is set to transmit the second audio stream. By configuring the second terminal to parse the first audio stream to obtain the trajectory message to be displayed and to parse the second audio stream to obtain the audio signal to be played, configuring the audio playback device of the second terminal to play the audio signal to be played, and configuring the display device of the second terminal to display the trajectory message to be displayed, the synchronization of playing the audio information and displaying the trajectory message to be displayed is realized on the second terminal.

[0091] The live streaming method provided in this embodiment involves setting up a computer at the teaching end to collect trajectory messages and audio signals according to a preset sampling period. Trajectory data is obtained from the trajectory messages, and audio data is obtained by processing the audio signals. The trajectory data and audio data are then synchronously transmitted to the computer at the listening end via the first and second audio channels, respectively. The listening end computer parses the trajectory data to obtain the trajectory message to be displayed, processes the audio data to obtain the audio signal to be played, and synchronously plays the audio signal and displays the trajectory message to be displayed, thus improving the effectiveness of remote teaching.

[0092] Figure 8 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 2 .like Figure 8 As shown, in Figure 7 Based on the provided embodiments, this invention provides specific implementation steps for obtaining trajectory data according to trajectory messages, the steps of which include:

[0093] S801: Determine the preset data frame format that is consistent with the touch operation type code contained in the touch data of each touch point in the trajectory message as the target frame format corresponding to the touch point, wherein the target frame format is one of the writing frame format, erasing frame format, undo frame format and back frame format.

[0094] In this embodiment of the invention, a preset data frame format that is consistent with the touch operation type code contained in the touch data of each touch point is determined as the target frame format corresponding to the touch point. Figure 9 This is a schematic diagram of a preset data frame format provided in an embodiment of the present invention. Figure 9 As shown, the writing frame format provided in this embodiment of the invention includes a touch number, a writing operation type code, coordinate values, time, note thickness parameters, and note color parameters; the erasure frame format includes a touch number, an erasure operation type code, coordinate values, time, and eraser size parameters; the undo frame format includes a touch number, an undo operation type code, and time; and the rollback frame format includes a touch number, a rollback operation type code, and time. Specifically, the touch number is 2 bytes, the touch operation type code is 1 byte, the coordinate values ​​are 16 bytes, the time is 8 bytes, the note thickness parameter is 2 bytes, the note color parameter is 2 bytes, and the eraser size parameter is 16 bytes. For example, when the touch operation type code of the touch point touch data is an erase operation type code, the corresponding target data frame format is the erase frame format.

[0095] S802: Encode the touch data of each touch point according to the target data frame format corresponding to each touch point to obtain the trajectory frame corresponding to each touch point.

[0096] For example, the touch data of each touch point is encoded according to the target data frame format corresponding to each touch point to obtain the trajectory frame corresponding to each touch point. For example, when the touch operation type code of the touch point's touch data is an erase operation type code, the corresponding target data frame format is an erase frame format. The touch data of the touch point is then encoded according to the erase frame format corresponding to that touch point to obtain the trajectory frame corresponding to that touch point. The erase frame format includes the touch number, erase operation type code, coordinate values, time, and eraser size parameters. Therefore, the touch number, touch operation type code, touch point coordinate values, touch operation time corresponding to the touch point, and touch parameters contained in the touch point's touch data are encoded according to the erase frame format to obtain the trajectory frame corresponding to that touch point's data. Through the above processing, that is, encoding the touch data of each touch point according to the target data frame of each touch point to obtain the trajectory frame corresponding to each touch point, all trajectory frames corresponding to the touch data of all touch points are obtained.

[0097] S803: Determine the total number of messages based on the number of all trajectory frames, determine the message length of the message body corresponding to each trajectory frame based on the data size contained in each trajectory frame, determine the message data of the message body corresponding to each trajectory frame based on the data of each trajectory frame, determine the message body corresponding to each trajectory frame based on the message length and message data of the message body corresponding to each trajectory frame, sort all message bodies according to the numbers contained in all trajectory frames to obtain the first message body to the Nth message body, and determine the total number of messages and the first message body to the Nth message body as trajectory data according to the preset message format.

[0098] For example, the preset message format includes the total number of messages and message bodies numbered from the first to the Nth, where each message body includes a message length and message data. The total number of messages is determined based on the number of all trajectory frames. Then, the message length of the message body corresponding to each trajectory frame is determined based on the data size contained in each trajectory frame. The message data of the message body corresponding to each trajectory frame is determined based on the data of each trajectory frame. Then, the message body corresponding to each trajectory frame is determined based on the message length and message data of the message body corresponding to each trajectory frame. Finally, all message bodies are sorted according to the numbers contained in all trajectory frames to obtain message bodies numbered from the first to the Nth, and the total number of messages and message bodies numbered from the first to the Nth are determined as the trajectory data.

[0099] In this embodiment of the invention, based on the method for obtaining trajectory data from trajectory messages described in S801 to S803, after the second terminal receives the trajectory data, it parses the trajectory data to obtain trajectory messages to be displayed. Specifically, the trajectory messages to be displayed are parsed according to a preset message format to obtain at least one trajectory frame, and all trajectory frames are parsed according to a preset data frame format to obtain trajectory messages to be displayed. The trajectory messages to be displayed contain touch data of all touch points to be displayed, and the touch device of the second terminal displays the corresponding touch operation based on the touch data of all touch points to be displayed.

[0100] The live streaming method provided in this embodiment encodes the collected trajectory messages into trajectory frames according to a preset data frame format using a first terminal, processes all trajectory frames according to the preset message format to obtain trajectory data, and parses the trajectory data using a second terminal to obtain trajectory messages and display them, so that annotations can be displayed synchronously on the listening computer, thereby improving the effectiveness of remote teaching.

[0101] Figure 10 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 3 The execution entity in this embodiment can be Figure 6 The second terminal controller in the illustrated embodiment. (As shown) Figure 10 As shown, the method includes:

[0102] S1001: Receive trajectory data and audio data transmitted by the first terminal through the first channel and the second channel respectively, wherein the trajectory data is obtained by the first terminal based on the trajectory message, the audio data is obtained by the first terminal processing the audio signal, and the trajectory message and audio signal are collected by the first terminal according to a preset sampling period.

[0103] S1002: Parse the trajectory data to obtain the trajectory message to be displayed, and process the audio data to obtain the audio signal to be played.

[0104] S1003: Play the audio signal to be played and display the trajectory message to be displayed.

[0105] In this embodiment of the invention, the methods of S1001 to S1003 are similar to... Figure 7 The methods and effects achieved by S701 to S703 in the embodiments are the same, and will not be described again here.

[0106] Figure 11 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 4 In this embodiment of the invention, in Figure 10 Based on the provided embodiment, the second terminal also integrates a touchscreen and a microphone. After the second terminal plays the audio signal to be played and displays the trajectory message to be displayed, as... Figure 11 As shown, another live streaming method provided by the present invention includes:

[0107] S1101: Acquire touch trajectory according to preset time interval, display and store touch trajectory.

[0108] In this embodiment of the invention, after the touch device of the second terminal displays the trajectory message to be displayed and the audio playback device of the second terminal plays the audio signal to be played, the user on the listening terminal is also provided with the function of annotating and drawing on the touch screen of the second terminal, which facilitates the user to write and save the annotation content shared by the first terminal. For example, the touch screen of the second terminal obtains the touch trajectory corresponding to all touch operations of the user on the touch screen of the second terminal at preset time intervals, stores and displays the touch trajectory.

[0109] S1102: In response to the collected voice signal, the voice signal is sent to the second terminal, causing the second terminal to play the voice signal.

[0110] In another embodiment of the invention, the audio acquisition device of the second terminal can also acquire the voice feedback from the user at the listening end and send the acquired voice to the first terminal for playback, thereby realizing interaction between the user at the teaching end and the user at the listening end. Specifically, when the audio acquisition device of the second terminal acquires a voice signal, it sends the acquired voice signal to the second terminal for playback through the audio channel.

[0111] The live streaming method provided in this embodiment, by setting up a second terminal to display and store touch trajectories, and sending the collected audio signals to the second terminal for playback, not only facilitates users to write and save the annotations shared on the first terminal, but also enables interaction between users on the teaching end and users on the listening end, thereby improving the effectiveness of remote teaching.

[0112] Figure 12 Flowchart of the live streaming method provided in the embodiments of the present invention Figure 5 ,exist Figure 6 Based on the provided live streaming system structure, such as Figure 12 As shown, the method includes:

[0113] S1201: The touch device of the first terminal acquires trajectory messages according to a preset sampling period, and the audio acquisition device of the first terminal acquires audio signals according to a preset sampling period.

[0114] S1202: The first terminal controller obtains trajectory data based on the trajectory message, processes the audio signal to obtain audio data, and transmits the trajectory data and audio data synchronously to the second terminal through the first channel and the second channel, respectively.

[0115] S1203: The second terminal controller parses the trajectory data to obtain the trajectory message to be displayed, and processes the audio data to obtain the audio signal to be played.

[0116] S1204: The touch device of the second terminal displays the trajectory message to be displayed, and the audio playback device of the second terminal plays the audio signal to be played.

[0117] In this embodiment of the invention, the methods of S1201 to S1204 are similar to... Figure 7 The methods and effects achieved by S701 to S703 in the embodiments are the same, and will not be described again here.

[0118] Figure 13 A schematic diagram of the live streaming device provided in an embodiment of the present invention. Figure 1 .like Figure 13 As shown, the live streaming device 130 includes: a data acquisition module 1301 and a transmission module 1302.

[0119] The acquisition module 1301 is used to acquire trajectory messages and audio signals according to a preset sampling period, obtain trajectory data based on the trajectory messages, and process the audio signals to obtain audio data.

[0120] The transmission module 1302 is used to synchronously transmit the trajectory data and the audio data to the second terminal through the first channel and the second channel, respectively, so that the second terminal can parse the trajectory data to obtain the trajectory message to be displayed, process the audio data to obtain the audio signal to be played, and enable the second terminal to play the audio signal to be played and display the trajectory message to be displayed.

[0121] In one possible implementation, the acquisition module 1301 is specifically used to encode the trajectory message according to a preset data frame format to obtain at least one trajectory frame; and to process all trajectory frames according to the preset message format to obtain trajectory data.

[0122] In one possible implementation, the acquisition module 1301 is specifically used to acquire touch data of all touch points within a preset sampling period, and to use the touch data of all touch points as trajectory messages. The touch data of each touch point includes at least one of the following: touch number, touch operation type code, touch point coordinate value, touch operation time corresponding to the touch point, and touch parameters. The touch operation type code is any one of the following: write operation type code, erase operation type code, undo operation type code, and back operation type code.

[0123] In one possible implementation, the preset data frame format is at least one of a write frame format, an erase frame format, an undo frame format, and a rollback frame format. The touch operation type code of the write frame format is a write operation type code, the touch operation type code of the erase frame format is an erase operation type code, the touch operation type code of the undo frame format is an undo operation type code, and the touch operation type code of the rollback frame format is a rollback operation type code. Accordingly, the acquisition module 1301 is specifically used to determine the preset data frame format that is consistent with the touch operation type code contained in the touch data of each touch point in the trajectory message as the target frame format corresponding to the touch point, wherein the target frame format is one of a write frame format, an erase frame format, an undo frame format, and a rollback frame format; and to encode the touch data of the touch point according to the target data frame format corresponding to each touch point to obtain the trajectory frame corresponding to each touch point.

[0124] In one possible implementation, the preset message format includes a total number of messages and message bodies from the first to the Nth, wherein each message body includes a message length and message data, and N is an integer greater than 0. Accordingly, the acquisition module 1301 is specifically used to determine the total number of messages based on the number of all trajectory frames; determine the message length of the message body corresponding to each trajectory frame based on the data size contained in each trajectory frame, and determine the message data of the message body corresponding to each trajectory frame based on the data of each trajectory frame; determine the message body corresponding to each trajectory frame based on the message length and message data of the message body corresponding to each trajectory frame; sort all message bodies according to the numbers contained in all trajectory frames to obtain message bodies from the first to the Nth, and determine the total number of messages and message bodies from the first to the Nth as trajectory data according to the preset message format.

[0125] In one possible implementation, the transmission module 1302 is specifically used to process the trajectory data and the audio data using a pulse code modulation (PCM) encoding format to obtain a first audio stream and a second audio stream; to transmit the first audio stream to a second terminal using a real-time message transmission protocol (RTP) format via a first channel, and to transmit the second audio stream to the second terminal using a RTP format via a second channel, so that the second terminal can parse the first audio stream to obtain the trajectory message to be displayed, and parse the second audio stream to obtain the audio signal to be played.

[0126] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0127] Figure 14 A schematic diagram of the live streaming device provided in an embodiment of the present invention. Figure 2 .like Figure 14 As shown, the live streaming device 140 includes: a receiving module 1401, a parsing module 1402, and a display module 1403.

[0128] The receiving module 1401 is used to receive trajectory data and audio data transmitted by the first terminal through the first channel and the second channel respectively, wherein the trajectory data is obtained by the first terminal based on the trajectory message, the audio data is obtained by the first terminal processing the audio signal, and the trajectory message and audio signal are collected by the first terminal according to a preset sampling period.

[0129] The parsing module 1402 is used to parse the trajectory data to obtain the trajectory message to be displayed, and to process the audio data to obtain the audio signal to be played.

[0130] Display module 1403 is used to play the audio signal to be played and to display the trajectory message to be displayed.

[0131] In one possible implementation, the parsing module 1402 is specifically used to parse the trajectory data according to a preset message format to obtain at least one trajectory frame; and to parse all trajectory frames according to a preset data frame format to obtain a trajectory message to be displayed, wherein the trajectory message to be displayed contains touch data of all touch points to be displayed.

[0132] In one possible implementation, the live streaming device 140 further includes a sending module, specifically configured to acquire touch trajectories at preset time intervals, display and store the touch trajectories; and in response to the acquired voice signal, send the voice signal to the first terminal, causing the first terminal to play the voice signal.

[0133] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0135] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of embodiments suitable for specific application considerations.

Claims

1. A live streaming method, characterized in that, Applied to the first terminal, including: Trajectory messages and audio signals are collected according to a preset sampling period, and the trajectory messages are encoded according to a preset data frame format to obtain at least one trajectory frame; The total number of messages is determined based on the number of all trajectory frames; The message length of the message body corresponding to each trajectory frame is determined based on the data size contained in each trajectory frame, and the message data of the message body corresponding to each trajectory frame is determined based on the data of each trajectory frame. The message body corresponding to each trajectory frame is determined based on the message length and message data of the message body corresponding to each trajectory frame. All message bodies are sorted according to the numbers contained in all trajectory frames to obtain the first to Nth message bodies. The total number of messages and the first to Nth message bodies are determined as trajectory data according to a preset message format. The audio signal is then processed to obtain audio data. The step of collecting trajectory messages according to a preset sampling period includes: collecting touch data from all touch points within the preset sampling period, and using the touch data from all touch points as trajectory messages. The touch data from each touch point includes at least one of the following: touch number, touch operation type code, touch point coordinates, touch operation time corresponding to the touch point, and touch parameters. The touch operation type code is any one of the following: write operation type code, erase operation type code, undo operation type code, and back operation type code. The trajectory data and the audio data are processed using the Pulse Code Modulation (PCM) encoding format to obtain a first audio stream and a second audio stream. The first audio stream and the second audio stream are synchronously transmitted to the second terminal through the first channel and the second channel, respectively, so that the second terminal can parse the trajectory data to obtain the trajectory message to be displayed, process the audio data to obtain the audio signal to be played, and then play the audio signal to be played and display the trajectory message to be displayed.

2. The method according to claim 1, characterized in that, The preset data frame format is at least one of the following: writing frame format, erasing frame format, undo frame format, and rollback frame format. The touch operation type code of the writing frame format is a writing operation type code, the touch operation type code of the erasing frame format is an erasing operation type code, the touch operation type code of the undo frame format is an undo operation type code, and the touch operation type code of the rollback frame format is a rollback operation type code. Accordingly, encoding the trajectory message according to a preset data frame format to obtain at least one trajectory frame includes: The preset data frame format that is consistent with the touch operation type code contained in the touch data of each touch point in the trajectory message is determined as the target frame format corresponding to the touch point, wherein the target frame format is one of the writing frame format, erasing frame format, undo frame format and rollback frame format. The touch data of each touch point is encoded according to the target data frame format corresponding to each touch point to obtain the trajectory frame corresponding to each touch point.

3. The method according to claim 2, characterized in that, The writing frame format includes touch ID, writing operation type code, coordinate value, time, note thickness parameter, and note color parameter; The erase frame format includes touch number, erase operation type code, coordinate value, time, and erase size parameter; The undo frame format includes the touch number, undo operation type code, and time; The rollback frame format includes the touch number, rollback operation type code, and time.

4. The method according to any one of claims 1 to 3, characterized in that, The step of synchronously transmitting the first audio stream and the second audio stream to the second terminal via the first channel and the second channel, respectively, includes: The first audio stream is transmitted to the second terminal via the first channel using a real-time message transmission protocol format, and the second audio stream is transmitted to the second terminal via the second channel using the same real-time message transmission protocol format. This enables the second terminal to parse the first audio stream to obtain the trajectory message to be displayed, and to parse the second audio stream to obtain the audio signal to be played.

5. A live streaming method, characterized in that, Applied to the second terminal, including: The system receives a first audio stream and a second audio stream transmitted from a first terminal through a first channel and a second channel, respectively. The first and second audio streams are obtained by the first terminal processing trajectory data and audio data using Pulse Code Modulation (PCM) encoding. The trajectory data is obtained by the first terminal processing all trajectory frames according to a preset message format. The audio data is obtained by the first terminal processing audio signals. The trajectory messages and audio signals are collected by the first terminal according to a preset sampling period. The step of collecting trajectory messages according to the preset sampling period includes: collecting touch data from all touch points within the preset sampling period, and using the touch data from all touch points as trajectory messages. Each touch point's touch data includes at least one of the following: touch number, touch operation type code, touch point coordinates, touch operation time corresponding to the touch point, and touch parameters. The touch operation type code is any one of a write operation type code, an erase operation type code, an undo operation type code, and a rollback operation type code. The trajectory data is parsed to obtain the trajectory message to be displayed, and the audio data is processed to obtain the audio signal to be played. Play the audio signal to be played and display the trajectory message to be displayed; The preset message format includes a total number of messages and message bodies from the first to the Nth, wherein each message body includes a message length and message data, and N is an integer greater than 0. Accordingly, the first terminal processes all trajectory frames according to the preset message format to obtain trajectory data, including: determining the total number of messages based on the number of all trajectory frames; determining the message length of the message body corresponding to each trajectory frame based on the data size contained in each trajectory frame, and determining the message data of the message body corresponding to each trajectory frame based on the data of each trajectory frame; determining the message body corresponding to each trajectory frame based on the message length and message data of the message body corresponding to each trajectory frame; sorting all message bodies according to the numbers contained in all trajectory frames to obtain message bodies from the first to the Nth, and determining the total number of messages and message bodies from the first to the Nth according to the preset message format as trajectory data.

6. The method according to claim 5, characterized in that, The step of parsing the trajectory data to obtain the trajectory message to be displayed includes: The trajectory data is parsed according to a preset message format to obtain at least one trajectory frame; All trajectory frames are parsed according to a preset data frame format to obtain the trajectory message to be displayed, wherein the trajectory message to be displayed contains the touch data of all touch points to be displayed.

7. The method according to claim 5 or 6, characterized in that, After playing the audio signal to be played and displaying the trajectory message to be displayed, the method further includes: The touch trajectory is acquired at preset time intervals, and the touch trajectory is displayed and stored. In response to the collected voice signal, the voice signal is sent to the first terminal, causing the first terminal to play the voice signal.

8. A terminal device, characterized in that, The terminal device includes an audio acquisition device, a touch device, and a first terminal controller; The touch device is used to collect trajectory messages according to a preset sampling period; The audio acquisition device is used to acquire audio signals according to a preset sampling period; The first terminal controller is configured to execute the live streaming method as described in any one of claims 1 to 4.

9. A terminal device, characterized in that, The terminal device includes an audio playback device, a display device, and a second terminal controller; The second terminal controller is configured to execute the live streaming method as described in any one of claims 5 to 7; The audio playback device is used to play the audio signal to be played; The display device is used to display the trajectory message to be displayed.

10. A live streaming system, characterized in that, The live streaming system includes a first terminal and at least one second terminal. The first terminal includes an audio acquisition device, a touch control device, and a first terminal controller. The second terminal includes an audio playback device, a display device, and a second terminal controller. The touch device of the first terminal is configured as follows: The system collects touch data from all touch points within a preset sampling period and uses this data as a trajectory message. Each touch point's touch data includes at least one of the following: touch number, touch operation type code, touch point coordinates, touch operation time, and touch parameters. The touch operation type code can be any one of the following: write operation type code, erase operation type code, undo operation type code, and back operation type code. The audio acquisition device of the first terminal is configured as follows: Audio signals are acquired according to a preset sampling period; The first terminal controller is configured as follows: The trajectory message is encoded according to a preset data frame format to obtain at least one trajectory frame; The total number of messages is determined based on the number of all trajectory frames; the message length of the message body corresponding to each trajectory frame is determined based on the data size contained in each trajectory frame; and the message data of the message body corresponding to each trajectory frame is determined based on the data of each trajectory frame. The message body corresponding to each trajectory frame is determined based on the message length and message data of the message body corresponding to each trajectory frame. All message bodies are sorted according to the numbers contained in all trajectory frames to obtain the first message body to the Nth message body. The total number of messages and the first message body to the Nth message body are determined as trajectory data according to a preset message format. The audio signal is processed to obtain audio data. The trajectory data and the audio data are processed using the Pulse Code Modulation (PCM) encoding format to obtain a first audio stream and a second audio stream. The first audio stream and the second audio stream are synchronously transmitted to the second terminal through the first channel and the second channel, respectively. The second terminal controller, connected to the first terminal controller, is configured as follows: The system receives trajectory data and audio data transmitted from the first terminal through the first and second channels, respectively, parses the trajectory data to obtain the trajectory message to be displayed, and processes the audio data to obtain the audio signal to be played. The touch device of the second terminal is configured as follows: Display the trajectory message to be displayed; The audio playback device of the second terminal is configured as follows: Play the audio signal to be played.

Images