Online interaction method and system based on recorded content

Abstract

The embodiment of the invention provides an online interaction method and system based on recorded content, and the method can obtain an online interaction instruction in a process of playing recorded content data, and generates interaction feedback data and plays the interaction feedback data according to the online interaction instruction in response to the online interaction instruction. Wherein the interaction feedback data comprises video data and / or interaction data extracted from the recorded content data based on the online interaction instruction, and an interaction picture rendered in real time based on the online interaction instruction. According to the method, an online interaction process among multiple users can be simulated through a dynamically triggered interaction mechanism, and the sense of participation of the users is enhanced, so that the problem that real-time interaction cannot be realized when recorded and broadcast contents are played is solved. Moreover, through the generation management of the recorded content data, the interaction organization cost can be reduced, and the operation efficiency of the online interaction platform can be improved.

Description

Technical Field

[0001] This application relates to the field of online education technology, and particularly to an online interaction method and system based on recorded content. Background Art

[0002] Online interaction is a multi-user instant messaging method that can transmit instant messaging data through an interaction platform to achieve data communication among multiple users. The online interaction method can be applied to scenarios such as webcasting and online classes. Taking an online class as an example, the users participating in the online interaction can include teachers and students. The teacher user can send video images and course-related data to at least one student client through the teacher client of the online class platform, and the student user can receive the video images and respond to the data for the course-related data feedback through the student client of the online class platform, so as to achieve the purpose of online education.

[0003] Since the above-mentioned online class needs to realize online interaction in the form of live broadcast, and the live broadcast form requires the teacher user and the student user to participate in the interaction at the same time, the application scope of the online class is severely limited. Therefore, in order to increase the application scope, the online class can also be realized in the form of video recording. For an online class based on the video recording form, the teacher user can record the course video content in advance and upload the recorded content to the online class platform, and the student user can request the recorded content according to needs.

[0004] However, the recorded content of the online class can only be presented in the form of one-way playback, and the student user cannot conduct online interaction with respect to the recorded content, making the online class in the form of video recording lack real-time interactivity and immersion. Moreover, since different student users may request the recorded content at different times, it is easy to generate a sense of isolation during the playback of the recorded content, and it is difficult to effectively participate in communication activities, resulting in a relatively single learning experience and making it difficult to meet the user's needs for high participation and high immersion in online education. That is, the online class in the form of video recording has the problems of insufficient real-time interactivity, affecting the learning experience, and relatively high classroom organization costs. Summary of the Invention

[0005] In view of this, an embodiment of this application provides an online interaction method and system based on recorded content to solve the problem of inability to conduct real-time interaction during the playback of video recording content.

[0006] According to one aspect of this application, an online interaction method based on recorded content is provided. The method includes:

[0007] Playing recorded content data, where the recorded content data includes pre-recorded video data and interaction data; the video data includes a first video and a second video with the same total playback duration and playback progress;

[0008] Obtain an online interaction instruction, where the online interaction instruction is input by a client and / or generated by detecting the playback progress of the recorded content data;

[0009] In response to the online interaction instruction, generate interaction feedback data according to the online interaction instruction, where the interaction feedback data includes the video data and / or the interaction data extracted from the recorded content data based on the online interaction instruction, and an interaction screen rendered in real time based on the online interaction instruction;

[0010] Play the interaction feedback data.

[0011] In some embodiments, playing the recorded content data includes:

[0012] Receive a content playback instruction sent by the client;

[0013] In response to the content playback instruction, obtain the recorded content data;

[0014] Call a first player and a second player, where the first player and / or the second player is set with a time synchronization algorithm, and the time synchronization algorithm is used to send a synchronization instruction to the second player based on the playback time of the first player;

[0015] Play the first video through the first player and play the second video through the second player.

[0016] In some embodiments, playing the recorded content data includes:

[0017] Obtain the first playback time of the first video and the second playback time of the second video;

[0018] If the first playback time is not equal to the second playback time, set the playback speed of the second video according to the first playback time;

[0019] If the first playback time is equal to the second playback time, continuously monitor the first playback time and the second playback time.

[0020] In some embodiments, playing the recorded content data includes:

[0021] Obtain the playback screen of the recorded content data, where the playback screen includes a first video screen, a second video screen, and an interaction screen; the first video screen is generated by playing the first video; the second video screen is generated by playing the second video; the interaction screen is generated by displaying the interaction data;

[0022] The playing screen is displayed through an online interaction interface, which includes a core display area, a user display area, and an interaction display area; the core display area is used to display the first video screen; the user display area is used to display the second video screen; the interaction display area is used to display the interaction screen.

[0023] In some embodiments, the second video includes a pre-recorded user video and / or a real-time user video. Displaying the playing screen through the online interaction interface includes:

[0024] Obtain the number of the second videos and the number of display slots in the user display area;

[0025] If the number of the second videos is less than the number of display slots, calculate the number of supplementary positions, where the number of supplementary positions is equal to the difference between the number of display slots and the number of the second videos;

[0026] Load supplementary data according to the number of supplementary positions, where the supplementary data includes virtual user models; the virtual user models are user image models constructed based on the interaction data;

[0027] Display the playing screen corresponding to the supplementary data through the user display area.

[0028] In some embodiments, the interaction data includes script data. Obtaining an online interaction instruction includes:

[0029] Obtain the current playing progress of the first video;

[0030] Invoke a script engine, which is configured to parse and execute script data, where the script data includes interaction timestamps, interaction action types, and interaction target objects;

[0031] Use the script engine to read the script data at preset time intervals;

[0032] If the current playing progress hits the interaction timestamp in the script data, generate the online interaction instruction.

[0033] In some embodiments, generating interaction feedback data according to the online interaction instruction includes:

[0034] Real-time obtain the interaction text information input by the client and the current playing progress of the first video;

[0035] Convert the interaction text information into a preset data format to generate a first-format text;

[0036] Extract associated text information from the interactive data based on the current playback progress and the first-format text, where the associated text information includes historical interactive text recorded during the playback of the recorded content data;

[0037] Convert the associated text information into a preset data format to generate a second-format text;

[0038] Generate the interactive feedback data based on the first-format text and the second-format text.

[0039] In some embodiments, generating interactive feedback data according to the online interaction instruction includes:

[0040] Parse the instruction content information from the online interaction instruction;

[0041] Perform data assembly according to the instruction content information to generate interactive instruction data, where the interactive instruction data includes at least one of interaction type, interaction time, and interaction rule;

[0042] Push the interactive instruction data to the core display area of the online interaction interface to display interactive controls in the core display area;

[0043] Receive the interactive operation data input by the client based on the interactive controls, and extract the interactive feedback data from the recorded content data according to the interactive operation data.

[0044] In some embodiments, extracting the interactive feedback data from the recorded content data according to the interactive operation data includes:

[0045] Obtain a preset judgment rule;

[0046] Perform a judgment process on the interactive operation data according to the preset judgment rule to generate a judgment result;

[0047] Render an interactive screen in real time according to the judgment result, and generate the interactive feedback data according to the interactive screen;

[0048] Send the interactive feedback data to the client.

[0049] According to another aspect of the present application, there is provided an online interaction system based on recorded content. The system includes a server and at least one client. The client establishes a communication connection relationship with the server. The server includes:

[0050] A first playback module for playing recorded content data, where the recorded content data includes pre-recorded video data and interactive data; the video data includes a first video and a second video with the same total playback duration and playback progress;

[0051] An instruction acquisition module, configured to acquire an online interaction instruction, where the online interaction instruction is input by a client and / or generated by detecting the playback progress of the recorded content data;

[0052] An interaction feedback module, configured to generate interaction feedback data in response to the online interaction instruction, where the interaction feedback data includes the video data and / or the interaction data extracted from the recorded content data based on the online interaction instruction, and an interaction screen rendered in real time based on the online interaction instruction;

[0053] A second playback module, configured to play the interaction feedback data.

[0054] According to another aspect of the present application, there is provided a computer device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the program, the above online interaction method based on recorded content is implemented.

[0055] According to still another aspect of the present application, there is provided a storage medium, on which a computer program is stored. When the program is executed by a processor, the above online interaction method based on recorded content is implemented.

[0056] By means of the above technical solution, an embodiment of the present application provides an online interaction method and system based on recorded content. The method can acquire an online interaction instruction during the playback of recorded content data, and in response to the online interaction instruction, generate interaction feedback data and play the interaction feedback data according to the online interaction instruction. Among them, the interaction feedback data includes video data and / or interaction data extracted from the recorded content data based on the online interaction instruction, and an interaction screen rendered in real time based on the online interaction instruction. The method can simulate the online interaction process among multiple users through a dynamically triggered interaction mechanism, enhance the user's sense of participation, and solve the problem of inability to interact in real time when playing recorded content. Moreover, through the generation management of the recorded content data, the interaction organization cost can be reduced, and the operation efficiency of the online interaction platform can be improved.

[0057] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the present application more obvious and understandable, the following specific embodiments of the present application are specifically given. Description of the Drawings

[0058] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0059] Figure 1 Schematic diagram of device connection for the online classroom system provided by the embodiment of the present application;

[0060] Figure 2 Schematic diagram of data transfer process for the online classroom system provided by the embodiment of the present application;

[0061] Figure 3 Schematic diagram of the process of the online interaction method based on the recorded content provided by the embodiment of the present application;

[0062] Figure 4 Schematic diagram of the data process for playing the recorded content provided by the embodiment of the present application;

[0063] Figure 5 Schematic diagram of the time synchronization process provided by the embodiment of the present application;

[0064] Figure 6 Schematic diagram of the online interaction interface provided by the embodiment of the present application;

[0065] Figure 7 Schematic diagram of the process of triggering interaction instruction data provided by the embodiment of the present application;

[0066] Figure 8 Schematic diagram of the data flow relationship provided by the embodiment of the present application;

[0067] Figure 9 Schematic diagram of the process of generating interaction feedback data provided by the embodiment of the present application;

[0068] Figure 10 Schematic diagram of the structure of the online interaction system based on the recorded content provided by the embodiment of the present application. Detailed implementation manners

[0069] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0070] In the embodiment of the present application, the online interaction is a multi-user instant messaging method. The online interaction can transmit instant messaging data through an interaction platform to realize data communication between multiple users. Among them, the instant messaging data can include video data for transmitting image screens, communication data for transmitting communication content, instruction data for transmitting control instructions, etc.

[0071] The online interaction method can be applied to scenarios such as webcasting and online classes. And in order to implement the online interaction function, an online interaction platform can be constructed based on specific application scenarios. In some embodiments, the online interaction platform may include a server and an online interaction application running on the server. The server can establish communication connections with multiple clients, receive interaction data sent by the multiple clients, and perform operations such as processing, storing, and forwarding the interaction data to provide the function of online interaction between the multiple clients.

[0072] Taking an online class as an example, the users participating in the online interaction may include teachers and students. Teachers and student users can use the client for user interaction. For the convenience of description, the client used by the teacher user is called the teacher client or the host client, and the client used by the student user is called the student client or the participating client. As Figure 1 shown, both the teacher client and the student client establish communication connections with the server (or the service end) to achieve data transfer in the online class.

[0073] During the data transfer process, the online interaction platform needs to collect data, that is, as Figure 2 shown, on the host client, video and audio data are collected through devices such as cameras and microphones. The collected data needs to be processed and encoded for data transmission. For example, the collected audio and video data needs to be processed and encoded. Data processing includes video (image) correction, adding watermarks, etc. to the video data. Data encoding is to compress the audio and video data into a format suitable for network transmission. For example, H.264 encoding is used for video and AAC encoding is used for audio. The encoded data is encapsulated into a specific streaming media format, such as FLV (Flash Video), TS (Transport Stream), etc.

[0074] The encoded data is sent to the streaming media server through a push stream protocol such as the Real Time Messaging Protocol (RTMP). In order to send the encoded data to the server, the push stream process may include steps such as establishing a Transmission Control Protocol (TCP) connection, RTMP handshake, negotiating the version number and timestamp, etc. And after the handshake is successful, the encoded data is encapsulated into RTMP message blocks for sending.

[0075] After receiving the push stream data, the streaming media server will perform data processing and distribution on the server side. For example, it will convert the bitstream of the RTMP protocol into a transmission protocol such as HTTP Live Streaming (HLS) to adapt to different clients. The server will also distribute the data to the edge nodes associated with the clients through the Content Delivery Network (CDN). When a user requests data, the CDN node checks the local cache. If the data exists in the local cache, it will send the data to the client. If there is no cached data in the local cache, it will request the data from the source station.

[0076] After the data processing and distribution on the server side, the student client can obtain data through pull stream protocols such as HLS and HTTP-FLV. That is, the student client can request data from the server, and the server will push the data to the client. After receiving the data, the client decodes and plays the data through a decoder.

[0077] Since the online live classroom also involves the transmission of interactive data, such as text chat, bullet screens, etc. Therefore, in order to achieve the transmission of interactive data, the client and the server can also maintain a persistent connection method such as full-duplex communication. For example, the interactive data can be transmitted in real time through the Websocket protocol. Among them, Websocket can maintain a persistent connection to ensure the real-time nature and low latency of the data. For this purpose, the client can initiate a WebSocket connection handshake request through an HTTP request. The request contains an Upgrade header, indicating the desire to upgrade the connection to the WebSocket protocol. After the server agrees, it returns a status code 101, indicating that the connection has been successfully upgraded. After the handshake is completed, the client and the server can send text or binary data frames through the WebSocket connection at any time.

[0078] In order to optimize the data transmission process, the live classroom can also adopt a variety of technologies for data optimization and latency control. For example, since the longer the Group of Pictures (GOP) length, the better the picture quality, but the higher the latency, an appropriate GOP length can be set to balance the picture quality and latency. In addition, through load balancing technology, user requests can be evenly distributed to multiple servers, improving the stability and response speed of the system.

[0079] It can be seen that through the data transmission method shown in the above embodiments, during the online classroom process, the teacher user can send video images and course-related data to at least one student client through the teacher client of the online classroom platform, and the student user can receive the video images and respond to the response data for the course-related data through the student client of the online classroom platform, so that the data of the live classroom can be efficiently and stably transmitted from the host end to the student end, realizing high-quality online teaching.

[0080] Since the above online classroom needs to achieve online interaction in the form of live broadcast, and the live broadcast form requires the teacher user and the student user to participate in the interaction at the same time, the application scope of the online classroom is severely limited. For example, when the teacher user initiates a live classroom invitation and sets an online live course for subject A from 13:00 on February 1st to 14:00 on February 1st, both the teacher user and the student user need to log in to their accounts through the client during this time period and enter the online live classroom room to participate in the live course.

[0081] Therefore, in order to increase the application scope, the online classroom can also be realized in the form of recorded broadcast. Based on the online classroom in the form of recorded broadcast, the teacher user can record the course video content in advance and upload the recorded content to the online classroom platform, and the student user can play back the recorded content as needed. For example, after logging in to the student user account, the student client can select any recorded broadcast course video in the course list to play and watch the pre-recorded course video content.

[0082] However, the recorded content of the online classroom can only be presented in a one-way playback form, and the student user cannot conduct online interaction with the recorded content, making the online classroom in the form of recorded broadcast lack real-time interactivity and immersion. Moreover, since different student users may play back the recorded content at different times, it is easy to generate a sense of isolation during the playback of the recorded content, making it difficult to effectively participate in communication activities, resulting in a relatively single learning experience and difficult to meet the user's needs for high participation and high immersion in online education. That is, the online classroom in the form of recorded broadcast has the problems of insufficient real-time interactivity, affecting the learning experience, and relatively high classroom organization costs.

[0083] In order to solve the problem of not being able to interact in real time during the playback of recorded content, some embodiments of this application provide an online interactive method based on recorded content, which can be applied to a server, as well as electronic devices that can serve as a server and have data processing capabilities, such as computers, mobile terminals, industrial hosts, etc. In some embodiments of this application, an electronic device or an online interactive platform is used as the execution subject of the online interactive method based on recorded content. Obviously, the electronic device or online interactive platform can refer to a server, or one or more combinations of the above-mentioned computers, mobile terminals, and industrial hosts, and other execution subjects associated with the technical personnel in this field based on the embodiments of this application also belong to the protection scope of this application.

[0084] like Figure 3 As shown, the online interaction methods based on recorded content include:

[0085] S101, playing recorded content data.

[0086] In the embodiment of the present application, online interaction is performed based on the playback process of recorded content. Therefore, when performing online interaction, the recorded content data can be played first. The recorded content data includes pre-recorded video data and interaction data. Video data refers to video content data transmitted between multiple clients during online interaction. Interaction data refers to non-video content data transmitted between multiple clients during online interaction, such as text, special effect controls, control instructions, etc.

[0087] In some embodiments, the video data includes a first video and a second video with the same total playback time and playback progress. The first video is the main video content recorded in advance, and the second video is the video content recorded in advance or in real time. Taking the live classroom as an example, the first video is a video obtained by recording the courseware playback process of the online course; the second video is a video obtained by recording the users participating in the online course. Since the users participating in the online classroom can include a teacher user and multiple student users, the second video can also include a user video of the teacher user and user videos of multiple student users.

[0088] In some embodiments, the online interaction platform can perform content recording to generate recorded content data. That is, the client running the online interaction platform application can respond to the content recording instruction and invoke the content recording process (or application). Then, the content recording process captures the image data on the screen. For example, the currently displayed content can be captured frame by frame through the system interface built into the operating system and passed to the content recording process. At the same time, the audio recording process is invoked to record the audio played by the electronic device and the audio received through the microphone separately or simultaneously according to the user settings. Then, the recorded content is converted into a storable file, that is, the captured images and audio are compressed and encoded. For example, for video data, encoding formats such as H.264 and H.265 can be used for compression. For audio data, encoding formats such as AAC can be used for compression.

[0089] While recording audio-visual content, the electronic device can also record the interaction data during the recording process, that is, record interaction information such as communication text, operation special effects, control instructions, etc., and record the generation time points corresponding to these interaction information to generate interaction data. For example, for a live classroom application, the teacher user can conduct live teaching according to the live teaching process and start the recording and broadcasting function to record the live teaching content. After starting the recording and broadcasting function, the recording and broadcasting process can record the courseware playback video to generate the first video. It can also record video content such as the teacher user video and student user video collected by the client camera during the teaching process to generate the second video. At the same time, during the live teaching process, there will be classroom interactions between the teacher user and the student user. For example, when the live broadcast reaches time T1, the teacher user initiates a question-and-answer operation Q1, then the content recording process can record the initiation time T1 of the question-and-answer operation Q1 to generate interaction data with the interaction type of the question-and-answer operation. It can be seen that the interaction data can record all interaction operations during the recording process, including but not limited to questions and answers, speeches, raising hands, rewards, homework, chats, etc.

[0090] It should be noted that the above video data has a broad meaning, not only referring to specific video files or video stream data, but also referring to other image change processes based on time as the playback reference. For example, the first video can be obtained through video recording or by generating an image and playback time association data pair by recording multiple slides and the playback time of each slide. When playing the first video, the corresponding slides can be presented one by one according to the playback time, thereby reducing the data volume of the first video. Similarly, the second video can be obtained through video recording or generated by artificial intelligence technologies such as digital humans.

[0091] Such as Figure 4As shown, in some embodiments, in order to play recorded content data, when the electronic device executes playing the recorded content data, it may receive a content play instruction sent by the client, and in response to the content play instruction, obtain the recorded content data; then call a first player and a second player, and play the first video through the first player and play the second video through the second player.

[0092] Among them, the first player is used to play the first video, that is, through video playback operations such as decoding, a playback picture is generated based on the first video. Similarly, the second player is used to play the second video. Obviously, the playback picture may include the first video picture and the second video picture. For the sake of convenience of description, the playback picture corresponding to the first video is the first video picture, and the playback picture corresponding to the second video is called the second video picture, that is, the first video picture is generated by playing the first video; the second video picture is generated by playing the second video.

[0093] In order to ensure the synchronization of the video data playback process, the first player and / or the second player is set with a time synchronization algorithm, and the time synchronization algorithm is used to send a synchronization instruction to the second player based on the playback time of the first player.

[0094] For example, when playing recorded content data, the live classroom application can start the time synchronization algorithm, which sends a synchronization instruction to the playback processes of the student user video and the teacher user video based on the playback time of the courseware area video. The live classroom application can obtain and process the duration information of the student user video and the teacher user video, compare and match it with the duration of the courseware video to ensure that the total playback duration and the current playback progress of the three are consistent, and achieve the synchronous playback of video data.

[0095] And through the time synchronization algorithm, ensure the synchronous playback of the courseware video, the teacher user video and the student user video. That is, if the video playback time is set as t, then the playback times of all video streams need to satisfy:

[0096] t teacher =t student =t;

[0097] Among them, t is the video playback time, t teacher is the duration of the teacher user video, and t student is the duration of the student user video.

[0098] During the entire playback process of the recorded content data, the live classroom application can also continuously monitor the playback time and finely adjust the playback times of each video stream based on the time synchronization algorithm to maintain the synchronous state. That is, as Figure 5As shown, in some embodiments, when the electronic device plays the recorded content data, it may obtain the first playback time of the first video and the second playback time of the second video, and compare the obtained first playback time and second playback time. If the first playback time is equal to the second playback time, continuously monitor the first playback time and the second playback time.

[0099] If the first playback time is not equal to the second playback time, set the playback speed of the second video according to the first playback time. For example, when the playback time of the courseware video is obtained as t p , and the playback time of the user video is t u , and when the playback time t p of the courseware video is ahead of the playback time t u of the user video, the playback speed multiple of the user video can be set to be greater than 1, such as 1.2, etc., to play the user video quickly until the playback time t p of the courseware video is synchronized with the playback time t u of the user video. Similarly, when the playback time t p of the courseware video lags behind the playback time t u of the user video, the playback speed multiple of the user video can be set to be less than 1, such as 0.8, etc., to play the user video slowly until the playback time t p of the courseware video is synchronized with the playback time t u of the user video.

[0100] The playback screen of the recorded content data can be displayed through a specific user interface. In some embodiments, the user interface for displaying the playback screen corresponding to the recorded content data is called an online interaction interface. Then, when playing the recorded content data, the electronic device can obtain the playback screen of the recorded content data and display the playback screen through the online interaction interface. Among them, in addition to the above-mentioned first video screen and second video screen, the playback screen may further include an interaction screen, and the interaction screen is generated by displaying the interaction data.

[0101] The online interaction interface includes a core display area, a user display area, and an interaction display area; the core display area is used to display the first video screen; the user display area is used to display the second video screen; the interaction display area is used to display the interaction screen. For example, Figure 6As shown, to more accurately understand the classroom structure and effects, the data stream of the online classroom can include the video stream of student users, and the video stream of student users can be displayed in the user display area corresponding to No. 1. The user display area can also include other types of user video display areas, such as the user display area for the teacher user video with the number 2, and the user video picture of the current user's camera in real time with the number 3. The interactive display area is used to display communication information, operation special effects, etc. during the content playback process. For example, the number 4 is the chat area, which can display the real-time chat content of real student users and the current user. The chat content can also include the chat content of actors and teacher users at different video recording time points during recording. The number 5 is the core display area, which is used to display the courseware content and can also be called the courseware area.

[0102] Since the video stream of student users can be generated by real-time recording of the cameras of real student users, or by playing back the pre-recorded video of actor users in real time, and can also be an AI-generated character model, the AI-generated character model can be used to alleviate the problem of reduced classroom immersion caused by insufficient number of participating users. Therefore, in some embodiments, the second video includes a pre-recorded user video and / or a real-time user video. When displaying the playback screen through the online interaction interface, the number of the second videos and the number of display columns in the user display area can be obtained first, and the number of the second videos and the number of display columns can be compared. If the number of the second videos is less than the number of display columns, the filling number is calculated, that is, the difference between the number of display columns and the number of the second videos is calculated to obtain the filling number. Then, the filling data is loaded according to the filling number, and the playback screen corresponding to the filling data is displayed through the user display area.

[0103] Among them, the filling data includes a virtual user model, and the virtual user model is a model constructed based on 3D digital human generation technology. For live classroom applications, AI character models for different age groups such as children, teenagers, young people, adults, and middle-aged people can be pre-constructed according to the age characteristics of the corresponding objects of the recorded courses. The AI character model can generate various character actions during the playback process and generate specific character actions at specific time points, that is, when a hand-raising interaction is required, a hand-raising action is generated, etc.

[0104] The virtual user model can also be a user image model constructed based on the interaction data. For example, text description information can be generated based on the interaction data, and then the text description information is input into digital human generation applications such as Make-A-Character to perform 3D modeling and rendering through the digital human generation application, and a complete 3D character model including details such as facial features, hairstyles, and clothing is generated according to the text description. These models support animations and advanced skeletal rigid bodies to complete specific actions.

[0105] S102. Obtain an online interaction instruction.

[0106] After the electronic device plays the recorded content data, it can also obtain an online interaction instruction during the process of playing the recorded content data. The online interaction instruction is a control instruction used to trigger an online interaction action. Adapted to the online interaction process described in the above embodiments, according to different specific online interaction methods, the online interaction instruction may include different instruction contents. That is, the online interaction instruction can specify the time, type, target, etc. of the interaction action.

[0107] In some embodiments, the online interaction instruction is input by the client. That is, the current user who requests to play the recorded content can input an online interaction instruction through the user client. For example, when the video of the live classroom starts playing, the data interaction process in the chat area starts to run. The data interaction process can receive the chat content "AAA" input by the user in the chat area at time t c and encapsulate the chat content into the online interaction instruction and send it to the server. The server can then obtain the online interaction instruction in response to the user's input action based on the chat area. Since the online interaction instruction is triggered by the chat function, the online interaction instruction can include the interaction time "t c ", the interaction type "chat", and the interaction target "chat area".

[0108] In some embodiments, the online interaction instruction can be generated by detecting the playback progress of the recorded content data. That is, when obtaining the online interaction instruction, the current playback progress of the first video can be obtained first, and the script engine is called. Then, using the script engine, the script data is read at a preset time interval. If the current playback progress hits the interaction timestamp in the script data, the online interaction instruction is generated.

[0109] Among them, the script engine is configured to parse and execute the script data. The script data is a part of the interaction data and is data obtained by recording and converting the data format according to the interaction actions during the recording process. The script data can include the interaction timestamp, the interaction action type, and the interaction target object. Then, during the period when the electronic device plays the recorded content data, the script engine can be started, and the script data can be parsed by the script engine to trigger an interaction event at a specified time according to the result parsed by the script engine. For example, the interaction event can include answering questions, speaking, etc. Assume that the script event trigger time is ti, then at time ti, the corresponding operation is performed:

[0110] if t = t i , then execute action (t i );

[0111] Therefore, during the playback of the recorded live class video, the script engine can read the script data from the database at a set time interval (e.g., every 1 second) and perform analysis and processing. When it is detected that the current playback progress reaches the trigger time t of the script event i an online interaction instruction can be generated, that is, triggering the electronic device to execute online interaction action control.

[0112] S103. In response to the online interaction instruction, generate interaction feedback data according to the online interaction instruction.

[0113] After obtaining the online interaction instruction, the electronic device can perform online interaction control in response to the online interaction instruction, that is, generate interaction feedback data according to the online interaction instruction. Among them, the interaction feedback data is used to form an interaction feedback in the interface for playing the recorded content data, such as displaying chat data, presenting interaction special effects, playing different audio and videos according to the interaction process, etc. Therefore, the interaction feedback data may include the video data and / or the interaction data extracted from the recorded content data based on the online interaction instruction, and the interaction screen rendered in real time based on the online interaction instruction.

[0114] For the recorded content, the electronic device can extract the video data and / or interaction data from the recorded content data according to the online interaction instruction. For the unrecorded content or the interaction content built in the current online interaction application, an interaction screen rendered in real time based on the online interaction instruction can be used.

[0115] In some embodiments, in order to generate interaction feedback data, when generating the interaction feedback data according to the online interaction instruction, the server can first parse the instruction content information from the online interaction instruction, and then perform data assembly according to the instruction content information to generate interaction instruction data. Among them, the interaction instruction data includes at least one of an interaction type, an interaction time, and an interaction rule. Then, the interaction instruction data is pushed to the core display area of the online interaction interface to display interaction controls in the core display area. During the display of the interaction controls, the server can also receive the interaction operation data input by the client based on the interaction controls, and extract the interaction feedback data from the recorded content data according to the interaction operation data.

[0116] For example, as Figure 7 shown, during the process of recording the live class video content, the teacher user can initiate a question-and-answer interaction with the student user at time ta. Then, during the playback of the recorded live class video, when the playback progress reaches time ta, the question-and-answer interaction will also be triggered. That is, at t = t iWhen this happens, the server can parse the instruction content information from the online interaction instruction, read the script data through the script engine, and obtain that the interaction content to be triggered at this time is a quiz interaction. Then, according to the instruction content information, data assembly is performed to generate interaction instruction data including information such as interaction type (quiz), interaction time (ta), and interaction rules (need to click on options to give feedback). Then, the interaction instruction data is pushed to the courseware area of the live classroom application interface, and the corresponding quiz interaction interface pops up in real time on the upper layer of the courseware area. During the period when the quiz interaction interface pops up, the user client can also be used to detect the interaction actions input by the user based on the quiz arc interface, that is, to detect the specific interaction content of the user's participation in the interaction.

[0117] In some embodiments, the online interaction platform can also perform real-time feedback on the interaction content input by the user. That is, when extracting the interaction feedback data from the recorded content data according to the interaction operation data, a preset evaluation rule can be obtained, and the interaction operation data can be evaluated according to the preset evaluation rule to generate an evaluation result. Then, an interaction screen is rendered in real time according to the evaluation result, and the interaction feedback data is generated according to the interaction screen, so as to send the interaction feedback data to the client.

[0118] For example, as Figure 8 shown, after the user participates in the interaction, the live classroom application can collect the user's answer data, process and analyze the answer data according to the preset evaluation rules, obtain the answer result, and generate corresponding feedback data according to the result, such as reward information or evaluation content, and send it to the user client.

[0119] For different online interaction instructions and different interaction operation data, the online interaction platform can generate interaction feedback data for representing different contents. For some interaction feedback data, different content continuation points can be associated. When generating the interaction feedback data, the player can be controlled to play the recorded content data according to different content continuation points according to the different interaction contents corresponding to the online interaction instructions and the interaction operation data.

[0120] For example, when an online live classroom application triggers a question-and-answer interaction based on the current playback progress, a question-and-answer window can pop up in the courseware area. In the question-and-answer window, there can be two operation controls, option A and option B, corresponding to two different question-and-answer results respectively. For example, option A is the correct answer and option B is the wrong answer. When recording the live classroom video, the explanation for the correct answer can start at time t1, and the explanation for the wrong answer can start at time t2. Then, after the user feeds back the interaction operation data by selecting option A, the online live classroom application can use time t1 as the content resume point and continue to play the first video and the second video in the courseware area and the user area. That is, the video data after time t1 is extracted from the video data as the interaction feedback data. Similarly, when the user feeds back the interaction operation data by selecting option B, the online live classroom application can use time t2 as the content resume point, so as to extract the video data after time t2 from the video data as the interaction feedback data.

[0121] In some embodiments, after obtaining the online interaction instruction, the server electronic device can continue to monitor the script data. When detecting the end of the interaction point, it generates a control instruction for ending the interaction operation and closes the interaction interface according to the control instruction, completing the full process processing of the interaction instruction data.

[0122] S104. Play the interaction feedback data.

[0123] After generating the interaction feedback data according to the online interaction instruction, the electronic device can play the interaction feedback data. Since the interaction feedback data can include one or more combinations of video data, interaction data, and interactively rendered pictures. Therefore, when playing the interaction feedback data, the electronic device can present the video data, interaction data, and interactive pictures respectively.

[0124] For example, during the recorded broadcast process, after triggering real-time interactions such as adding questions and answers, students' microphones being turned on, chat messages, and teachers' microphones being turned on based on the script data, the interaction feedback data can be generated according to the above embodiments. That is, for the added question-and-answer interaction, the interaction feedback data is used to control the online live classroom application to pop up a question-and-answer window above the courseware area; for the students' microphone-on interaction, the interaction feedback data is used to control the online live classroom application to play the audio of the student user whose microphone is turned on; for the chat message interaction, the interaction feedback data is used to control the online live classroom application to display the real-time input chat information, the chat information recorded during recording, and the chat information at the current playback progress during previous playbacks of the live course in the chat area, etc. For the teachers' microphone-on interaction, the interaction feedback data is used to control the online live classroom application to play the audio of the teacher user, etc., so as to achieve the real-time interaction effect.

[0125] By applying the technical solutions of the above embodiments, the online interaction method based on recorded content provided in the above embodiments can overcome the problems of high resource consumption, insufficient interactivity, and poor immersion through innovative interactive platform design and technical means, and can provide an efficient, highly interactive, and immersive online interaction experience. Moreover, during the playback of the recorded content, various forms of interaction operations can be triggered based on the script data, and different interaction feedbacks can be generated for different interaction operations to achieve the effect of real-time interaction.

[0126] In some embodiments, as a refinement and extension of the specific implementation manner of the above embodiments, in order to fully illustrate the specific implementation process of this embodiment, some embodiments of the present application further provide an online interaction method based on recorded content. As Figure 9 shown, based on the above embodiments, the method can also synchronously record interaction operations and generate script data during the process of generating interaction feedback data according to the online interaction instruction. That is, the method includes:

[0127] S201. Real-time obtain the interaction text information input by the client and the current playback progress of the first video;

[0128] S202. Convert the interaction text information into a preset data format to generate a first-format text;

[0129] S203. Based on the current playback progress and the first-format text, extract associated text information from the interaction data, where the associated text information includes historical interaction texts recorded during the playback of the recorded content data;

[0130] S204. Convert the associated text information into a preset data format to generate a second-format text;

[0131] S205. Generate the interaction feedback data according to the first-format text and the second-format text.

[0132] During the playback of the recorded content data, the current user can input interaction text information through the client. Then, the online interaction platform can collect the interaction text information in real time and convert the interaction text information into a format to generate a first-format text. For example, when the video starts playing, the data interaction process in the chat area starts. For real student users, the data interaction process can collect the text information input by real student users in the chat area in real time and push the text information to the chat area for display. While displaying the input text information, the data interaction process can process the collected text information according to a predetermined format to generate text information that conforms to the script data specification.

[0133] Among them, the preset data format is a data format predefined to adapt to the reading and processing specifications of the script engine. The preset data format can be a lightweight data exchange format such as JSON (JavaScript Object Notation), which is convenient for machine parsing and generation. For example, JSON data consists of key-value pairs, and the overall structure is an object or an array. Among them, an object consists of a series of key-value pairs. The key is of string type, and the value (Value) can be of various types, such as string, number, boolean, array, object, etc. An array consists of a series of values, and the values can be of any type (including objects).

[0134] Based on the preset data format, various interaction actions and interaction information input by the current user can be converted into a unified data format. For example, to convert the interaction text information into the preset data format to generate the first-format text, when the current user inputs the interaction text information "uuuu" in the chat area, the data interaction process can, after collecting the interaction text information, convert the input text information into the first-format text. In the first-format text, the input role of the text information can be represented by "fromRole":"1"; the specific content of the text information can be represented by "mes":\\u0075\\u0075\\u0075\\u0075; the data range can be represented by "isOwn":1; the data type can be represented by "type":"words"; the input (send) time of the text information can be represented by "sendTime":××42892612, etc.

[0135] In the process of generating the first-format text, the data content can also be expressed in multiple dimensions according to the parameter items of the script data. For example, the first-format text can also include "fromUserInfo" (source user information), "teamFlag" (team flag), "groupId" (team ID), "groupName" (team name), "headIcon":https: / / ××.jpg (profile picture), "headFrame" (profile frame), "nickName":"18××" (nickname), "userId":"9667579" (user ID), "winNum":"0" (number of wins), "tutorId":"PV000205" (tutor ID), "area":"02501" (region), "totalWinPinYin":"nengshou" (full name in pinyin), "msgState":3 (message state), "msgId":"×613536" (message ID) and other parameter items to respectively represent different data-related information.

[0136] After generating the first - format text, it is also possible to extract associated text information based on the playback progress of the current recorded content data and the first - format text in the interaction data. Among them, the associated text information includes historical interaction texts recorded during the playback of the recorded content data. For example, when the live - class video is played to time t3, if the current user's input content is the chat data "uuuu", the live - class application can extract alternative chat data input or displayed at time t3 (or around time t3) from the database. When there is a large amount of alternative chat data, it is also possible to extract chat data related to the current input content "uuuu" from the alternative chat data as associated text information based on the first - format text corresponding to "uuuu".

[0137] For the extracted associated text information, the online interaction platform can also perform format conversion according to the data format specified in the script data to convert the associated text information into a preset data format, generate the second - format text, and then generate the interaction feedback data based on the first - format text and the second - format text.

[0138] For example, for actor - student users and virtual - teacher users, the online - class application can retrieve the chat content corresponding to the time point of video playback from the pre - stored chat data, and after the same formatting process, push it to the chat area for display. Through this way of displaying interaction information, it is possible to achieve text - chat data interaction among multiple users in the chat area, ensuring the accuracy and real - time nature of chat information display.

[0139] In some embodiments, as a specific implementation of the online interaction method based on recorded content in the above - mentioned embodiments, some embodiments of the present application also provide an online interaction system based on recorded content. The system architecture of the system can include a front - end module, a back - end module, a script engine, and a database. Among them, the front - end module is responsible for presenting the user interface. For example, the user interface includes teacher - user videos, student - user videos, interactive interfaces, etc. The front - end module can implement video playback and real - time interaction through technologies such as WebRTC.

[0140] The back - end module is responsible for script management, video - stream control, and data interaction, etc. The back - end module can use the built - in storage unit of the server to store pre - recorded video data and trigger corresponding interaction events according to the script data. The script engine can be one of the core components of the online interaction system, used to parse and execute the script data. The script data format is such as JSON, which contains information such as timestamps, action types, target objects, etc. The database is used to store user videos, script data, and user information, etc.

[0141] As Figure 10 shown, the system includes a server and at least one client, and the client establishes a communication connection relationship with the server. The server includes:

[0142] A first playback module for playing recorded content data, where the recorded content data includes pre-recorded video data and interaction data; the video data includes a first video and a second video with the same total playback duration and playback progress.

[0143] An instruction acquisition module for acquiring online interaction instructions, where the online interaction instructions are input by a client and / or generated by detecting the playback progress of the recorded content data.

[0144] An interaction feedback module for generating interaction feedback data in response to the online interaction instructions, where the interaction feedback data includes the video data and / or the interaction data extracted from the recorded content data based on the online interaction instructions, and an interaction screen rendered in real time based on the online interaction instructions.

[0145] A second playback module for playing the interaction feedback data.

[0146] By applying the technical solutions of the above embodiments, the above embodiments provide an online interaction system based on recorded content. The system can acquire online interaction instructions during the playback of recorded content data, and in response to the online interaction instructions, generate interaction feedback data and play the interaction feedback data according to the online interaction instructions. Among them, the interaction feedback data includes the video data and / or the interaction data extracted from the recorded content data based on the online interaction instructions, and an interaction screen rendered in real time based on the online interaction instructions. The system can simulate the online interaction process among multiple users through a dynamically triggered interaction mechanism, enhance the user's sense of participation, and solve the problem of inability to interact in real time when playing recorded content. Moreover, through the generation and management of the recorded content data, the interaction organization cost can be reduced, and the operation efficiency of the online interaction platform can be improved.

[0147] It should be noted that for other corresponding descriptions of each functional unit involved in the online interaction system based on recorded content provided in the embodiments of the present application, reference can be made to the corresponding descriptions in the online interaction method based on recorded content provided in the above embodiments, and details are not described herein again.

[0148] The embodiments of the present application further provide a computer device, which may specifically be a personal computer, a server, a network device, etc. The computer device includes a bus, a processor, a memory, and a communication interface, and may further include an input / output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements the steps in the method embodiments.

[0149] Those skilled in the art can understand that the structure of the above computer device is only a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0150] In one embodiment, a computer-readable storage medium is further provided. The computer-readable storage medium may be non-volatile or volatile, and has a computer program stored thereon. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0151] In one embodiment, a computer program product is further provided, including a computer program. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data that have been authorized by the user or fully authorized by all parties.

[0153] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the above method embodiments.

[0154] Among them, any reference to a memory, a database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memories. The non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0155] The volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0156] The databases involved in the embodiments provided in this application may include at least one of relational databases and non-relational databases. The non-relational databases may include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0157] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0158] The above-described embodiments only represent several implementation manners of this application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application should be subject to the appended claims.

Claims

1. An online interaction method based on recorded content, characterized in that, The method includes: Playing recorded content data, where the recorded content data includes pre-recorded video data and interaction data; the video data includes a first video and a second video with the same total playing duration and playing progress; Obtaining an online interaction instruction, where the online interaction instruction is input by a client and / or generated by detecting the playing progress of the recorded content data; Responding to the online interaction instruction, generating interaction feedback data according to the online interaction instruction, where the interaction feedback data includes the video data and / or the interaction data extracted from the recorded content data based on the online interaction instruction, and an interaction screen rendered in real time based on the online interaction instruction; Playing the interaction feedback data.

2. The method according to claim 1, wherein Playing the recorded content data includes: Receiving a content playing instruction sent by the client; Responding to the content playing instruction, obtaining the recorded content data; Invoking a first player and a second player, where the first player and / or the second player is set with a time synchronization algorithm, and the time synchronization algorithm is used to send a synchronization instruction to the second player based on the playing time of the first player; Playing the first video through the first player and playing the second video through the second player.

3. The method according to claim 1, wherein Playing the recorded content data includes: Obtaining a first playing time of the first video and a second playing time of the second video; If the first playing time is not equal to the second playing time, setting the playing speed of the second video according to the first playing time; If the first playing time is equal to the second playing time, continuously monitoring the first playing time and the second playing time.

4. The method according to claim 1, wherein Playing the recorded content data includes: Obtaining a playing screen of the recorded content data, where the playing screen includes a first video screen, a second video screen, and an interaction screen; the first video screen is generated by playing the first video; the second video screen is generated by playing the second video; the interaction screen is generated by displaying the interaction data; Displaying the playing screen through an online interaction interface, where the online interaction interface includes a core display area, a user display area, and an interaction display area; the core display area is used to display the first video screen; the user display area is used to display the second video screen; the interaction display area is used to display the interaction screen.

5. The method according to claim 4, characterized in that, The second video includes pre-recorded user video and / or real-time user video. Displaying the playing screen through the online interaction interface includes: Obtaining the number of second videos and the number of display columns in the user display area; If the number of second videos is less than the number of display columns, calculating a padding number, where the padding number is equal to the difference between the number of display columns and the number of second videos; Loading padding data according to the padding number, where the padding data includes virtual user models; the virtual user models are user image models constructed based on the interaction data; Displaying the playing screen corresponding to the padding data through the user display area.

6. The method according to claim 1, wherein The interaction data includes script data. Obtaining the online interaction instruction includes: Obtain the current playback progress of the first video; Invoke a script engine, which is configured to parse and execute the script data, and the script data includes interaction timestamps, interaction action types, and interaction target objects; Use the script engine to read the script data at a preset time interval; If the current playback progress hits the interaction timestamp in the script data, generate the online interaction instruction.

7. The method according to claim 1, characterized in that, Generate interaction feedback data according to the online interaction instruction, including: Obtain the interaction text information input by the client in real time and the current playback progress of the first video; Convert the interaction text information into a preset data format to generate a first-format text; Based on the current playback progress and the first-format text, extract associated text information from the interaction data, and the associated text information includes historical interaction texts recorded during the playback of the recorded content data; Convert the associated text information into a preset data format to generate a second-format text; Generate the interaction feedback data according to the first-format text and the second-format text.

8. The method according to claim 1, characterized in that Generate interaction feedback data according to the online interaction instruction, including: Parse the instruction content information from the online interaction instruction; Perform data assembly according to the instruction content information to generate interaction instruction data, and the interaction instruction data includes at least one of interaction types, interaction times, and interaction rules; Push the interaction instruction data to the core display area of the online interaction interface to display interaction controls in the core display area; Receive the interaction operation data input by the client based on the interaction controls, and extract the interaction feedback data from the recorded content data according to the interaction operation data.

9. The method according to claim 8, characterized in that, Extract the interaction feedback data from the recorded content data according to the interaction operation data, including: Obtain a preset evaluation rule; Perform an evaluation process on the interaction operation data according to the preset evaluation rule to generate an evaluation result; Render an interaction screen in real time according to the evaluation result, and generate the interaction feedback data according to the interaction screen; Send the interaction feedback data to the client.

10. An online interaction system based on recorded content, characterized in that, The system includes a server and at least one client, the client establishes a communication connection relationship with the server, and the server includes: A first playback module for playing recorded content data, and the recorded content data includes pre-recorded video data and interaction data; the video data includes a first video and a second video with the same total playback duration and playback progress; An instruction acquisition module for acquiring an online interaction instruction, which is input by the client and / or generated by detecting the playback progress of the recorded content data; An interaction feedback module for responding to the online interaction instruction and generating interaction feedback data according to the online interaction instruction, and the interaction feedback data includes the video data and / or the interaction data extracted from the recorded content data based on the online interaction instruction, and an interaction screen rendered in real time based on the online interaction instruction; A second playback module for playing the interaction feedback data.

Images