Interaction method, device and equipment of electronic whiteboard all-in-one machine and storage medium

Abstract

The invention provides an interaction method and device of an electronic whiteboard all-in-one machine, equipment and a storage medium, a virtual exhibition stand interface is displayed through a trigger instruction of the electronic whiteboard all-in-one machine, under the condition that the virtual exhibition stand interface is displayed, playing content is determined and played in the virtual exhibition stand interface, and in response to an annotation operation, the interaction efficiency of the electronic whiteboard all-in-one machine is improved. The comment content is generated, the content type of the comment content is determined, and the playing control operation and the comment operation are supported through the independently designed virtual exhibition stand interface, so that a teacher can select the playing content and perform the comment operation at any time in the playing process of the playing content to generate the comment content; teaching material forms can be enriched, interactivity can be enhanced, and the teaching effect can be enhanced; according to the content type, the associated content corresponding to the annotation content is generated, and the interactive playing content is synthesized according to the annotation content, the associated content and the playing content, so that when the interactive playing content is watched, the annotation content and the associated content are directly watched, and review and consolidation of knowledge are facilitated.

Description

Technical Field

[0001] This application relates to the field of electronic whiteboards, and more particularly to an interactive method, apparatus, device, and storage medium for an all-in-one electronic whiteboard. Background Technology

[0002] With the development of interactive whiteboard technology, interactive whiteboards have been widely used in various industries, such as education and conference settings. However, in teaching scenarios, when teachers need to play video or audio content related to a lesson, explanations are limited to verbal descriptions or post-playing explanations. The lack of support for annotation during playback results in poor interactivity and hinders effective teaching. Summary of the Invention

[0003] This application provides an interactive method, apparatus, device, and storage medium for an all-in-one electronic whiteboard to solve at least one problem existing in the related technology. The technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide an interactive method for an all-in-one electronic whiteboard, comprising:

[0005] The virtual booth interface is displayed by triggering commands to the interactive whiteboard.

[0006] When the virtual booth interface is displayed, determine the content to be played and play it in the virtual booth interface. The virtual booth interface supports playback control operations and annotation operations.

[0007] In response to annotation operations, generate annotation content and determine the content type of the annotation content;

[0008] Based on the content type, generate associated content corresponding to the annotation content, and synthesize interactive playback content based on the annotation content, associated content, and playback content.

[0009] In one implementation, generating annotation content in response to the annotation operation includes:

[0010] At the start of the annotation operation, obtain the first system absolute time of the electronic whiteboard all-in-one machine and the playback time of the content, and extract the key elements corresponding to the playback time in the content.

[0011] During the annotation operation, annotation information is collected until the annotation operation ends, and the absolute time of the second system of the electronic whiteboard all-in-one machine is obtained when the annotation operation ends.

[0012] Determine the time difference between the absolute time of the second system and the absolute time of the first system, and determine the correction time based on the time difference and the playback time;

[0013] Insert a blank segment in the playback content with a duration from the playback time to the correction time, and generate annotation content based on key elements and annotation information; the blank segment is used to play the annotation content.

[0014] In one implementation, generating annotation content based on key elements and annotation information includes:

[0015] When the annotation information is audio and the playback content is video, extract key video frames and convert the audio into text content, display it in the extracted key video frames, and generate annotation content.

[0016] When the annotation information is speech and the playback content is audio, the speech will be used as the annotation content;

[0017] When the annotation information is a trajectory and the playback content is video, extract key video frames and display the trajectory in the extracted key video frames to generate annotation content.

[0018] Key elements include key video frames or key audio frames.

[0019] In one implementation, the content type for determining the annotation content includes:

[0020] The annotation content is identified using a text recognition model to determine whether text exists. If text exists, the content type is determined to include text.

[0021] The annotation content is subjected to symbol feature detection to determine whether symbols exist. If symbols exist, the annotation content is subjected to spatial structure analysis to determine whether formulas exist. If formulas exist, the content type is determined to include formulas.

[0022] In one implementation, generating associated content corresponding to the annotation content based on the content type includes:

[0023] When the content type includes formulas, the formula structure is broken down by the formula parser to determine the formula parameters. Based on the formula parameters, all related formulas associated with the formula parameters are matched from the preset knowledge graph as related content.

[0024] When the content type includes text, determine whether the electronic whiteboard all-in-one machine system has courseware running in the background or in the foreground of an inactive window. If so, match the text with the courseware, determine the matching page, generate index tags for the matching page, and obtain the associated content.

[0025] In one implementation, the step of synthesizing interactive playback content based on annotation content, associated content, and playback content includes:

[0026] When the content being played is a video, the annotations and related content are inserted into the blank segment for display. Based on the inserted blank segment and the content being played, interactive playback content is synthesized.

[0027] When the playback content is audio, if the annotation content is speech, the annotation content will be inserted into the blank segment, and the related content will be displayed through the word interface within the duration of the blank segment. If the annotation content is text, the text content will be converted into speech and inserted into the blank segment, and the related content will be displayed through the word interface within the duration of the blank segment. If the annotation content is a trajectory, the text or formula will be displayed through the word interface within the duration of the blank segment.

[0028] In one embodiment, the method further includes:

[0029] When playing interactive content on the virtual booth interface, during blank segments:

[0030] If you click the index tab, the matching page from the corresponding courseware will be displayed in the virtual booth interface;

[0031] If you click on any of the related formulas in the related content, the corresponding derivation and explanation video will be searched and played in the virtual booth interface.

[0032] Secondly, embodiments of this application provide an interactive device for an all-in-one electronic whiteboard, comprising:

[0033] The display module is used to display the virtual booth interface by triggering commands to the electronic whiteboard all-in-one machine;

[0034] The playback module is used to determine the content to be played and play it in the virtual booth interface when the virtual booth interface is displayed. The virtual booth interface supports playback control operations and annotation operations.

[0035] The response module is used to respond to annotation operations, generate annotation content, and determine the content type of the annotation content.

[0036] The synthesis module is used to generate associated content corresponding to the annotation content based on the content type, and to synthesize interactive playback content based on the annotation content, associated content, and playback content.

[0037] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, wherein the memory stores instructions that are loaded and executed by the processor to implement the methods in any of the above-described embodiments.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed, implements the methods in any of the above-described embodiments.

[0039] The beneficial effects of the above technical solution include at least the following:

[0040] The interactive whiteboard displays a virtual showcase interface via a trigger command. Once displayed, the playback content is selected and played within the virtual showcase. In response to annotation operations, annotation content is generated and its type is determined. This self-designed virtual showcase interface supports playback control and annotation operations, allowing teachers to select playback content based on actual needs. Furthermore, annotations can be generated at any time during playback, enriching teaching materials, enhancing interactivity, and strengthening teaching effectiveness. Based on the content type, corresponding related content is generated for the annotations. These annotations, related content, and the playback content are then combined to create interactive playback content. This allows teachers and students to directly view the annotations and related content while watching the interactive playback, facilitating knowledge review and consolidation.

[0041] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0042] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0043] Figure 1 This is a flowchart illustrating the steps of an interactive method for an electronic whiteboard all-in-one machine according to an embodiment of this application;

[0044] Figure 2 This is a structural block diagram of the interactive device of an electronic whiteboard all-in-one machine according to an embodiment of this application;

[0045] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0046] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0047] Reference Figure 1The flowchart illustrates an interactive method for an electronic whiteboard all-in-one machine according to an embodiment of this application. This interactive method may include at least steps S100-S400:

[0048] S100: By triggering the electronic whiteboard all-in-one machine, a virtual booth interface is displayed.

[0049] S200: When the virtual booth interface is displayed, determine the content to be played and play it in the virtual booth interface. The virtual booth interface supports playback control operations and annotation operations.

[0050] S300: In response to the annotation operation, generate annotation content and determine the content type of the annotation content.

[0051] S400. Based on the content type, generate associated content corresponding to the annotation content, and synthesize interactive playback content based on the annotation content, associated content, and playback content.

[0052] The interactive method of the electronic whiteboard all-in-one machine in this application embodiment can be executed by the electronic whiteboard all-in-one machine.

[0053] The technical solution of this application embodiment displays a virtual exhibition interface by triggering an instruction on the electronic whiteboard all-in-one machine. While the virtual exhibition interface is displayed, the playback content is determined and played within it. In response to annotation operations, annotation content is generated and its type is determined. Through a self-designed virtual exhibition interface, playback control and annotation operations are supported, allowing teachers to select playback content based on actual needs. Furthermore, annotation operations can be performed at any time during playback to generate annotation content, enriching teaching materials, enhancing interactivity, and strengthening teaching effectiveness. Based on the content type, corresponding related content is generated, and interactive playback content is synthesized from the annotation content, related content, and playback content. This allows teachers and students to directly see the annotation content and related content while viewing the interactive playback content, facilitating knowledge review and consolidation.

[0054] In one implementation, a virtual booth (e.g., a program or plugin) is designed using development tools and a programming language. Triggering the virtual booth leads to its interface, where users can control audio and video playback (e.g., add, play, pause, fast forward, etc.) and add annotations. Users can input trigger commands on the interactive whiteboard, such as clicking the virtual booth or waking it up with voice commands, thus displaying the virtual booth interface on the whiteboard's screen.

[0055] In one implementation, when the virtual booth interface is displayed, the user can add and specify the content to be played through the add operation, and then play the content in the virtual booth interface. The content can be audio or video.

[0056] In one implementation, step S300, in response to the annotation operation, generates annotation content, including steps S310-S340:

[0057] S310. When the annotation operation begins, obtain the first system absolute time of the electronic whiteboard all-in-one machine and the playback time of the content, and extract the key elements corresponding to the playback time in the content.

[0058] Optionally, at the start of the annotation operation, i.e., the initial moment when the user begins the annotation operation, the first system absolute time of the electronic whiteboard and the playback time of the content are obtained. For example, if the first system absolute time is 13:31:00 and the playback time is 00:00:13 (i.e., the 13th second), then the key element corresponding to this playback time is extracted. Optionally, if the playback content is video, the key element can be a key video frame; if the playback content is audio, the key element can be a key audio frame. For example, if the playback content is video, the video frame displayed at 00:00:13 is used as the key video frame; if the playback content is audio, the audio frame at 00:00:13 is used as the key audio frame.

[0059] S320. During the annotation operation, collect annotation information until the annotation operation ends, and obtain the second system absolute time of the electronic whiteboard all-in-one machine when the annotation operation ends.

[0060] Optionally, during the annotation operation, annotation information is collected until the annotation operation ends, and the absolute time of the second system of the electronic whiteboard is obtained when the annotation operation ends, for example, 13:31:09.

[0061] S330. Determine the time difference between the absolute time of the second system and the absolute time of the first system, and determine the correction time based on the time difference and the playback time.

[0062] For example, if the absolute time of the second system is 13:31:09 and the absolute time of the first system is 13:31:00, then the time difference is determined to be 9 seconds, meaning that the user made a 9-second annotation. Then, based on the 9-second time difference, the playback time (e.g., 00:00:13) is compensated, and the corrected time is determined to be 00:00:22.

[0063] S340. Insert a blank segment in the playback content with a duration equal to the playback time to the correction time, and generate annotation content based on key elements and annotation information; the blank segment is used to play the annotation content.

[0064] Optionally, a blank segment with a duration equal to the playback time up to the corrected time (i.e., 9 seconds starting from 00:00:13) is inserted into the playback content. Then, annotation content is generated based on key elements and annotation information, including:

[0065] 1. When the annotation information is audio and the playback content is video, extract key video frames and convert the audio into text content, display it in the extracted key video frames, and generate annotation content.

[0066] Optionally, users (such as teachers) can provide explanations via voice, meaning the annotations are in audio format. When the content is video, the audio is converted to text, key video frames (equivalent to frozen static backgrounds) are extracted, and the text content is displayed within these key video frames to generate annotations. For example, if a teacher is explaining the characteristics of a geometric shape, and the key video frame contains the geometric shape, then the text content is displayed on that key video frame, allowing students to see the explanation while viewing the geometric shape, facilitating learning.

[0067] 2. When the annotation information is voice and the playback content is audio, the voice will be used as the annotation content.

[0068] Optionally, when the annotation information is speech and the playback content is audio, the speech can be directly used as the annotation content.

[0069] 3. When the annotation information is a trajectory and the playback content is video, extract key video frames and display the trajectory in the extracted key video frames to generate annotation content.

[0070] Optionally, users can also provide explanations by handwriting, where the annotation information is a handwritten trajectory. If the content being played is a video, key video frames are extracted and the trajectory is displayed in the extracted key video frames to generate annotation content. The display principle is similar to that of the text content mentioned above, and will not be elaborated further.

[0071] Optionally, when the annotation information is a trajectory and the playback content is audio, the user is prompted to input the annotation information by voice.

[0072] In this embodiment, the blank segment is used for subsequent playback of annotation content; annotations are recorded independently using system time, which helps to ensure independence and prevents them from being confused or lost.

[0073] In one implementation, determining the content type of the annotation in step S300 includes steps S350-S360:

[0074] S350. The annotation content is identified by a text recognition model to determine whether there is text. If there is text, the content type is determined to include text.

[0075] Optionally, the text recognition model can be a CRNN model or a Transformer model to recognize the annotation content and determine whether text exists. If text exists, the content type is determined to include text (e.g., Chinese characters); otherwise, the content type is determined not to include text, and may be symbols, lines, patterns, etc.

[0076] S360. Perform symbol feature detection on the annotation content to determine whether symbols exist. If symbols exist, perform spatial structure analysis on the annotation content to determine whether formulas exist. If formulas exist, determine that the content type includes formulas.

[0077] Optionally, symbol feature detection can be performed based on a deep learning-based character recognition model to determine the presence of symbols. For example, CNN (Convolutional Neural Network) or Transformer architectures (such as ViT) can be used to classify mathematical symbols, and the mathematical symbol dataset can be fine-tuned using pre-trained models (such as ResNet or EfficientNet). During the classification process, attention mechanisms (such as the Squeeze-and-Excitation module) can be introduced to enhance the sensitivity to small-scale symbols, thereby improving the accuracy of mathematical symbol classification and recognition. If symbols exist, graph neural networks (GNNs) or relational networks can be used to treat symbols as nodes, and the spatial relationships between symbols can be modeled through graph structures (such as the vertical relationship of fraction lines, the dependency relationship between integral symbols and dx). Spatial structure analysis can be performed on the annotation content to determine whether formulas exist. If formulas exist, the content type is determined to include formulas.

[0078] It should be noted that when there are no symbols or symbols but no formulas, and if there is no text, the annotation content can be considered as lines and patterns, and the subsequent related content is determined to be "blank".

[0079] In one implementation, step S400 generates associated content corresponding to the annotation content based on the content type, including steps S410-S420:

[0080] S410. When the content type includes formulas, the formula structure is broken down by the formula parser to determine the formula parameters. Based on the formula parameters, all related formulas associated with the formula parameters are matched from the preset knowledge graph as related content.

[0081] Optionally, the formula parser includes, but is not limited to, MathPix API, Pix2Tex, and SymPy. The preset knowledge graph is a pre-built structure relating the various formula parameters involved in the course and the various formulas involved in those formula parameters. In this embodiment, the formula parser decomposes the formula structure to determine the formula parameters, i.e., the various parameters involved in the formula. Then, based on the formula parameters, all associated formulas related to the formula parameters are matched from the preset knowledge graph as associated content.

[0082] S420. When the content type includes text, determine whether the electronic whiteboard all-in-one machine system has courseware running in the background or in the foreground of an inactive window. If so, match the text with the courseware, determine the matching page and generate the index tag of the matching page to obtain the associated content.

[0083] Optionally, when the content type includes text, the system determines whether the interactive whiteboard system contains courseware running in the background or in the foreground of an inactive window. For example, if the teacher opens the courseware and the virtual showcase is on top, the courseware is running in the foreground of an inactive window. Alternatively, if the courseware window is minimized after opening, the courseware is running in the background. If such courseware exists, it means the teacher is likely using video or audio playback to supplement a part of the courseware. In this case, the system automatically matches the text with the courseware, finds the page in the courseware containing the text as the matching page, and generates an index tag for the matching page to obtain the associated content.

[0084] In this embodiment, the index label can be a hyperlink that allows direct navigation, or a screenshot of the matching page, helping users quickly locate the matching page in the courseware. Through the above method, the matching page corresponding to the currently playing content is automatically found based on text matching with the courseware, which is convenient, fast, and highly intelligent.

[0085] In one implementation, step S400 involves synthesizing interactive playback content based on annotation content, related content, and playback content, including steps S430-S440:

[0086] S430. When the content being played is a video, the annotation content and related content are inserted into the blank segment for display. Based on the inserted blank segment and the content being played, interactive playback content is synthesized.

[0087] In this embodiment, when the playback content is a video, the annotation content and related content are directly inserted into the blank segment for display. Then, the inserted blank segment and the playback content are used to synthesize the complete interactive playback content. When students or teachers watch the interactive playback content later, in addition to the original playback content, they can also directly see the annotation content and related content when the playback time of the blank segment is reached, which is conducive to enhancing the teaching effect and interactivity.

[0088] S440. When the playback content is audio, if the annotation content is speech, the annotation content is inserted into the blank segment, and the related content is displayed through the word interface within the duration of the blank segment. If the annotation content is text, the text content is converted into speech and inserted into the blank segment, and the related content is displayed through the word interface within the duration of the blank segment. If the annotation content is a trajectory, the text or formula is displayed through the word interface within the duration of the blank segment.

[0089] In this embodiment, when the playback content is audio, if the annotation content is speech, the annotation content is inserted into the blank segment, and the related content is displayed through the word interface (a sub-area in the virtual booth interface, similar to the lyrics interface of a music player) during the duration of the blank segment. If the annotation content is text, the text content is converted into speech and inserted into the blank segment, and the related content is displayed through the word interface during the duration of the blank segment. If the annotation content is a trajectory, the result of trajectory recognition, i.e., text or formula, is displayed through the word interface during the duration of the blank segment. Therefore, even when playing audio, the viewer can still be provided with the display of annotation content and related content, making it more user-friendly.

[0090] In one embodiment, the interactive method of the electronic whiteboard all-in-one machine of this application may further include: when playing interactive content on a virtual booth interface, during the playback of a blank segment:

[0091] 1. If you click the index tab, the matching page from the corresponding courseware will be displayed in the virtual showcase interface.

[0092] Optionally, after the interactive content is generated by the electronic whiteboard all-in-one machine, the virtual display interface will pop up options for teachers to choose a saving method or a sharing method. Teachers can save it locally, on a USB flash drive or cloud drive, or share it to a shared cloud drive or communication software for students to access the interactive content for review.

[0093] In one implementation, since a teacher may teach more than one class, after annotating the content for one class, the interactive content can be played directly in the virtual showcase interface when teaching the next class. When playing interactive content in the virtual showcase interface, if the teacher clicks an index tag during a blank segment, the corresponding page from the courseware will be displayed directly in the virtual showcase interface. For example, if it's a hyperlink, it will directly jump to the corresponding page in the courseware and display it; if it's a screenshot, it will display the corresponding page in the courseware, allowing the teacher to quickly find the location of the corresponding page in the courseware.

[0094] In another implementation, students can access the virtual showcase via their terminals and play interactive content within it. Similarly, when playing interactive content on the virtual showcase, students can view annotations and related information for learning. Furthermore, if the teacher has shared courseware, students can click on index tags to display the corresponding pages within the courseware, helping them quickly locate the relevant sections and improving learning efficiency and effectiveness.

[0095] 2. If you click on any of the related formulas in the related content, the corresponding derivation and explanation video will be searched and played in the virtual exhibition stand interface.

[0096] Optionally, if a user (including a teacher or student) clicks on any of the related formulas in the related content, the derivation and explanation video corresponding to the formula will be searched from the preset knowledge graph, the shared cloud drive pre-set by the teacher, or the Internet and played in the virtual exhibition interface, which is convenient for teachers to teach and students to learn.

[0097] In some embodiments, the knowledge graph may also include links to explanatory videos of the derivation of each formula explained by the teacher. That is, the associated content may also include links to the explanatory videos, so the derivation videos can be played in the virtual booth interface based on the links from the knowledge graph.

[0098] Reference Figure 2 The diagram shows a structural block diagram of an interactive device for an electronic whiteboard all-in-one machine according to an embodiment of this application. The device may include:

[0099] The display module is used to display the virtual booth interface by triggering commands to the electronic whiteboard all-in-one machine;

[0100] The playback module is used to determine the content to be played and play it in the virtual booth interface when the virtual booth interface is displayed. The virtual booth interface supports playback control operations and annotation operations.

[0101] The response module is used to respond to annotation operations, generate annotation content, and determine the content type of the annotation content.

[0102] The synthesis module is used to generate associated content corresponding to the annotation content based on the content type, and to synthesize interactive playback content based on the annotation content, associated content, and playback content.

[0103] The functions of each module in the device of this application embodiment can be found in the corresponding description in the above method, and will not be repeated here.

[0104] Reference Figure 3 The diagram illustrates a structural block diagram of an electronic device according to an embodiment of this application. The electronic device includes a memory 310 and a processor 320. The memory 310 stores instructions that can be executed on the processor 320. The processor 320 loads and executes these instructions to implement the interactive method of the electronic whiteboard all-in-one machine in the above embodiment. The number of memories 310 and processors 320 can be one or more.

[0105] In one embodiment, the electronic device further includes a communication interface 330 for communicating with external devices and exchanging data. If the memory 310, processor 320, and communication interface 330 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 310, processor 320 and communication interface 330 are integrated on a single chip, the memory 310, processor 320 and communication interface 330 can communicate with each other through an internal interface.

[0107] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the interactive method of the electronic whiteboard all-in-one machine provided in the above embodiments.

[0108] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the method provided in this application.

[0109] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0110] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.

[0111] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0112] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0113] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0114] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0115] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0117] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An interactive method for an all-in-one electronic whiteboard, characterized in that, include: The virtual booth interface is displayed by triggering commands to the interactive whiteboard. When the virtual booth interface is displayed, determine the content to be played and play it in the virtual booth interface. The virtual booth interface supports playback control operations and annotation operations. In response to annotation operations, generate annotation content and determine the content type of the annotation content; Based on the content type, generate associated content corresponding to the annotation content, and synthesize interactive playback content based on the annotation content, associated content, and playback content.

2. The interactive method of the electronic whiteboard all-in-one machine according to claim 1, characterized in that: The response to the annotation operation generates annotation content including: At the start of the annotation operation, obtain the first system absolute time of the electronic whiteboard all-in-one machine and the playback time of the content, and extract the key elements corresponding to the playback time in the content. During the annotation operation, annotation information is collected until the annotation operation ends, and the absolute time of the second system of the electronic whiteboard all-in-one machine is obtained when the annotation operation ends. Determine the time difference between the absolute time of the second system and the absolute time of the first system, and determine the correction time based on the time difference and the playback time; Insert a blank segment in the playback content with a duration from the playback time to the correction time, and generate annotation content based on key elements and annotation information; the blank segment is used to play the annotation content.

3. The interactive method of the electronic whiteboard all-in-one machine according to claim 2, characterized in that: The process of generating annotation content based on key elements and annotation information includes: When the annotation information is audio and the playback content is video, extract key video frames and convert the audio into text content, display it in the extracted key video frames, and generate annotation content. When the annotation information is speech and the playback content is audio, the speech will be used as the annotation content; When the annotation information is a trajectory and the playback content is video, extract key video frames and display the trajectory in the extracted key video frames to generate annotation content. Key elements include key video frames or key audio frames.

4. The interactive method of the electronic whiteboard all-in-one machine according to claim 2, characterized in that: The content types for determining the annotation content include: The annotation content is identified using a text recognition model to determine whether text exists. If text exists, the content type is determined to include text. The annotation content is subjected to symbol feature detection to determine whether symbols exist. If symbols exist, the annotation content is subjected to spatial structure analysis to determine whether formulas exist. If formulas exist, the content type is determined to include formulas.

5. The interactive method of the electronic whiteboard all-in-one machine according to claim 4, characterized in that: The process of generating associated content corresponding to annotation content based on content type includes: When the content type includes formulas, the formula structure is broken down by the formula parser to determine the formula parameters. Based on the formula parameters, all related formulas associated with the formula parameters are matched from the preset knowledge graph as related content. When the content type includes text, determine whether the electronic whiteboard all-in-one machine system has courseware running in the background or in the foreground of an inactive window. If so, match the text with the courseware, determine the matching page, generate index tags for the matching page, and obtain the associated content.

6. The interactive method of the electronic whiteboard all-in-one machine according to claim 5, characterized in that: The process of synthesizing interactive playback content based on annotation content, related content, and playback content includes: When the content being played is a video, the annotations and related content are inserted into the blank segment for display. Based on the inserted blank segment and the content being played, interactive playback content is synthesized. When the playback content is audio, if the annotation content is speech, the annotation content will be inserted into the blank segment, and the related content will be displayed through the word interface within the duration of the blank segment. If the annotation content is text, the text content will be converted into speech and inserted into the blank segment, and the related content will be displayed through the word interface within the duration of the blank segment. If the annotation content is a trajectory, the text or formula will be displayed through the word interface within the duration of the blank segment.

7. The interactive method of the electronic whiteboard all-in-one machine according to claim 6, characterized in that: The method further includes: When playing interactive content on the virtual booth interface, during blank segments: If you click the index tab, the matching page from the corresponding courseware will be displayed in the virtual booth interface; If you click on any of the related formulas in the related content, the corresponding derivation and explanation video will be searched and played in the virtual booth interface.

8. An interactive device for an all-in-one electronic whiteboard, characterized in that, include: The display module is used to display the virtual booth interface by triggering commands to the electronic whiteboard all-in-one machine; The playback module is used to determine the content to be played and play it in the virtual booth interface when the virtual booth interface is displayed. The virtual booth interface supports playback control operations and annotation operations. The response module is used to respond to annotation operations, generate annotation content, and determine the content type of the annotation content. The synthesis module is used to generate associated content corresponding to the annotation content based on the content type, and to synthesize interactive playback content based on the annotation content, associated content, and playback content.

9. An electronic device, characterized in that, include: A processor and a memory, wherein instructions are stored in the memory and loaded and executed by the processor to implement the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program therein, which, when executed, implements the method as described in any one of claims 1-7.

Images