AR-based video teaching method and electronic equipment

Abstract

The present invention belongs to the technical field of online education, and provides an AR-based video teaching method and electronic device, the method comprising: in response to an instruction to turn on an AR interactive function, performing action recognition processing on a user's real-time video to determine whether the user makes a predefined characteristic action; if the user makes the predefined characteristic action, obtaining a target virtual object corresponding to the predefined characteristic action; drawing the target virtual object in the user's real-time video to form a teaching video. The AR-based video teaching method and electronic device disclosed in the present invention can provide users with a convenient and fast operation method to call out virtual objects, assist teaching work through virtual objects, increase interactive fun, and improve teaching effects.

Description

Technical Field

[0001] The present invention relates to the field of online education, and in particular to an AR-based video teaching method and electronic equipment. Background Art

[0002] In recent years, as traditional teaching models have become increasingly unable to meet students' needs for modern teaching methods such as multimedia, information technology, and easy playback, with the rapid development and popularization of Internet technology, especially mobile Internet technology, various Internet-based teaching platforms or teaching systems have emerged and flourished. With the rapid development of mobile terminal technology, online training has gradually replaced offline training models.

[0003] In teaching videos involving a lot of professional knowledge, lecturers usually use teaching files such as PPT or WORD to show the structured knowledge context, and explain the course content in voice. In a one-to-many teaching scenario, the interaction between teachers and students is limited. Basically, teachers explain on the teacher terminal, and students watch and learn on the student terminal. This makes online teaching less interesting, and there may also be a problem of low student participation in class.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0005] In view of this, the present invention provides an AR-based video teaching method and electronic device, which can easily and quickly call out virtual objects during teaching, assist teaching work through virtual objects, increase interactive fun, and improve teaching effects.

[0006] To solve the above technical problems, one aspect of the present invention proposes an AR-based video teaching method, which includes: in response to a user's instruction to turn on an AR interactive function, performing action recognition processing on the user's real-time video to determine whether the user makes a predefined feature action; if the user makes the predefined feature action, obtaining a target virtual object corresponding to the predefined feature action; and drawing the target virtual object in the user's real-time video to form a teaching video.

[0007] In an exemplary embodiment of the present invention, the method further includes: acquiring a display position of the target virtual object; and drawing the target virtual object in the real-time video based on the display position.

[0008] In an exemplary embodiment of the present invention, based on the target virtual object, obtaining the display position of the target virtual object includes: when the display position of the target virtual object is related to the action area where the user performs the predefined feature action, obtaining the action area where the user performs the predefined feature action; determining the display position of the target virtual object based on the action area; and / or, when the display position of the target virtual object is related to the position of a reference object, obtaining the position information of the reference object; and determining the display position of the target virtual object based on the position information of the reference object.

[0009] In an exemplary embodiment of the present invention, it also includes: when the target virtual object is drawn in the real-time video for display, obtaining the user's operation instructions for the target virtual object; in the teaching video formed based on the real-time video of the user, displaying the operation process and / or operation effect of the target virtual object based on the operation instructions.

[0010] In an exemplary embodiment of the present invention, obtaining the operation instruction of the user includes: when the target virtual object is displayed in the real-time video, obtaining the user's operation action on the target virtual object by performing action recognition processing on the user's real-time video; based on the target virtual object and the operation action, obtaining the operation instruction for the target virtual object corresponding to the operation action.

[0011] In an exemplary embodiment of the present invention, the target virtual object is a teaching auxiliary item; the operation actions include: scaling, moving, changing display angle, releasing, and removing.

[0012] In an exemplary embodiment of the present invention, one or more of the following situations are included: the target virtual object is a virtual writing tool, and as the virtual writing tool moves, the moving trajectory of the virtual writing tool is displayed in the teaching video as a writing trace; the target virtual object is a virtual erasing tool, and as the virtual erasing tool moves, the writing traces on the moving path of the virtual erasing tool are cleared in the teaching video; the target virtual object is a virtual experimental tool, and according to the user's operating actions on the virtual experimental tool, the predefined operation process and / or operation effect of the virtual experimental tool is displayed in the teaching video.

[0013] In an exemplary embodiment of the present invention, the real-time video of the user is processed to determine whether the user performs a predefined characteristic action, including: inputting the real-time video into an action recognition model for action recognition, and determining whether the recognized user action belongs to an action in a predefined characteristic action set, and the action recognition model is implemented by a deep learning model.

[0014] In an exemplary embodiment of the present invention, before turning on the AR interaction function, the video teaching method further includes: performing action recognition processing on the user's real-time video to determine whether the user makes a first feature action corresponding to turning on the AR interaction function; after turning on the AR interaction function, the video teaching method further includes: performing action recognition processing on the user's real-time video to determine whether the user makes a second feature action corresponding to turning off the AR interaction function.

[0015] The second aspect of the present invention proposes an AR-based video teaching device, which includes: a recognition module, which is used to respond to the user's instruction to turn on the AR interactive function, and perform action recognition processing on the user's real-time video to determine whether the user makes a predefined feature action; a virtual object acquisition module, which is used to acquire the target virtual object corresponding to the predefined feature action when it is determined that the user makes the predefined feature action; and a display module, which is used to draw the target virtual object in the user's real-time video to form a teaching video.

[0016] The third aspect of the present invention proposes an electronic device, comprising one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement any of the video teaching methods described above.

[0017] A fourth aspect of the present invention further proposes a computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the video teaching method as described above is implemented.

[0018] The fifth aspect of the present invention further proposes a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements any of the above-mentioned video teaching methods.

[0019] According to the AR-based video teaching method, device, electronic device and computer-readable medium of the present invention, in response to the user's instruction to turn on the AR interactive function, the user's real-time video is subjected to action recognition processing to determine whether the user makes a predefined characteristic action; if the user makes the predefined characteristic action, the target virtual object corresponding to the predefined characteristic action is obtained; the target virtual object is drawn in the user's real-time video to form a teaching video. The AR-based video teaching method of the present invention can provide users with a convenient and fast operation method to call out virtual objects, assist teaching work through virtual objects, increase interactive fun, and improve teaching effects.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail its exemplary embodiments with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can also be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 The present invention is a system block diagram of a video teaching system according to an exemplary embodiment.

[0023] Figure 2 The figure is a flowchart of a video teaching method based on AR according to an exemplary embodiment.

[0024] Figure 3 is a flowchart of an AR-based video teaching method according to another exemplary embodiment.

[0025] Figure 4 is a flowchart of an AR-based video teaching method according to another exemplary embodiment.

[0026] Figure 5 It is a schematic diagram showing the effect of an AR-based video teaching method according to another exemplary embodiment.

[0027] Figure 6 It is a block diagram of an AR-based video teaching device according to an exemplary embodiment.

[0028] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment.

[0029] Figure 8 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0031] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. Without departing from the main purpose of the present invention, the execution order shown in the flowchart can be changed according to the actual situation and is not limited to the example order of the flowchart.

[0032] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0033] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0034] The present invention proposes an AR-based video teaching method and device, which can increase the interactivity between students and teachers in learning, promote students to master learning knowledge, and assist teachers in presenting teaching content. The content of the present invention is described in detail below with the help of specific embodiments.

[0035] AR (Augmented Reality) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0036] Figure 1 1 is a block diagram of a video teaching system according to an exemplary embodiment. The system can be used to implement the AR-based video teaching method provided in an embodiment of the present invention.

[0037] like Figure 1As shown, the system architecture 10 may include student terminals 101, 102, 103, a network 104, and a teacher terminal 105. The network 104 is used to provide a medium for communication links between the student terminals 101, 102, 103 and the teacher terminal 105. The network 104 may include various communication connection types, such as wired or wireless communication links, and the network 104 may include optical fibers, cables, and the like.

[0038] Students can use student terminals 101, 102, 103 to interact with teacher terminal 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on student terminals 101, 102, 103 and teacher terminal 105, such as video learning applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The student terminals 101 , 102 , 103 and the teacher terminal 105 may be various electronic devices having display screens and supporting video teaching, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0040] The teacher terminal 105 can obtain video data related to the teacher's teaching, process the video data, and then display the video to the student terminals 101, 102, and 103.

[0041] The teacher terminal 105 may, for example, respond to an instruction from the user to turn on the AR interaction function by performing action recognition processing on the user's real-time video to determine whether the user makes a predefined feature action; if the user makes the predefined feature action, the teacher terminal 105 may, for example, obtain a target virtual object corresponding to the predefined feature action; the teacher terminal 105 may, for example, draw the target virtual object in the user's real-time video to form a teaching video.

[0042] In other application scenarios, the system architecture 10 may include a teacher terminal 105, student terminals 101, 102, 103 and a remote server used by teachers. The remote server may be a physical server, or may be composed of multiple physical servers, for example. The teacher user conducts video teaching through the teacher terminal 105, and the teacher terminal 105 may obtain real-time teacher teaching videos, and then the teacher terminal 105 transmits the video data to the remote server, and the remote server recognizes the gestures, body movements, etc. of the teacher user in real time, and generates virtual objects and displays them in the teacher teaching video when conditions are met. The virtual objects may be cartoon images, virtual teaching aids or experimental tools, or virtual characters, animal images, etc. The remote server may combine the virtual objects with the teacher user's video and generate a final teaching video. The remote server may also send the teaching video to the student terminals 101, 102, 103 and the teacher terminal 105, and the student users may watch the virtual objects to better understand the content of the lecture, and the teacher users may watch the synthesis effect of the video. Teacher users can use virtual objects to better design the teaching process, express course content more intuitively and attract students' attention.

[0043] Furthermore, an action recognition model may be provided in the teacher terminal 105 or the remote server. The action recognition model may be generated by: determining a plurality of preset actions; generating / obtaining a plurality of action image sample sets based on the plurality of preset actions; and then training a deep learning model with the plurality of action image sample sets to generate the above-mentioned action recognition model.

[0044] Furthermore, in the teacher terminal 105 or the remote server, multiple virtual objects can be generated through three-dimensional model technology; and the multiple virtual objects are associated with multiple predefined feature actions. When it is recognized that the user has made a predefined feature action, the background of the teacher terminal 105 or the remote server calls the virtual object corresponding to the predefined feature action from multiple virtual objects as the target virtual object, and displays the target virtual object in the user video. It should be noted that the AR-based video teaching method provided in the embodiment of the present invention can be executed in whole or in part by the teacher terminal 105; it can also be executed in whole or in part by the remote server; it can also be executed in part by the teacher terminal 105 and in part by the remote server. The AR-based video teaching method provided in the embodiment of the present invention can also be partially executed by the student terminal. The embodiment of the present invention is not limited to this.

[0045] Accordingly, the AR video teaching device that implements the AR-based video teaching method of the present invention can be partially or completely set in the teacher terminal 105, can also be partially or completely set in the remote server, or can also be partially set in the student terminal.

[0046] The teacher terminal 105 may also be a server.

[0047] Based on the above-mentioned system or terminal, the AR-based video teaching method of the present invention is described below. In the following, the technical content of the present invention is described in detail mainly through the application scenario of online live classes. It can be understood that the technical content of the present invention can also be applied in other scenarios, such as live broadcast links, shopping guide links, other human-computer interaction links, etc., and the present invention does not limit it.

[0048] In addition, the present invention can also be used as a teaching video synthesis method to add AR effects to general teaching videos provided by users based on the teacher's actions in the video.

[0049] It should be noted that the user's "real-time video" hereinbelow is only used to indicate that the video can obtain the actions of the characters in the video through action recognition, and is not used to limit the video to be a real-time recorded video.

[0050] Figure 2 FIG. 2 is a flowchart of a video teaching method based on AR according to an exemplary embodiment. The video teaching method 20 at least includes steps S202, S204 and S206. Figure 2 As shown, in S202, in response to the user's instruction to turn on the AR interaction function, action recognition processing is performed on the user's real-time video to determine whether the user performs a predefined characteristic action.

[0051] Exemplarily, in S202, the real-time video of the user is processed to determine whether the user performs a predefined characteristic action. In some embodiments, step S202 may include: in response to a user's instruction to turn on the AR interaction function, obtaining the real-time video of the user; inputting the real-time video into an action recognition model for action recognition, and determining whether the recognized user action belongs to an action in the predefined characteristic action set, and the action recognition model may be implemented by a deep learning model.

[0052] The user's instruction to turn on the AR interactive function can be, for example, obtained based on the user's voice or mouse or keyboard input operations; it can also be, for example, real-time acquisition of user actions and action recognition, and turn on when the user's actions meet preset conditions. In one embodiment, the teacher user can snap his fingers during a video lecture, and the background recognizes the sound of the snap to automatically turn on the AR interactive function. In another embodiment, in order not to affect the continuity of the teacher's lecture, the teacher terminal (or remote server) can recognize the teacher user's actions in real time, and automatically turn on the AR interactive function when the teacher shows certain specific actions.

[0053] In S204, if the user performs the predefined characteristic action, a target virtual object corresponding to the predefined characteristic action is obtained. When the AR interaction function is turned on, a real-time video of the user is obtained for action recognition.

[0054] Exemplarily, the real-time video can be input into a motion recognition model to generate a motion recognition label. The motion recognition model can be implemented by a deep learning model. More specifically, for example, multiple continuous video images can be extracted from the real-time video; the multiple video images are input into the motion recognition model; the motion recognition model performs image recognition based on the multiple video images to output motion recognition labels, each of which can correspond to a predefined feature action. After the action recognition label is output, it can be determined which predefined feature action the user has made through the association between the pre-stored action recognition label and the predefined feature action. After the real-time video is input into the motion recognition model, if the model does not output the motion recognition label after calculation, it can be considered that the user has not made the predefined feature action.

[0055] In some embodiments, in order to more accurately identify the user's actions or gestures and balance the amount of calculation, multiple video images in the video can be extracted for image recognition. For example, 10 consecutive video images can be extracted at intervals of 0.05 seconds and input into the action recognition model for recognition. More specifically, the video images can be input into the action recognition model in sequence according to the order of the timestamps of each video image for recognition.

[0056] In some embodiments, before the AR interactive function is turned on, the video teaching method further includes: performing action recognition processing on the user's real-time video to determine whether the user has made a first characteristic action corresponding to turning on the AR interactive function; after the AR interactive function is turned on, the video teaching method further includes: performing action recognition processing on the user's real-time video to determine whether the user has made a second characteristic action corresponding to turning off the AR interactive function. The first characteristic action and the second characteristic action are generally different from the above-mentioned predefined characteristic actions for calling virtual objects.

[0057] In one embodiment, before the user opens the AR interactive function instruction, the real-time video of the user is extracted in a first cycle for action recognition to determine whether the user performs the first characteristic action corresponding to the instruction to open the AR interactive function; after the user opens the AR interactive function instruction, the real-time video of the user is extracted in a second cycle for action recognition, wherein the first cycle is greater than the second cycle. At this time, the duration of the first characteristic action can be defined as longer, at least greater than the first cycle.

[0058] More specifically, when the teacher user is teaching through video, a lower sampling frequency can be used to collect real-time video images for action recognition to determine whether the user issues an instruction to turn on the AR interactive function by making the first characteristic action. For example, a video image can be extracted from the video image at intervals of 0.1 seconds and input into the action recognition model for recognition. After turning on the AR interactive function, in order to more accurately recognize the actions of the teacher user, a higher sampling frequency can be used to extract the user's real-time video graphics. For example, a real-time video image can be extracted from the video image every 0.05 seconds and input into the action recognition model for recognition. Among them, the first characteristic action corresponding to turning on the AR interactive function is required to last at least 0.1.

[0059] In addition, after the AR interactive function is turned on, the AR interactive function can be exited based on the closing instruction. After exiting the AR interactive function, a video image can be continuously extracted from the video image at intervals of 0.1 seconds and input into the action recognition model for recognition to determine whether the user issues an instruction to turn on the AR interactive function by performing the first characteristic action.

[0060] In one embodiment, before the AR interactive function is turned on, the video teaching method further includes: performing action recognition processing on the user's real-time video to determine whether the user has made a first characteristic action corresponding to turning on the AR interactive function; the predefined first characteristic action for turning on the AR interactive function may be making fists with both hands and pressing them together. In the recognition of the real-time video, when the teacher user is identified to have made the above action (making fists with both hands and pressing them together), the background generates and executes an AR start instruction, the AR interactive function is turned on, and step S202 is started to monitor whether the user has made a predefined characteristic action of calling a virtual object. Before turning on the AR interactive function, the background only monitors whether the user has made the first characteristic action corresponding to turning on the AR interactive function.

[0061] In one embodiment, after turning on the AR interactive function, the video teaching method further includes: performing action recognition processing on the user's real-time video to determine whether the user makes a second feature action corresponding to turning off the AR interactive function. The second feature action for turning off the AR interactive function can be predefined as opening the five fingers of both hands and putting the fingers of the left and right hands together. The second feature action for turning off the AR interactive function can also be predefined as the same action as turning on the AR interactive function, such as making fists with both hands and pressing them together. When it is recognized that the teacher user makes the above action, the background generates an AR closing instruction and executes it, and exits the AR interactive function, that is, it no longer monitors whether the user makes the predefined feature action of calling the virtual object. It is worth mentioning that the action recognition model can be set in a remote server or on the teacher user terminal.

[0062] In S206, the target virtual object is drawn in the real-time video of the user to form a teaching video. For example, multiple virtual objects can be generated in advance through three-dimensional model technology; the multiple virtual objects are associated with multiple action recognition tags. Based on the above association relationship, when executing step S204, the corresponding target virtual object can be determined according to the action recognition tag output by the action recognition model. When executing step S206, the corresponding target virtual object is displayed in the real-time video of the user to form a teaching video.

[0063] Multiple virtual objects can be pre-designed, and virtual objects and user actions (action recognition tags) can be matched one by one. Virtual objects can also be set with different parameters and attributes as needed. After the AR interaction function is turned on, the user's real-time video is processed for action recognition to determine whether the user has made a predefined characteristic action; if the user has made a predefined characteristic action, the target virtual object corresponding to the predefined characteristic action is obtained from multiple virtual objects; the target virtual object is drawn in the user's real-time video to form a teaching video.

[0064] In some embodiments, when the virtual object is a cartoon character, the label of the cartoon character can be determined according to the user's expression, and the position and action of the cartoon character can also be determined according to the user's gestures.

[0065] In some embodiments, the above-mentioned virtual objects can also support the customization function of teacher users, and teacher users can set virtual objects and their appearance locations. Specifically, for example, users can set or select the properties of virtual objects themselves. For example, when the virtual object is a cartoon object, the teacher user can customize the color, clothing, accessories, etc. of the cartoon image. Among them, calling the predefined feature actions of virtual objects can also support customization by teacher users. Teacher users can customize the feature actions of calling virtual objects, and can also customize the first and second feature actions of turning off / on the AR interactive function. In this way, it is easier for teachers to design classes.

[0066] In some embodiments, the unconscious habitual small movements made by the teacher user during class can be collected in advance, and when the user customizes the characteristic movements, these small movements are automatically avoided or suggested to the user to avoid, so as to prevent the teacher from unconsciously calling virtual objects and disrupting the class design.

[0067] In other embodiments, before step S202 (for example, when the device based on the present invention is used for the first time), the AR-based video teaching method provided in this embodiment may also include: collecting unconscious habitual small movements made by the teacher user during class and performing movement recognition, automatically adjusting or proposing adjustment suggestions for calling predefined feature movements of virtual objects, to facilitate teachers in classroom design.

[0068] In one embodiment, step S206 includes: obtaining the display position of the target virtual object; and drawing the virtual object in the real-time video based on the display position. Exemplarily, when the display position of the target virtual object is related to the action region where the user performs the predefined feature action, obtaining the action region where the user performs the predefined feature action; and determining the display position of the target virtual object based on the action region. Exemplarily, when the display position of the target virtual object is related to the position of a reference object, obtaining the position information of the reference object; and determining the display position of the target virtual object based on the position information of the reference object.

[0069] Depending on the target virtual object, its display position may be different. Some target virtual objects, such as a virtual blackboard, a virtual blackboard eraser, a virtual chalk, and a virtual desk, may be set in a specific area of ​​the video page. For example, the display position of the virtual blackboard can be preset to be located behind the user, roughly in the center of the video. The display position of some target virtual objects may be set in or near the action area. For example, when a teacher makes a circle-drawing action, a circle or a spherical object appears at the circle-drawing position; for another example, when a teacher puts his hands on his hips, an Ultraman cartoon image appears next to the teacher. The display position of some target virtual objects may be set near a reference object. For example, the display position of a virtual blackboard eraser may be: at the lower edge of the blackboard, or at a certain position on the virtual desk. For another example, when a teacher taps the virtual desk, a virtual pencil case containing multiple virtual writing tools appears in a corner of the virtual desk.

[0070] The display position of the virtual object can be set together when setting the virtual object or when setting the association relationship between the virtual object and the predefined feature action. In some embodiments, the display position of the virtual object can be set as an attribute of the virtual object (such as the display attribute of the virtual object) and can be predefined or customized.

[0071] When the display position of the target virtual object is pre-set to a specific area in the video page, to obtain the display position of the target virtual object, it is generally sufficient to obtain the position of the relevant reference point. For example, if the display position of the virtual blackboard is set to be behind the user (teacher), in the center of the video. The information required to determine the display position of the virtual blackboard generally requires: the target virtual object is the virtual blackboard, the position of the user (teacher), and information related to the display area of ​​the entire teaching video. When determining the information required for the display position of the virtual blackboard, when it is determined that the target virtual object is the virtual blackboard, and the display position is determined from its display properties to be "located behind the user, roughly in the center of the video", the user's position and information related to the display area of ​​the entire teaching video (used to determine the center of the video) are obtained.

[0072] If the display position of the target virtual object is the action occurrence area or its vicinity, it is necessary to obtain the action occurrence area where the user performs the predefined characteristic action. In one embodiment, when the virtual object needs to be displayed in the action occurrence area, the action occurrence area can be output simultaneously during action recognition to determine the display position of the target virtual object. For example, the predefined characteristic action is that the teacher user taps the virtual desktop, and a virtual blackboard eraser appears at the tapping position of the virtual desktop. In this case, the tapping position of the virtual desktop is the action occurrence area, and the blackboard eraser can be displayed in the area around the tapping position.

[0073] If the display position of the target virtual object is related to the position of a reference object, when determining the display position of the target virtual object, the position information of the reference object is first obtained, and then the display position of the target virtual object is determined according to the position information of the reference object. The reference object can be a user, an object in the video, another virtual object that has appeared in the teaching video, or the middle point of the video screen, etc.

[0074] According to the AR-based video teaching method of the present invention, in response to the user's instruction to turn on the AR interactive function, action recognition processing is performed on the user's real-time video to determine whether the user makes a predefined feature action; if the user makes the predefined feature action, a target virtual object corresponding to the predefined feature action is obtained; the target virtual object is drawn in the user's real-time video to form a teaching video, which can provide users with a convenient and fast operation method to call out virtual objects, assist teaching work through virtual objects, increase interactive fun, and improve teaching effects.

[0075] It should be clearly understood that the present invention describes how to form and use specific examples, but the principles of the present invention are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present invention, these principles can be applied to many other embodiments.

[0076] Figure 3 is a partial flow chart of an AR-based video teaching method according to another exemplary embodiment. Figure 3 The process 30 shown is a detailed description of “establishing the action recognition model”.

[0077] like Figure 3 As shown, in S302, multiple preset actions are determined. Multiple actions that are easy to implement can be predetermined, such as standing with hands on hips, making gestures with hands, stretching five fingers, etc. The multiple preset actions at least include all predefined feature actions and a first feature action and a second feature action for turning on and off the AR interaction function.

[0078] In S304, multiple action image sample sets are generated based on the multiple preset actions, wherein each action image sample set corresponds to an action recognition label. Different users are selected to perform the above operations to generate multiple action image samples. Action image samples of users of different heights, weights, and genders can be captured. Action image samples of different users at different distances from the collection end can also be captured. Action image samples of the user's full body or half body can also be captured, etc. The above methods increase the diversity of samples and ensure the accuracy of subsequent image recognition.

[0079] In S306, the deep learning model is trained by the plurality of action image sample sets to generate the action recognition model. Deep learning is a general term for a class of pattern analysis methods, and the action recognition model in the present invention can be generated by training by combining one or more of the following three network models:

[0080] (1) Neural network system based on convolution operation, namely convolutional neural network (CNN);

[0081] (2) Autoencoder neural network based on multi-layer neurons;

[0082] (3) Pre-training is performed in the form of a multi-layer autoencoder neural network, and then the deep belief network (DBN) is used to further optimize the neural network weights by combining the identification information.

[0083] Since all video data will be input into the action recognition model for recognition, in some embodiments, the accuracy and performance of recognition can be taken into account, and the gestures or actions of the teacher user to turn on and off the AR interactive function can be continuously recognized during the entire video teaching process. In order to save model calculation time and computing resources, other actions (for example, other actions except turning on the AR interactive function) will not be recognized when the AR interactive function is not turned on. When the AR interactive function is not turned on, in some cases, even if the action recognition model recognizes other actions, it is irrelevant to the implementation scheme of the present invention, and the recognition result can be directly discarded or processed according to other schemes. The action label corresponding to the action is only output after the AR interactive function is turned on to determine the target virtual object and output it, that is, the virtual object is output after the AR interactive function is turned on.

[0084] In one embodiment, when the target virtual object is drawn in the real-time video for display, the user's operation instructions for the target virtual object are obtained; in the teaching video formed based on the user's real-time video, the operation process and / or the operation effect of the above virtual object based on the operation instructions are displayed. The operation instructions can be voice operation instructions, or mouse and keyboard operation instructions, which are not limited in this embodiment. Multiple virtual objects can be displayed simultaneously in the teaching video. When the user clicks on or operates a virtual object, the virtual object is determined to be the target virtual object herein or as described below.

[0085] Among them, obtaining the user's operation instructions includes: when a target virtual object is displayed in the real-time video, obtaining the user's operation action on the target virtual object by performing action recognition processing on the user's real-time video; based on the target virtual object and the operation action, obtaining the operation instruction for the target virtual object corresponding to the operation action. Figure 4 is a partial flow chart of an AR-based video teaching method according to another exemplary embodiment. Figure 4 The process 40 shown is for Figure 2 Supplementary description of the AR interactive function of the process shown.

[0086] like Figure 4 As shown, the AR-based video teaching method of this embodiment has Figure 2 In addition to the process shown, steps S402, S404 and S406 are also included. In step S402, when the target virtual object is drawn in the real-time video for display, the user's operation action on the target virtual object is obtained by performing action recognition processing on the user's real-time video.

[0087] In step S404, an operation instruction for the target virtual object corresponding to the operation action is acquired according to the target virtual object and the operation action.

[0088] The target virtual object may be a teaching auxiliary object; the operation action may include: scaling, moving, changing display angle, releasing, and removing. Depending on the actual situation and needs, the operation action may also include other actions.

[0089] In step S406, in the teaching video formed based on the real-time video of the user, the operation process and / or the operation effect of the target virtual object based on the operation instruction are displayed.

[0090] Exemplarily, the target virtual object may be a virtual writing tool, and as the virtual writing tool moves, the moving track of the virtual writing tool is displayed in the teaching video as a writing trace. In one embodiment, the target virtual object may be a virtual chalk, and the virtual chalk may be moved by the user's operation instructions, and the moving track of the virtual chalk is displayed as writing handwriting or writing traces on the teaching video image.

[0091] Exemplarily, the target virtual object may be a virtual erasing tool, and as the virtual erasing tool moves, the writing traces on the moving path of the virtual erasing tool are cleared in the teaching video. For example, the target virtual object may be a virtual blackboard eraser; the virtual blackboard eraser operation may include: scaling, moving position, releasing, removing, etc. For example, the virtual blackboard eraser may be moved by the user's operation instruction. As the virtual blackboard eraser moves, the image on the moving track of the blackboard eraser is cleared. For example, according to the user's zoom in and out instruction (which can be called by a certain feature action), the size of the blackboard eraser can be adjusted to match the user's movement operation.

[0092] Exemplarily, the target virtual object may be a virtual experimental tool, and according to the user's operation on the virtual experimental tool, the predefined operation process and / or operation effect of the virtual experimental tool is displayed in the teaching video. For example, the virtual experimental tool is a virtual spring, and the virtual spring can be operated in addition to scaling, moving, changing the display angle, releasing, and removing, and can also be compressed or stretched. If the user performs a stretching operation on the virtual spring, the process of stretching the virtual experimental spring and the linkage operation effect produced by the stretching are displayed in the teaching video.

[0093] like Figure 5 As shown, in one embodiment, it also includes: when a virtual object is displayed in the real-time video, obtaining the user's operation instructions for the virtual object; and displaying the operation process and / or effect of the virtual object based on the operation instructions in the teaching video. In one embodiment, the virtual object is a teaching auxiliary item, and a virtual celestial image can be generated, for example, when the teacher user drags the object with both hands, and the celestial image can be set to be displayed above the video image by default. If the teacher user feels that the display position is on the left and it is inconvenient to explain, and wants to move the virtual celestial body to the right, the teacher user can make an action of dragging the virtual celestial body to the right, and the background generates a virtual object movement instruction through action recognition. Then, in the teaching video image, the virtual celestial image is moved to the right and displayed on the right. In addition, in some embodiments, a rotation instruction can also be generated according to the teacher's gestures or mouse and keyboard operations, and then the virtual celestial body is rotated for display at multiple angles.

[0094] In one embodiment, it also includes: when the virtual object is displayed in the teaching video, in addition to identifying the predefined characteristic action, the user's action trajectory is also obtained; the display of the virtual object is related to the user's action trajectory. For example, it is displayed following the user's action trajectory. In one embodiment, the virtual object can be a virtual sun image. When the teacher user gestures to draw a circle with both hands, an image of a virtual sun will appear, and the image of the virtual sun will be displayed directly following the trajectory of the user's hands drawing a circle, and the virtual sun will be displayed where the user's hands draw a circle. For example, when the teacher user makes a fist at the position of the virtual sun, a selection instruction of the virtual sun is generated. When the teacher user moves his fist at this time, the virtual sun will move along the trajectory of the hand. It is also possible that when the teacher user releases his fist, an instruction to release the virtual sun is generated in the background, the virtual sun stops moving, and the virtual sun stays at the current position of the video screen, etc.

[0095] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above method provided by the present invention is executed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk.

[0096] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0097] The following is an embodiment of a device for implementing the above method of the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the embodiment of the method of the present invention.

[0098] Figure 6 FIG. 1 is a block diagram of an AR-based video teaching device according to another exemplary embodiment. Figure 6 As shown, the AR video teaching device 60 includes: a recognition module 602 , a virtual object acquisition module 604 , and a display module 606 .

[0099] The recognition module 602 is used to respond to the user's instruction to turn on the AR interaction function and perform action recognition processing on the user's real-time video to determine whether the user performs a predefined characteristic action.

[0100] The virtual object acquisition module 604 is used to acquire a target virtual object corresponding to the predefined feature action when it is determined that the user performs the predefined feature action;

[0101] The display module 606 is used to draw the target virtual object in the real-time video of the user to form a teaching video. The display module 606 is also used to obtain the display position of the target virtual object; based on the display position, draw the virtual object in the real-time video.

[0102] According to the AR-based video teaching device of the present invention, in response to the user's instruction to turn on the AR interactive function, the user's real-time video is subjected to action recognition processing to determine whether the user makes a predefined feature action; if the user makes the predefined feature action, the target virtual object corresponding to the predefined feature action is obtained; the target virtual object is drawn in the user's real-time video to form a teaching video, which can provide users with a convenient and fast operation method to call out virtual objects, assist teaching work through virtual objects, increase interactive fun, and improve teaching effects. The present invention also provides another AR-based video teaching system, which includes student terminals 101, 102, 103, a network 104 and a teacher terminal 105. A remote server is provided in the network 104. In response to the instruction to turn on the AR interactive function, the student terminals 101, 102, 103 upload the instruction to turn on the AR interactive function to the remote server. The above-mentioned recognition module 602, virtual object acquisition module 604 and display module 606 are provided in the remote server. In response to the instruction, the remote server performs predefined feature action recognition and target virtual object call, generates a teaching video including the target virtual object, and sends it to the student terminals 101, 102, 103. The student terminals 101, 102, 103 display the teaching video including the virtual object.

[0103] In another embodiment, the difference from the previous embodiment is that the teacher terminal 105 is provided with the above-mentioned recognition module 602, virtual object acquisition module 604 and display module 606, the teacher terminal 105 sends the teaching video to the remote server, and the student terminals 101, 102, 103 obtain the teaching video containing the virtual object from the remote server.

[0104] In another embodiment, the difference from the previous embodiment is that the teacher terminal 105 is provided with the above-mentioned recognition module 602, the remote server is provided with a virtual object acquisition module 604 and a display module 606, the teacher terminal 105 sends the real-time video and the predefined feature actions corresponding to the real-time video time to the remote server, and the remote server generates a teaching video.

[0105] As mentioned above, there are many possibilities for the module settings of the identification module 602, the virtual object acquisition module 604 and the display module 606. For example, a certain functional module or part of its functions can be set in the teacher terminal, and other functional modules or parts of the functions can be set in the student terminal. The present invention does not limit this and all should be included in the protection scope of the present invention. Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment.

[0106] like Figure 7 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0107] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0108] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0109] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0110] It should be understood that Figure 7 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0111] The present invention also provides a computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the computer program / instruction implements any of the above-mentioned video teaching methods.

[0112] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 8 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0113] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0114] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0115] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0116] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium implements the following functions: in response to the user's instruction to turn on the AR interaction function, action recognition processing is performed on the user's real-time video to determine whether the user makes a predefined feature action; if the user makes the predefined feature action, the target virtual object corresponding to the predefined feature action is obtained; the target virtual object is drawn in the user's real-time video to form a teaching video.

[0117] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0118] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0119] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A video teaching method based on AR, characterized in that: include: Before the user issues an instruction to start the AR interactive function, extract the real-time video of the user in a first period to perform action recognition, determine whether the user performs a first characteristic action corresponding to the instruction to start the AR interactive function, and start the AR interactive function when the user's action meets the preset condition; After the user opens the AR interactive function, extracting the real-time video of the user for action recognition in a second cycle, wherein the first cycle is greater than the second cycle; After the user opens the AR interactive function, the user's real-time video is processed for action recognition to determine whether the user performs a predefined characteristic action; If the user performs the predefined feature action, obtaining a target virtual object corresponding to the predefined feature action; Get the display position of the target virtual object; When the display position of the target virtual object is related to the action region where the user performs the predefined feature action, the action region where the user performs the predefined feature action is obtained; and the display position of the target virtual object is determined according to the action region; or, when the display position of the target virtual object is related to the position of a reference object, the position information of the reference object is obtained; and the display position of the target virtual object is determined according to the position information of the reference object; Drawing the target virtual object in the user's real-time video to form a teaching video; When the target virtual object is displayed in the real-time video, obtaining the user's operation instruction for the target virtual object; and displaying the operation process and / or effect of the target virtual object based on the operation instruction in the teaching video; Wherein, obtaining the user's operation instruction for the target virtual object comprises: when the target virtual object is drawn in the real-time video for display, obtaining the user's operation action for the target virtual object by performing action recognition processing on the user's real-time video; based on the target virtual object and the operation action, obtaining the operation instruction for the target virtual object corresponding to the operation action; When the target virtual object is displayed in the teaching video, the motion track of the user is also obtained; the display of the target virtual object is related to the motion track of the user; The target virtual object is a virtual writing tool, and as the virtual writing tool moves, the moving trajectory of the virtual writing tool is displayed in the teaching video as a writing trace; or, the target virtual object is a virtual erasing tool, and as the virtual erasing tool moves, the writing traces on the moving path of the virtual erasing tool are cleared in the teaching video; or, the target virtual object is a virtual experimental tool, and according to the user's operating actions on the virtual experimental tool, the predefined operation process and / or operation effect of the virtual experimental tool is displayed in the teaching video; after turning on the AR interactive function, the video teaching method also includes: performing action recognition processing on the user's real-time video to determine whether the user makes a second feature action corresponding to turning off the AR interactive function.

2. The method according to claim 1, characterized in that Recognize and process the user's real-time video to determine whether the user has made predefined characteristic actions, including: The real-time video is input into an action recognition model for action recognition, and it is determined whether the recognized user action belongs to an action in a predefined feature action set. The action recognition model is implemented by a deep learning model.

3. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-2.

Images