Airwriting method, device, online live broadcast system and computer equipment

Abstract

The present invention belongs to the field of education, and provides an air writing method, an online live broadcast system, a device, and a computer device, the method comprising: monitoring the distance between the user and the camera device; when it is detected that the distance between the user and the camera device is within a preset distance range, detecting the user's real-time image to monitor the user's hand shape; when it is detected that the user's hand shape is a writing hand shape, starting to detect the writing hand shape to identify the writing key points, and recording the positions of the writing key points; generating a writing trajectory according to the positions of the writing key points at multiple time points. The method of the present invention can improve the speed and accuracy of writing recognition, and can improve the user experience, and can further optimize the air writing method in the online live class interaction.

Description

Technical Field

[0001] The present invention belongs to the field of education, and is particularly suitable for the field of online education. More specifically, it relates to an air writing method, device, online live broadcast system and computer equipment. Background Art

[0002] In recent years, the Internet has become increasingly popular, and online course live broadcast has become one of the latest development trends and research hotspots in modern distance education research and application. Combining online education with real-time and interactive learning will bring online education into a new stage. The current online education model is mostly online video courses and online video interactive courses.

[0003] In fact, online video interactive courses involve various interaction methods such as voice interaction, somatosensory interaction, eye movement interaction and gesture interaction. For gesture interaction, gesture recognition is required in the process of completing gesture interaction. Existing methods include image-based recognition and gesture sensor recognition. However, there is still much room for improvement in recognition accuracy, data processing speed, complexity of the calculation process and user experience.

[0004] Therefore, it is necessary to provide a kind of sky writing method, to solve the above problems. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The present invention aims to solve technical problems such as low recognition accuracy of gesture interaction, slow data processing speed, and poor user experience in online video interactive courses.

[0007] (II) Technical solution

[0008] To solve the above technical problems, one aspect of the present invention proposes an aerial writing method for online live classes, the method comprising the following steps: monitoring the distance between the user and the camera device; when it is detected that the distance between the user and the camera device is within a preset distance range, detecting the real-time image of the user to monitor the user's hand shape; when it is detected that the user's hand shape is a writing hand shape, starting to detect the writing hand shape to identify the writing key points, and recording the positions of the writing key points; generating a writing trajectory according to the positions of the writing key points at multiple time points.

[0009] Optionally, monitoring the distance between the user and the camera device includes: performing real-time detection on the real-time image of the user to monitor the distance between the user and the camera device. Further optionally, performing real-time detection on the real-time image of the user to monitor the distance between the user and the camera device includes: detecting the area ratio of the user's body shape in the real-time image, and calculating the distance between the user and the camera device according to the area ratio.

[0010] According to a preferred embodiment of the present invention, the detection of the user's real-time image to monitor the user's hand shape includes: using a first image recognition model to detect whether the user's real-time image contains a predefined writing hand shape, the image recognition model is a machine learning model that has been trained to recognize the writing hand shape of a human hand, and when it is detected that the real-time image contains a predefined writing hand shape, the detection process of the writing key points is started.

[0011] According to a preferred embodiment of the present invention, the writing key point detection process includes: using a second image recognition model to detect the writing key points, and the image recognition model is a machine learning model that has been trained to recognize the writing key points in the writing hand shape.

[0012] According to a preferred embodiment of the present invention, the predefined writing hand shape is a posture in which the index finger is extended and the other fingers are clenched, and the writing key point of the human hand shape is the tip of the index finger.

[0013] According to a preferred embodiment of the present invention, generating a writing trajectory based on the positions of the writing key points at multiple time points includes: performing straight line or curve fitting on the positions of the writing key points at the multiple time points to generate the motion trajectory; generating a writing trajectory based on the motion trajectory.

[0014] According to an optional embodiment of the present invention, when a user writes on a writing end, a writing trajectory is generated, and the writing trajectory is transmitted in real time to an opposite end interacting with the writing end.

[0015] According to an optional embodiment of the present invention, it also includes: coupling the writing track generated at the writing end to the video stream, and displaying or hiding the writing track in the user video stream according to the user instruction of the writing end or the opposite end.

[0016] According to an optional embodiment of the present invention, the method further comprises: displaying the writing track at the writing end and / or the opposite end.

[0017] A second aspect of the present invention proposes an online live broadcast system, which includes: a writing end and a relative end; at least one server, which is used for interaction between the writing end and the relative end during an online live class, and the writing end or the at least one server uses any of the above-described air writing methods to generate a writing trajectory; the at least one server transmits the generated writing trajectory to the relative end in the form of a video stream, and displays it in the writing end and / or the relative end.

[0018] The third aspect of the present invention proposes an aerial writing device for online live classes, and the aerial writing device includes: a first monitoring module, used to monitor the distance between the user and the camera device; a second monitoring module, when it is detected that the distance between the user and the camera device is within a preset distance range, detects the real-time image of the user to monitor the user's hand shape; a third monitoring module, when it is detected that the hand shape of the first user is a writing hand shape, starts to detect the writing hand shape to identify writing key points; a generation module, generates a writing trajectory according to the positions of the writing key points at multiple time points.

[0019] A fourth aspect of the present invention provides a computer device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the air writing method for online live classes described in the present invention.

[0020] A fifth aspect of the present invention proposes a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, it implements the air writing method for online live classes described in the present invention.

[0021] (III) Beneficial effects

[0022] Compared with the prior art, the present invention starts the user's hand shape recognition when it is monitored that the distance between the user and the camera device is within a preset distance range; when it is monitored that the user's hand shape is a writing hand shape, the process of identifying the writing key points is started, and the writing trajectory is generated according to the position trajectory of the writing key points. This can improve the speed and accuracy of writing recognition, improve the user experience, and further optimize the air writing method in the online live class interaction.

[0023] Furthermore, by using the image monitoring model, it is possible to more accurately detect the area ratio of the first user's body shape in the real-time image, and to obtain more accurate location information of the first user; through the first image recognition model, it is possible to more accurately detect whether the first user's hand shape is a writing hand shape; through the second image recognition model, it is possible to more accurately determine the position of the key points of writing, thereby also being able to more quickly and accurately determine the writing trajectory of the air writing; it is possible to improve the speed and accuracy of writing recognition, realize a variety of display methods of the writing trajectory in the interaction, improve the user experience, and further optimize the air writing method in the interaction of online live classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of an example of the air writing method according to the first embodiment of the present invention;

[0025] Figure 2 is a flowchart of another example of the air writing method according to Embodiment 1 of the present invention;

[0026] Figure 3 is a schematic diagram of an example of a writing hand shape in the air writing method according to embodiment 1 of the present invention;

[0027] Figure 4 is a schematic diagram of another example of a writing hand shape in the air writing method according to the first embodiment of the present invention;

[0028] Figure 5 is a flowchart of another example of the air writing method according to Embodiment 1 of the present invention;

[0029] Figure 6 It is a schematic diagram of an example of an online live broadcast system of embodiment 2 of the present invention.

[0030] Figure 7 is a schematic diagram of an example of an aerial writing device according to Embodiment 3 of the present invention;

[0031] Figure 8 is a schematic diagram of another example of the aerial writing device according to the third embodiment of the present invention;

[0032] Fig. 9 is a schematic diagram of another example of the aerial writing device according to the third embodiment of the present invention;

[0033] Fig.10 is a schematic diagram of another example of the aerial writing device according to the third embodiment of the present invention;

[0034] Fig.11 is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0035] Fig.12It is a schematic diagram of a computer program product according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0037] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0038] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0039] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0040] In order to further optimize the air writing method, the present invention proposes an air writing method for online live classes, which detects the hand shape of the first user through multiple detection steps. When the handwriting hand shape is detected, the writing key points of the writing hand shape are detected and identified, and the writing key points and their corresponding writing trajectories are recorded to display the writing trajectories in the writing end and / or the opposite end. The opposite end can interact or communicate with the writing end. Therefore, the method of the present invention can improve the speed and accuracy of writing recognition, improve the user experience, and further optimize the air writing method in the interaction of online live classes.

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0042] It should be noted that, for ease of understanding, the embodiments of the present invention are described in this article using the scenario of an online live class, but those skilled in the art will appreciate that the application of the present invention is not limited thereto. The present invention can also be used in other live broadcast scenarios, or can also be used as a video processing method to process existing videos, identify the writing actions of characters in the video and generate writing trajectories based on them, which are coupled to existing videos or transmitted / displayed as independent existing videos.

[0043] Figure 1 This is a flowchart of an example of the aerial writing method according to the first embodiment of the present invention.

[0044] like Figure 1 As shown, the present invention provides a method for writing in the air, the method comprising:

[0045] Step S101, monitoring the distance between the user and the camera device.

[0046] Step S102: When it is detected that the distance between the user and the camera device is within a preset distance range, the real-time image of the user is detected to monitor the hand shape of the user.

[0047] Step S103, when it is detected that the user's hand shape is a writing hand shape, the writing hand shape is detected to identify the writing key points, and the positions of the writing key points are recorded.

[0048] Step S104, generating a writing trajectory according to the positions of the writing key points at multiple time points.

[0049] In this example, in the application scenario of online live classes, there are a writing end and a corresponding end, and an online live broadcast system running on the two or one of them. The air writing method is used for user interaction between the writing end and the corresponding end during the online live class. The writing end and the corresponding end can both be connected to a camera device, and each camera device continuously captures real-time images to form a respective video stream. The video stream formed at the writing end can be transmitted to the corresponding end through the server of the online live broadcast system. The specific process of the air writing method will be described in detail below.

[0050] First, in step S101 , the distance between the user and the camera device is monitored.

[0051] In this example, the writing end is a first client corresponding to a first user who may be a teacher, and the opposite end is a second client interacting with the writing end corresponding to a second user who may be a student.

[0052] Specifically, during an online live class, a camera device of the first client (eg, a camera device of a computer, a mobile phone, etc.) monitors the real-time image of the teacher in real time.

[0053] Optionally, the real-time image of the user may be detected in real time to monitor the distance between the user and the camera device. For example, the area ratio of the user's body shape in the real-time image may be detected, and the distance between the user and the camera device may be calculated based on the area ratio.

[0054] Furthermore, an image monitoring model may be used to monitor the distance between the first user and the camera device of the first client.

[0055] It should be noted that the above is only described as an example and cannot be understood as limiting the present invention. In other examples, the monitoring of the distance between the user and the camera device can also be achieved through the hardware of the first client or the second client, such as an infrared remote rangefinder, a distance sensor, etc.

[0056] Figure 2 FIG. 1 is a flowchart showing another example of the method for writing in the air during an online live class according to Embodiment 1 of the present invention. Figure 2 As shown, this example also includes step S201 of establishing an image monitoring model.

[0057] In step S201, an image monitoring model is established, and training data is used to train the image monitoring model, wherein the training data set includes historical human body frame data, pictures containing human bodies in historical live video streams, and position information of human bodies in manually annotated pictures.

[0058] Specifically, each picture monitored by the first client is input into the image monitoring model, and the position information of the first user in the picture is output, for example, the area ratio of the body shape of the first user in each real-time image can be output.

[0059] Furthermore, it is calculated based on the area ratio whether the position of the first user is within a specific distance range to further determine whether to perform the next detection process.

[0060] Therefore, by using the image monitoring model, the area ratio of the first user's body shape in the real-time image can be detected more accurately, and more accurate location information of the first user can be obtained.

[0061] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0062] Next, in step S102, when it is monitored that the distance between the user and the camera device is within a preset distance range, the real-time image of the user is detected to monitor the hand shape of the user.

[0063] Specifically, when the calculated position of the first user is within a specific distance range, the first image recognition model is started for monitoring. The first image recognition model is used to detect the writing hand shape of the user.

[0064] Furthermore, the first image recognition model is used to monitor in real time whether each image contains a predefined hand shape, that is, to detect whether the hand shape of the first user is a predefined writing hand shape.

[0065] Preferably, according to different application scenarios, a writing hand shape for writing, drawing or marking is predefined and used for detection and comparison to determine whether the detected hand shape is a predefined writing hand shape.

[0066] In this example, the predefined writing hand shape may be a gesture in which the index finger is extended and the other fingers are closed, such as Figure 3 Alternatively, the predefined writing hand shape may further include: Figure 4 Common pen-holding postures shown.

[0067] Furthermore, a first image recognition model can be established based on a convolutional neural network, and the first image recognition model can be trained using a training data set, wherein the training data set includes pictures containing hands in historical live streams, and hand information in each picture, wherein the hand information includes hand positions and hand shape (hand shape) category information. The hand shapes of writing in these pictures have been annotated by the necessary users.

[0068] Specifically, the corresponding monitored pictures (pictures within a specific distance range) are input into the first image recognition model, and the hand position and hand shape category information of the first user in each picture are output.

[0069] Preferably, the specific distance ranges from 0.1 m to 1 m.

[0070] Therefore, by using the first image recognition model, it is possible to more accurately detect whether the hand shape of the first user is a predefined writing hand shape.

[0071] It should be noted that the above description is only given as an example and should not be construed as a limitation to the present invention.

[0072] Next, in step S103, when it is detected that the user's hand shape is a writing hand shape, the writing hand shape is detected to identify the writing key points, and the positions of the writing key points are recorded.

[0073] Specifically, when the first image recognition model detects that the user's hand shape is a predefined writing hand shape, the writing key point detection process is started, that is, the writing key points of the writing hand shape are detected and the writing key points are identified.

[0074] Figure 3 Schematic diagram of an example of hand shape in the air writing method of online live class in embodiment 1 of the present invention. Figure 3 The writing key point of the writing hand shape is the tip of the index finger (or the position of the tip of the index finger).

[0075] It should be noted that the above is only described as a preferred example and cannot be understood as limiting the present invention. In other examples, for example, when a user uses a special pen to write, the writing hand shape may also include a common user's hand holding the pen posture, and the writing key point of the writing hand shape is the pen tip. For details, see Figure 4 .

[0076] In this example, a second image recognition model is used to monitor the writing key points of the writing hand shape.

[0077] Specifically, a second image recognition model may be pre-established and trained using training data, wherein the training data includes pictures of writing hand shapes in historical live streams, hand shape category information, and writing key points corresponding to each marked writing hand shape.

[0078] Furthermore, the trained second image recognition model is used to process the picture identified as the writing hand shape and the corresponding hand shape category information to perform writing key point monitoring and recognition, and identify the writing key points of the writing hand shape.

[0079] In this example, the online live broadcast system includes at least one server, which can record and store relevant information data of air writing for displaying the air writing content in live broadcast or replay.

[0080] Specifically, when the server of the online live broadcast system recognizes a writing key point, it records the position of the writing key point and the position information of the writing key point.

[0081] For example, when using the image recognition model for detection, when it is monitored that the index finger of the first user (such as a teacher) is extended and the other fingers are clenched, the hand shape of the first user is judged to be a writing hand shape, and the writing key points of the first user are detected. The writing key point is the position of the index finger tip, and the position of the index finger tip is recorded in real time to the server of the online live broadcast system.

[0082] Therefore, through multiple detection processes, the positions of the key points of writing can be determined more accurately.

[0083] It should be noted that the above is only an example and cannot be understood as limiting the present invention. For example, the detection and recognition of the user's hand shape and the recognition of the writing key points of the hand shape can also be performed at the writing end, which is not limited in this embodiment.

[0084] Next, in step S104, a writing trajectory is generated according to the positions of the writing key points at multiple time points.

[0085] In this example, according to the positions of the writing key points identified at multiple time points, the movement trajectory of the writing key points is calculated as the writing trajectory.

[0086] Specifically, the positions of the writing key points are identified according to a plurality of time points, or the positions of the writing key points are identified from each frame image with a predetermined number of frame intervals, so as to obtain the positions of the writing key points identified at a plurality of time points.

[0087] More specifically, the predetermined frame interval is 0, 1, 2 or other numbers, and when it is 0, it is all frames. Therefore, it takes a relatively long time to process all frame images, so recognition is performed from each frame image with the predetermined frame interval. In this way, the data recognition processing speed of air writing can be improved.

[0088] Furthermore, the motion trajectory of the writing key points is calculated as the writing trajectory.

[0089] Specifically, the positions of the writing key points detected at the multiple time points are fitted with a straight line or a curve to generate a motion trajectory, and the generated motion trajectory is used as the writing trajectory, thereby generating a writing trajectory. Therefore, the writing trajectory of the air writing can be determined more quickly and accurately by the above method.

[0090] For example, when the position of the teacher's index finger is detected, the motion trajectory of the teacher's index finger is continuously recorded, but when the position of the index finger is no longer detected or the teacher's writing hand shape changes, the position of the index finger is stopped. In addition, the positions of multiple writing key points are fitted to generate a motion trajectory as the writing trajectory.

[0091] It should be noted that the above description is only given as an example and should not be construed as a limitation to the present invention.

[0092] Figure 5 It is a flowchart of another example of the air writing method for an online live class according to the first embodiment of the present invention.

[0093] like Figure 5 As shown, the method further includes step S501 of displaying the writing track at the writing end and / or the opposite end.

[0094] In this example, the interaction between the writing end and the opposite end is performed through an online live broadcast system, and the writing track can be saved on the online live broadcast system for display again during playback.

[0095] Preferably, the writing trace is separately saved in the first client and / or the second client so as to display or hide the writing trace in the video stream according to the operation of the first user or the second user.

[0096] Furthermore, the writing trace is coupled to the video stream so as to display the writing trace in real time in the video screen of the video stream of the first client and / or the second client.

[0097] Optionally, the writing track can be displayed or hidden in the user video stream of the writing end according to the user instruction of the writing end; or, the writing track can be displayed or hidden in the user video stream of the opposite end according to the user instruction of the opposite end.

[0098] It should be noted that the generated writing track can be superimposed on the video stream, and the writing track can be hidden / removed as needed. The above is only described as a preferred example and cannot be understood as a limitation of the present invention. For example, the writing track generated at the writing end may not be superimposed on the video stream, and can also be stored and transmitted independently.

[0099] Preferably, the writing track can be displayed in a courseware screen corresponding to the online live class.

[0100] In this way, a variety of display modes of writing tracks during interaction can be achieved, and the user experience can be improved.

[0101] It should be noted that the above description is only given as an example and should not be construed as a limitation to the present invention.

[0102] The process of the above-mentioned air writing method is only used to illustrate the present invention, wherein the order and number of steps are not particularly limited. In addition, the steps in the above-mentioned method can also be divided into two or three, or some steps can also be combined into one step, and adjustments are made according to actual examples.

[0103] Compared with the prior art, the present invention can more accurately detect the area ratio of the first user's body shape in the real-time image by using an image monitoring model, and can obtain more accurate position information of the first user; through the first image recognition model, it can more accurately detect whether the hand shape of the first user is a writing hand shape; through the second image recognition model, it can more accurately determine the position of the writing key points, thereby more quickly and accurately determining the writing trajectory of the air writing; it can improve the speed and accuracy of writing recognition, can realize a variety of display methods of writing trajectories in interaction, can improve the user experience, and can further optimize the air writing method in the online live class interaction. The above model can be placed on the server side, and the equipment on the writing side does not need to be improved in any way. It only needs to upload real-time video to have the air writing function during online live broadcast.

[0104] Example 2

[0105] Reference Figure 6 , the online live broadcast system of embodiment 2 of the present invention will be described.

[0106] According to a second aspect of the present invention, the present invention also provides an online live broadcast system, comprising: a writing end and a relative end; and at least one server for interaction between the writing end and the relative end during an online live class.

[0107] One of the at least one server generates a writing track using the air writing method described in Example 1, and transmits it to the opposite end and / or the writing end in the form of a video stream, and displays it on the writing end and / or the opposite end. Optionally, in some embodiments, the writing end generates a writing track using the air writing method described in Example 1, and uploads it in the form of a video stream.

[0108] Preferably, the server couples the generated writing trace to the video stream by direct transmission, transmits it to the opposite end (ie, the second client), and displays it in real time in the picture of the video stream of the opposite end.

[0109] Specifically, the writing end and the opposite end correspond to the first user and the second user respectively.

[0110] In another example, the writing track may be displayed in real time at the writing end and / or the opposite end, respectively.

[0111] Furthermore, according to the operation of the first user or the second user, the motion track can be displayed or hidden in the video stream. Specifically, according to the user instruction of the writing end, the writing track can be displayed or hidden in the user video stream of the writing end; or, according to the user instruction of the opposite end, the writing track can be displayed or hidden in the user video stream of the opposite end.

[0112] It should be noted that the above is only described as a preferred example and cannot be understood as a limitation of the present invention. In other examples, video stream transmission can also be performed by forwarding through other servers. In addition, in Example 2, the description of the same parts as in Example 1 is omitted.

[0113] Compared with the prior art, the system of the present invention can determine the writing trajectory of air writing more quickly and accurately by using the air writing method of the embodiment; it can improve the speed and accuracy of writing recognition, can realize multiple display methods of writing trajectories in interaction, and can improve the user experience.

[0114] Example 3

[0115] The following describes an apparatus embodiment of the present invention, which can be used to execute a method embodiment of the present invention. The details described in the apparatus embodiment of the present invention should be regarded as supplementary to the above method embodiment; details not disclosed in the apparatus embodiment of the present invention can be implemented with reference to the above method embodiment.

[0116] Reference Figures 7 to 10 , the air writing device 700 for online live classes according to the third embodiment of the present invention will be described.

[0117] According to the third aspect of the present invention, the present invention also provides an aerial writing device 700 for online live classes, which includes a first monitoring module 701 for monitoring the distance between the user and the camera device; a second monitoring module 702, when it is detected that the distance between the user and the camera device is within a preset distance range, detects the real-time image of the user to monitor the user's hand shape; a third monitoring module 703, when it is detected that the hand shape of the first user is a writing hand shape, starts to detect the writing hand shape to identify writing key points; a generation module 704, generates a writing trajectory according to the positions of the writing key points at multiple time points.

[0118] Preferably, the real-time detection of the user's real-time image is performed in real time to monitor the distance between the user and the camera device; optionally, the real-time detection of the user's real-time image is performed in real time to monitor the distance between the user and the camera device, including: detecting the area ratio of the user's body shape in the real-time image, and calculating the distance between the user and the camera device based on the area ratio.

[0119] like Figure 8As shown, the aerial writing device 700 may also include a model building module 801, which is used to build a first image recognition model so that the second monitoring module 702 uses the first image recognition model to detect whether the real-time image of the user contains a predefined writing hand shape. The image recognition model is a machine learning model that has been trained to recognize the writing hand shape of a human hand. When the real-time image is detected to contain a predefined writing hand shape, the detection process of the writing key points is started.

[0120] Preferably, the third monitoring module 703 can use a second image recognition model to detect the writing key points of the human hand shape, and the image recognition model is a machine learning model that has been trained to recognize the writing key points of the human hand shape. The model building module 801 can also be used to build the second image recognition model.

[0121] Preferably, the predefined writing hand shape is a posture in which the index finger is extended and the other fingers are clenched, and the writing key point of the human hand shape is the tip of the index finger.

[0122] Preferably, the generation module 704 can generate a writing trajectory according to the positions of the writing key points at multiple time points. This step may include: performing straight line or curve fitting on the positions of the writing key points at the multiple time points to generate the motion trajectory; generating a writing trajectory according to the motion trajectory.

[0123] Preferably, when the user writes on the writing end, a writing trajectory is generated and the writing trajectory is transmitted in real time to the opposite end interacting with the writing end.

[0124] Preferably, the method further comprises: coupling the writing track generated at the writing end to the video stream, and the writing track can be displayed or hidden in the video stream.

[0125] like Fig. 9 As shown, it also includes a display module 901, and the display module 901 is used to display the writing track at the writing end and / or the opposite end.

[0126] In another example, if Fig.10As shown, the air writing device 700 of the online live class is used for the interaction between the first client (i.e., the writing end) and the second client (i.e., the opposite end) during the online live class. The first client is connected to a camera device, which continuously captures real-time images to form a video stream and transmits it to the second client through the first client. The air writing device 700 includes: a first monitoring module 701, which monitors the distance between the first user and the camera device of the first client by real-time detection of the real-time image; a second monitoring module 702, which starts to detect the real-time image to monitor the hand shape of the first user when the distance is within a specific range; a third monitoring module 703, which starts to detect the writing hand shape to identify the writing key points when the hand shape of the monitored first user is a writing hand shape; and a generation module including a recording module 604 and a calculation module 605. The recording module 604 is used to record the position of the writing key point when the writing key point is identified; the calculation module 605 is used to calculate the movement trajectory of the writing key point as the writing trajectory based on the position of the writing key point identified at multiple time points; the display module 606 displays the writing trajectory in the first client and / or the second client.

[0127] It should be noted that, in Example 3, the description of the same parts as in Example 1 and Example 2 is omitted.

[0128] Those skilled in the art will appreciate that the modules in the above device embodiments may be distributed in the device as described, or may be changed accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments may be combined into one module, or may be further split into multiple submodules.

[0129] Compared with the prior art, the present invention can more accurately detect the area ratio of the first user's body shape in the real-time image by using an image monitoring model, and can obtain more accurate position information of the first user; through the first image recognition model, it can more accurately detect whether the hand shape of the first user is a writing hand shape; through the second image recognition model, it can more accurately determine the position of the writing key points, thereby being able to more quickly and accurately determine the writing trajectory of the air writing; it can improve the speed and accuracy of writing recognition, can realize a variety of display methods of the writing trajectory in the interaction, can improve the user experience, and can further optimize the air writing method in the online live class interaction.

[0130] Example 4

[0131] The following describes a computer device embodiment of the present invention, which can be regarded as a specific physical implementation of the method and device embodiments of the present invention. The details described in the computer device embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the computer device embodiment of the present invention can be implemented with reference to the above method or device embodiments.

[0132] Fig.11 is a structural diagram of a computer device according to an embodiment of the present invention, the computer device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes any one of the methods in the embodiments, including but not limited to Figure 1 method.

[0133] like Fig.11 As shown, the computer device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The computer device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0134] The memory stores a computer executable program, usually a machine-readable code, which can be executed by the processor to enable the computer device to perform the method of the present invention, or at least some steps in the method.

[0135] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0136] Optionally, in this embodiment, the computer device further includes an I / O interface, which is used for the computer device to exchange data with external devices. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0137] It should be understood that Fig.11 The computer device shown is only an example of the present invention, and the computer device of the present invention may also include elements or components not shown in the above examples. For example, some computer devices also include display units such as display screens, and some computer devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the computer device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as a computer device covered by the present invention.

[0138] Fig.12 Schematic diagram of a computer program product according to an embodiment of the present invention. Fig.12 As shown, a computer executable program is stored in a computer program product, and when the computer executable program is executed, the above-mentioned air writing method of the present invention is implemented. The computer program product may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer program product may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained in the computer program product may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0139] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0140] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the computer device to execute the method according to the present invention.

[0141] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or computer equipment, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for writing in the air, used to optimize the recognition of writing in the air during online live class interaction, characterized in that: include: Performing real-time detection on the real-time image of the user to monitor the distance between the user and the camera device, wherein the detection includes using an image monitoring model to detect the area ratio of the user's body shape in the real-time image and calculating the distance between the user and the camera device according to the area ratio; When it is detected that the distance between the user and the camera device is within a preset distance range, the user's hand shape recognition is started, and the real-time image of the monitored user is detected using a first image recognition model and the user's hand position and hand shape category information in each image are output to monitor the user's hand shape; the first image recognition model is a machine learning model that can recognize the hand shape of writing; When the user's hand shape is detected to be a writing hand shape, the second image recognition model is used to detect the writing hand shape to identify the writing key points, and the positions of the writing key points are recorded; the second image recognition model is a machine learning model that is trained to identify the writing key points in the writing hand shape; The motion trajectory of the writing key points is calculated based on the positions of the writing key points identified at multiple time points or the positions of the writing key points identified in each frame image at a predetermined frame interval to generate a writing trajectory, and the writing trajectory generated by the user at the writing end is transmitted in real time to the opposite end that interacts with the writing end, and the writing trajectory is displayed at the writing end and / or the opposite end.

2. The air writing method according to claim 1, characterized in that: Detect the area ratio of the user's body shape in the real-time image, including: Establish an image monitoring model, and use historical human frame data, human body pictures contained in historical live video streams, and human body position information in each picture that has been manually annotated as training data to train the image monitoring model; Input each picture of the real-time image of the monitored first client into the trained image monitoring model and output the position information of the first user corresponding to the first client in the picture, wherein the position information includes the area ratio of the first user's body shape in each picture of the real-time image, and the writing end is the first client.

3. The air writing method according to claim 1, characterized in that: The method further includes detecting the real-time image of the monitored user using the first image recognition model and outputting the hand position and hand shape category information of the user in each image to monitor the hand shape of the user. A first image recognition model is established based on a convolutional neural network, and a training data set is used to train the first image recognition model, wherein the training data set includes pictures containing hands in historical live streams and hand information in each picture, wherein the hand information includes hand positions and hand shape category information, wherein the hand shapes of writing in each picture are annotated by a user; The first image recognition model detects whether the real-time image of the user contains a predefined writing hand shape, and when it is detected that the real-time image contains the predefined writing hand shape, starts a writing key point detection process; Among them, according to different application scenarios, predefined writing hand shapes for writing, drawing or marking are used for detection and comparison to determine whether the detected hand shape is a predefined writing hand shape; and the predefined writing hand shape is a posture with the index finger extended and the other fingers closed.

4. The air writing method according to claim 3, characterized in that: The detection process of the writing key points includes: Pre-establishing a second image recognition model, and using training data to train the second image recognition model, the training data including pictures of hand shapes in historical live streams, hand shape category information, and writing key points corresponding to each marked hand shape; Using the trained second image recognition model, the image identified as a hand shape and the corresponding hand shape category information are processed to identify the writing key points of the hand shape, where the writing key points include the tip of a pen or the tip of an index finger; Record the location information of key writing points in real time.

5. The air writing method according to claim 1, characterized in that: The motion trajectory of the writing key points is calculated to generate the writing trajectory, including: Performing straight line or curve fitting on the positions of the writing key points at the multiple time points to generate motion trajectories of the writing key points; A writing trajectory is generated according to the motion trajectory of the writing key points.

6. The air writing method according to claim 1, characterized in that: The method of transmitting the writing track generated by the user at the writing end to the opposite end interacting with the writing end in real time also includes: The writing track generated at the writing end is coupled to the user video stream, and the writing track is displayed or hidden in the user video stream according to the user instruction at the writing end or the opposite end; or, The writing track generated at the writing end is sent to the opposite end in the form of an independent existing video.

7. An online live broadcast system, characterized in that: The online live broadcast system comprises: writing end and opposite end; at least one server for interaction between the writing end and the opposite end during the online live class, and The writing end or the at least one server generates a writing trajectory using the air writing method described in claim 1; the at least one server transmits the generated writing trajectory to the opposite end in the form of a video stream and displays it in the writing end and / or the opposite end.

8. An air writing device, characterized in that: Used to optimize the air writing recognition during online live class interactions, including: A first monitoring module is used to perform real-time detection on the real-time image of the user to monitor the distance between the user and the camera device, wherein the detection includes using an image monitoring model to detect the area ratio of the user's body shape in the real-time image and calculating the distance between the user and the camera device according to the area ratio; The second monitoring module, when detecting that the distance between the user and the camera device is within a preset distance range, starts the user's hand shape recognition, uses the first image recognition model to detect the user's real-time image and outputs the user's hand position and hand shape category information in each image to monitor the user's hand shape; the first image recognition model is a machine learning model that can recognize the hand shape of writing; The third monitoring module, when detecting that the user's hand shape is a writing hand shape, starts to use the second image recognition model to detect the writing hand shape to identify the writing key points; the second image recognition model is a machine learning model that has been trained to identify the writing key points in the writing hand shape; A generation module calculates the motion trajectory of the writing key points based on the positions of the writing key points identified at multiple time points or the positions of the writing key points identified in each frame image with a predetermined number of frame intervals to generate a writing trajectory, and transmits the writing trajectory generated by the user at the writing end to the opposite end interacting with the writing end in real time, and displays the writing trajectory at the writing end and / or the opposite end.

9. A computer device comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer executable program is executed by the processor, the processor performs the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images