Integrated gesture control PPT system, method and platform based on cloud-edge collaborative technology

Abstract

The present invention belongs to the field of PPT intelligent control technology, and discloses an integrated gesture control PPT system, method, and platform based on cloud-edge collaborative technology. The edge device obtains the original data of human joint coordinates; the network transmission technology is selected; three-dimensional reconstruction; the random forest model is used to judge the human posture; on the PC that controls the PPT, the preset software program is run, and the corresponding keyboard and mouse operations are performed according to the data analysis results transmitted from the cloud to complete the corresponding PPT operation. With the present invention, you can turn to the next page of PPT by waving your hand to the left, turn to the previous page of PPT by waving your hand to the right, call out the laser pen by pointing your hand, and mark with a fluorescent pen by swiping your hand. The shadow-like teaching platform provided by the present invention is an auxiliary teaching platform that can support teachers to upload PPT, students to download PPT, and can realize simultaneous interpretation of teachers' PPT projection.

Description

Technical Field

[0001] The present invention belongs to the field of PPT intelligent control technology, and in particular relates to an integrated gesture control PPT system, method, computer equipment, computer-readable storage medium, and shadow-like teaching cloud platform based on cloud-edge collaborative technology. Background Art

[0002] At present, in the process of teachers giving lectures, organizing reports, and corporate press conferences, the invention needs to show PPT. However, various situations are often encountered: the page turning pen is not brought, the page turning pen is out of power, and the computer controlling the PPT is too far away and someone has to be found to control the PPT. This seriously affects the effect of the explanation.

[0003] If there is a product that can complete a wonderful PPT presentation with just hands without the constraints of equipment, and can even write and draw in the air when necessary, it will definitely greatly facilitate PPT presenters, so new products need to be developed.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] (1) In the prior art, when showing PPT, if some conditions are not met during operation, the effect of explanation will be affected.

[0006] (2) The existing technology does not combine the camera face + skeleton dual recognition solution to control PPT, which cannot improve the efficiency of explanation.

[0007] (3) Existing PPT presentation equipment cannot interact with the audience remotely.

[0008] The difficulty of solving the above problems and defects is:

[0009] (1) With all the current technologies, the conventional way to explain PPT is to use a mouse and keyboard or a page-turning pen, which may make you feel uncomfortable during use.

[0010] (2) The common solution for existing wireless page turning technologies (such as wireless presenters) is to use Bluetooth to transmit page turning signals.

[0011] (3) With the existing laser pointer technology, the commonly used solutions are laser pointers of external devices.

[0012] (4) How to achieve the function of moving the laser pen to where the finger points simply by using the human skeleton in three-dimensional space and the position of each bone point.

[0013] (5) When the user uses the highlighter function with his left hand, how does the system determine the user's pen-down status and pen-release status?

[0014] (6) How to reduce the impact of misjudgment of some unnecessary actions by users on system functions.

[0015] (7) How to ensure that students who are far away from the screen can also see the content on the screen clearly while the speaker is showing the PPT.

[0016] The significance of solving the above problems and defects is:

[0017] (1) It allows users to control PPT through gestures only, reducing dependence on presenters, mice, and keyboards.

[0018] (2) Avoid situations where a user forgets to bring a wireless presenter or other PPT device, or the wireless presenter runs out of battery, or the wireless presenter is too far from the computer, causing the wireless presenter to become disconnected.

[0019] (3) The lecturer can control the PPT without being confined to the computer, which makes it easier for the lecturer to get close to the audience and improve the lecture effect.

[0020] (4) The lecturer can complete the operation of PPT through movements that are very consistent with his or her own body movement habits, which can greatly facilitate the lecturer's explanation of PPT.

[0021] (5) Through the platform provided by the present invention, teachers can upload PPT in real time, and students can also download PPT, which is conducive to the preservation and acquisition of teaching information.

[0022] (6) The platform provided by the present invention also supports real-time simultaneous interpretation of PPT, allowing listeners who are far away from the big screen to watch PPT in the cloud. Summary of the invention

[0023] In order to overcome the problems existing in the related art, the embodiments disclosed in the present invention provide an integrated gesture control PPT system, method, computer device, computer readable storage medium, and shadow-like teaching cloud platform based on cloud-edge collaborative technology. The present invention specifically provides a cloud (data transmission, data processing and analysis), edge (Raspberry Pi 4B + Intel Realsense D435 + Intel second-generation neural computing stick) collaborative integrated gesture control PPT solution.

[0024] The technical solution is as follows: an integrated gesture control PPT method based on cloud-edge collaborative technology, comprising:

[0025] Step 1: The edge device obtains the original data of human joint coordinates:

[0026] Step 2: Selection of network transmission technology;

[0027] Step 3: 3D reconstruction;

[0028] Step 4: The random forest model is used to judge the human body posture;

[0029] Step 5. On the PC that controls the PPT, run the preset software program, perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud, and complete the corresponding PPT operations.

[0030] In one embodiment of the present invention, in step one, the two-dimensional coordinates of 18 bone points of the human body in the RGB image and the depth values ​​of the corresponding points are determined using the openpose neural network; the two-dimensional coordinates and depth values ​​of each joint point constitute the original data of the current joint point.

[0031] In one embodiment of the present invention, the OpenPose network model is a bottom-up detection model that first identifies all possible key points in the image, outputs a colorful heat map to show the approximate position of the human body in the image, and identifies the connection relationship between all key points.

[0032] In one embodiment of the present invention, the OpenPose network model performs a single person posture estimation method comprising:

[0033] (1) One branch of the OpenPose network model is used to predict the confidence maps (S), and one branch is used to predict partial affinity PAFs;

[0034] (2) Use the Loss Function to ensure reliable convergence of the neural network.

[0035] (3) Use Gaussian distribution to calculate the heatmap and bone direction Vectormap.

[0036] (4) Perform skeletal joint connection and posture recognition prediction;

[0037] The Loss Function in step (2) is:

[0038]

[0039]

[0040]

[0041] When the human skeleton point Р slowly approaches the standard point k, it gradually reaches the peak value;

[0042] The normal peak expression of the skeleton point detection annotation point is:

[0043]

[0044] In one embodiment of the present invention, the step 2 selects to use socket technology in the intranet for data transmission to complete zero-delay transmission from the edge device to the PC;

[0045] The three-dimensional reconstruction in step three includes:

[0046] Using the camera internal parameters camera_factor, camera_cx, camera_cy, camera_fx,

[0047] camera_fy

[0048] Parameters, convert 2D coordinates into 3D coordinates x, y, z through the pinhole imaging model formula;

[0049] The pinhole imaging model is:

[0050]

[0051] In one embodiment of the present invention, the step 4 of using the random forest model to determine the human body posture specifically includes:

[0052] The bone angles of the human body in three-dimensional space are used as the basis for the prediction of the random forest model. Multiple pieces of data are then collected and the random forest model is trained using the calculated bone angles. The trained model is deployed to the cloud and the cloud is used to identify the bone angles in three-dimensional space.

[0053] Another object of the present invention is to provide an integrated gesture control PPT system based on cloud-edge collaboration technology, including:

[0054] Camera, used to realize the recognition of human body posture;

[0055] Raspberry Pi, used to obtain the original data of human joint coordinates and perform 3D reconstruction;

[0056] The Neural Compute Stick, which is installed on the Raspberry Pi, is used to judge human posture using the random forest model;

[0057] The cloud is used to transmit data analysis results;

[0058] The PC is used to perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud to complete the corresponding PPT operations.

[0059] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the integrated gesture control PPT method based on cloud-edge collaboration technology.

[0060] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the integrated gesture control PPT method based on cloud-edge collaboration technology.

[0061] Another object of the present invention is to provide a shadow-like teaching cloud platform, which supports teachers uploading PPTs, students downloading PPTs, and enables teachers to show PPTs and perform simultaneous interpretation; specifically, the following are implemented:

[0062] In the first step, the edge device obtains the original data of human joint coordinates;

[0063] The second step is to choose the network transmission technology;

[0064] The third step is 3D reconstruction;

[0065] The fourth step is to use the random forest model to judge the human body posture;

[0066] Step 5: Run the pre-installed software on the PC that controls the PPT. The program will perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud to complete the corresponding PPT operations.

[0067] Step 6: When the teacher registers on the platform, the camera scans the face and skeleton information and enters it. The system prompts that the registration is successful and the teacher's identity information is automatically entered into the system.

[0068] When the system registration is successful, the system will automatically log in as the teacher and give PPT lectures.

[0069] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows:

[0070] Through the present invention, the user can turn to the next page of PPT by waving the hand to the left, turn to the previous page of PPT by waving the hand to the right, call out the laser pen by pointing the hand, and mark with a fluorescent pen by swiping the hand.

[0071] The shadow-like teaching platform provided by the present invention is an auxiliary teaching platform that can support teachers to upload PPTs, students to download PPTs, and can realize simultaneous interpretation of teachers showing PPTs.

[0072] The platform of the present invention is different from the previous platforms. In order to maximize the convenience of teachers' class operations and simplify the process of teachers logging into the system, the present invention innovatively uses a camera face + skeleton dual recognition solution to register and log in the teacher's identity.

[0073] The comparison between the present invention and the existing technology on the market is as follows:

[0074]

[0075] Taking the university classroom that needs to show PPT as an example, the integrated gesture control PPT system based on cloud-edge collaborative technology provided by the present invention will have the following positive effects:

[0076] Teachers will no longer be bound by external devices (such as mouse, keyboard, and page turning pen) when teaching, and can complete wonderful classes with gestures alone. The range of motion of teachers manipulating PPT is not large, and long-term use will not cause muscle fatigue. Teachers can walk into students to explain PPT, which can increase the interactivity of the classroom and the students' sense of participation in the classroom. Students who cannot see the PPT on the big screen in the classroom can watch it synchronously on the teaching platform provided by the present invention to improve the efficiency of students' listening to the class. When teachers want to use functions such as fluorescent pens that require a mouse, through the integrated gesture control PPT system based on cloud-edge collaborative technology provided by the present invention, teachers will no longer need to walk to the computer to operate, and can use it anytime, anywhere. At the beginning of the class, the classroom walks to the camera, and the camera has already identified the teacher's identity and automatically logged into the system for it, without the teacher performing cumbersome operations such as account login. When the teacher wants to register on the platform, he can stand in front of the camera and scan a circle to enter the face and skeleton information. The system prompts that the registration is successful, and the teacher's identity information will be automatically entered into the system. Whenever a teacher who has successfully registered in the system stands on the podium, the system will automatically log in as the teacher. The teacher can find the PPT that he wants to explain in the files he has uploaded to the system, open it and start explaining the PPT. At the same time, the student terminal can watch the current page of the PPT explained by the teacher in real time. At the same time, in order to facilitate the teacher to upload the PPT in a place without a camera (such as home), the system of the present invention also supports account and password login.

[0077] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0079] Figure 1 It is a flow chart of an integrated gesture control PPT method based on cloud-edge collaboration technology provided by an embodiment of the present invention.

[0080] Figure 2 It is a schematic diagram of an integrated gesture-controlled PPT system based on cloud-edge collaboration technology provided by an embodiment of the present invention.

[0081] In the picture: 1. Camera; 2. Raspberry Pi; 3. Neural Compute Stick.

[0082] Figure 3 It is a schematic diagram of an integrated gesture-controlled PPT system based on cloud-edge collaboration technology provided in an embodiment of the present invention.

[0083] Figure 4 This is a rendering of the original data of the coordinates of 18 joint points of the simulated human body provided by an embodiment of the present invention.

[0084] Figure 5 It is a flow chart for obtaining original data of joint points provided by an embodiment of the present invention.

[0085] Figure 6 This is a three-dimensional reconstruction flow chart provided by an embodiment of the present invention.

[0086] Figure 7 It is a three-dimensionally reconstructed human skeleton image provided by an embodiment of the present invention.

[0087] Figure 8 The present invention provides a flow chart of a method for human body posture recognition.

[0088] Fig. 9 It is a flow chart of the cloud platform control method of the shadow-like teaching system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0090] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used in the present invention are for illustrative purposes only and do not represent the only implementation method.

[0091] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used in the present invention in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in the present invention includes any and all combinations of one or more of the related listed items.

[0092] like Figure 1 As shown, the integrated gesture control PPT method based on cloud-edge collaboration technology provided by the present invention includes:

[0093] The edge device obtains the original data of human joint coordinates:

[0094] Selection of network transmission technology;

[0095] 3D reconstruction;

[0096] The random forest model is used to judge human posture;

[0097] On the PC that needs to control PPT, run the pre-installed software. The program can perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud to complete the corresponding PPT operations.

[0098] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0099] Example

[0100] 1. The integrated gesture control PPT system based on cloud-edge collaboration technology provided by the present invention includes (such as Figure 2 shown):

[0101] Camera A selection:

[0102] To achieve accurate recognition of human posture, the camera that shoots the human body needs to have a higher RGB sensor resolution and frame rate and a wider depth range. Therefore, the present invention selects the Intel Realsense D435 depth camera that is most suitable for the product of the present invention on the market today. Its RGB sensor resolution and frame rate can reach 1920×1080, 30fps, and the depth range can reach 0.105m~10m. It exceeds 95% of the products on the market, but the price is only slightly higher than the average of the cameras on the market.

[0103] Choice of edge device:

[0104] Since the camera of the present invention cannot be used on a PC, it needs to be connected to an edge device. However, the common disadvantage of edge devices is that the computing power is insufficient and cannot meet the high-speed data calculation and transmission required by the present invention. Therefore, the present invention selects the Raspberry Pi (model 4B) B with a fairly good computing power, and matches it with the Intel Neural Compute Stick C to improve the computing power of the Raspberry Pi.

[0105] Figure 3 It is a schematic diagram of an integrated gesture-controlled PPT system based on cloud-edge collaboration technology provided in an embodiment of the present invention.

[0106] 2. In a preferred embodiment of the present invention, the integrated gesture control PPT method based on cloud-edge collaboration technology provided by the present invention includes:

[0107] The first step is for the edge device to obtain the original data of human joint coordinates:

[0108] like Figure 5 As shown: In order to obtain the original data of the coordinates of the 18 joints of the human body, the process of obtaining the original data of the joints is as follows: Figure 5 As shown, the present invention uses the openpose neural network to determine the two-dimensional coordinates of 18 human skeleton points in the RGB image, as well as the depth values ​​of the corresponding points. The two-dimensional coordinates and depth values ​​of each joint point constitute the original data of the current joint point.

[0109] The specific implementation process is as follows:

[0110] 1. Connect the camera and the Intel neural computing stick to the Raspberry Pi, and run the data acquisition program of the present invention on the Raspberry Pi.

[0111] 2. Since the computing power of the Raspberry Pi cannot support the huge amount of calculation required by the present invention to calculate the coordinates of the human joint points, the present invention uses the OpenVINO suite to improve the calculation speed. OpenVINO is a tool suite developed by Intel based on its existing hardware platform to accelerate the development of high-performance computer vision and deep learning visual applications. It supports deep learning on hardware accelerators on various Intel platforms and allows direct heterogeneous execution. It supports Windows and Linux systems, Python / C++ languages.

[0112] 3. The OpenPose network model is the basis of the present invention. It is widely used in the field of human posture recognition. It can effectively detect multiple people in an image. Part Affinity Fields is used to learn the parts of the body and their associations. The Part Affinity Fields (PAFs) method connects some body joints to form human posture. It uses global texture information and a bottom-up method to achieve real-time performance and high precision. This is a bottom-up human posture detection model. It is different from the top-down network model. The top-down detection model mainly determines the number of people in the picture first, and then detects key points for each person after determining the person. Openpose is a bottom-up detection model. It first identifies all possible key points in the picture. The output of the model is a heatmap (i.e., a colorful heatmap that can show the approximate position of the human body in the image), as well as identifying the connection relationship between all key points. This connection between key points is called a vector field or a human key point affinity field. The output of the model is paf (Part Affinity Fields partial affinity). This model simultaneously detects and connects human key points.

[0113] The idea of ​​OpenPose's single-person pose estimation algorithm is that the entire algorithm process is:

[0114] (1) One branch is used to predict confidence maps (S), and the other branch is used to predict partial affinities (PAFs).

[0115] (2) Use the Loss Function to ensure reliable convergence of the neural network.

[0116] (3) Use Gaussian distribution to calculate the heatmap and bone direction Vectormap.

[0117] (4) Perform skeletal joint connection and posture recognition prediction.

[0118] The specific algorithm idea of ​​OpenPose multi-person posture estimation. On the basis of the original data, the specific parameters are corrected through the previous ten-layer neural network and transmitted to the initial stage as the mapping F. PAFs, generally referred to as bone direction vectors in the present invention, represent the specific direction of the limbs in the skeleton. It is divided into two parts: blue and orange. Blue represents CNNs, which are generally used for engine reasoning and prediction. After repeated training, S is improved, which is generally referred to as confidence in the present invention. Orange represents the so-called convolution kernel, which can greatly reduce the workload of the present invention and improve the efficiency and number of layers of refinement. In other words, the confidence S mentioned above in the present invention is improved, that is, the specific position and posture behavior of the body are more accurately identified.

[0119] Formulas (2-1), (2-2), and (2-3) reveal the neural network posture recognition loss function. In the following formula, PAFs is represented by L*, and confidence is represented by S*. If a bone point is blocked or not detected, the point is removed and not calculated.

[0120]

[0121] Formula (2-4) reveals the normal peak value of the skeleton point detection annotation point. When the human skeleton point Р of the present invention slowly approaches the standard point k, it will gradually reach the peak value as time goes by.

[0122]

[0123] The second step is to choose the network transmission technology:

[0124] After selecting the underlying device, the present invention needs to determine the data transmission layer technology. Since the amount of single data transmitted by the present invention is very small and the data encoding is not complicated, the stability and transmission rate of the transmission layer are particularly important. Therefore, the present invention chooses to use socket technology in the intranet for data transmission to complete zero-delay transmission from the edge device to the PC.

[0125] The third step is 3D reconstruction:

[0126] The 2D coordinates and depth value data of the 18 skeleton points of each frame of the human body are uploaded to the PC through the cloud for 3D reconstruction. Figure 6 , the human skeleton after 3D reconstruction is as follows Figure 7 The two-dimensional coordinates and depth value data of the 18 bone points are shown in the following table:

[0127] 2021-08-16 18:10:05: data-41: (270,189, 2.70), (270,216, 2.74), (248,219, 2.83), (206,223, 2.86), (169,219, 2.80), (293,216,2.78), (330,219,2.87), (364,216,2.90), (255,298,2.79), (259,351,2.78), (263, 396, 2.90), (285, 298, 2.73), (289, 3472.87), (289, 392, 3.01), (263, 182, 2.69), (274, 182, 2.69), (259, 189, 2.72), (282, 186, 3.18),

[0128] like Figure 6As shown, the specific steps of 3D reconstruction are:

[0129] 1. The PC communicates remotely with the Raspberry Pi through the socket and receives the data of 18 skeleton points of the human body in the current frame (2D coordinates + depth value).

[0130] 2. With the two-dimensional coordinates and depth values ​​of the human skeleton points, the present invention can use the camera internal parameters camera_factor, camera_cx, camera_cy, camera_fx, camera_fy and other parameters to obtain the human skeleton points.

[0131] The hole imaging model formula (2-5) converts 2D coordinates into 3D coordinates x, y, and z.

[0132]

[0133] 3. After calculating the 3D coordinates of the current bone point, connect the corresponding bone points to complete the 3D reconstruction of the human skeleton. Figure 4 If the data of the current frame cannot be connected to the human skeleton (such as occlusion of bone points, data loss), the next frame will be reconstructed immediately.

[0134] 4. After the 3D reconstruction of the skeleton of the current frame is completed, the 3D reconstruction of the skeleton of the next frame can be carried out.

[0135] The fourth step is to use the random forest model to judge the human posture:

[0136] In the technical design of the present invention, the most important thing is the construction of a human posture recognition model. Figure 8 As shown. Most of the human posture recognition technologies on the market today are based on RGB image recognition. If the solution of the present invention only uses recognition technology based on two-dimensional images, it will have great limitations. The angle between the human body and the camera, or even the distance, will have a huge impact on the model prediction results. Therefore, the present invention innovatively uses the bone angle of the human body in three-dimensional space (such as the angle between the left shoulder and the left arm is called the left shoulder angle) as the basis for random forest model prediction. Tens of thousands of data were collected, and the accuracy of the random forest model trained by the calculated bone angle can reach more than 97%. The trained model is deployed to the Huawei cloud. By utilizing the processing speed of the cloud, the present invention can save most of the computing power and minimize the time required from data collection to data result transmission.

[0137] like Figure 8 As shown in the figure, the specific construction method of the human posture recognition model is as follows:

[0138] 1. First, the original data set for training the model of the present invention is collected according to different actions (such as waving to the left, waving to the right, etc.)

[0139] 2. In order to prevent the prediction results of the model from being affected by external factors such as distance angle, the present invention calculates 9 bone angles of the human skeleton in three-dimensional space: flexion angle (angle of body flexion), hip angle (left and right), shoulder angle (left and right), knee angle (left and right), and arm angle (left and right) as the training set for model training of the present invention.

[0140] 3. In order to enable the model to have different recognition results for different actions, the present invention manually adds different labels to different actions. Different labels correspond to different recognition results.

[0141] 4. Train the random forest model using the labeled dataset.

[0142] The process of using the human posture recognition model is as follows:

[0143] 1. PC can achieve remote communication with Raspberry Pi through socket. After Raspberry Pi calculates the human skeleton data of the current frame, it transmits the skeleton data of 18 points of the human body to PC through socket.

[0144] 2. When the PC receives the data of 18 bone points of the human body in the current frame, it immediately calculates the bone angles of the human body.

[0145] 3. After the human skeleton angle of the current frame is sent to the trained human posture recognition random forest model, the predicted label of the random forest for the current frame action, that is, the predicted action, can be obtained.

[0146] The fifth step is to run the pre-installed software on the PC that needs to control the PPT. The program can perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud to complete the corresponding PPT operations.

[0147] The shadow-like teaching cloud platform of the present invention is an auxiliary teaching platform that can support teachers to upload PPTs, students to download PPTs, and can realize simultaneous interpretation of teachers showing PPTs.

[0148] The platform of the present invention is different from the previous platforms. In order to maximize the convenience of teachers' class operations and simplify the process of teachers logging into the system, the present invention innovatively uses a camera face + skeleton dual recognition solution to register and log in the teacher's identity.

[0149] The operating principle of the shadow-like teaching cloud platform of the present invention is as follows: Fig. 9 shown.

[0150] Specifically include:

[0151] The first step is for the edge device to obtain the original data of human joint coordinates:

[0152] In order to obtain the 18 joints of the human body (such as Figure 5 ) coordinates, the process of obtaining the original data of the joint points is as follows Figure 5 As shown, the present invention uses the openpose neural network to determine the two-dimensional coordinates of 18 human skeleton points in the RGB image, as well as the depth values ​​of the corresponding points. The two-dimensional coordinates and depth values ​​of each joint point constitute the original data of the current joint point.

[0153] The OpenPose network model is the basis of the present invention. It is widely used in the field of human posture recognition. It can effectively detect multiple people in an image. Part Affinity Fields is used to learn the parts of the body and their associations. The Part Affinity Fields (PAFs) method connects some body joints to form human posture. It uses global texture information and a bottom-up method to achieve real-time performance and high precision. This is a bottom-up human posture detection model. It is different from the top-down network model. The top-down detection model mainly determines the number of people in the picture first, and then detects key points for each person after determining the person. Openpose is a bottom-up detection model. It first identifies all possible key points in the picture. The output of the model is a heatmap (i.e., a colorful heatmap that can show the approximate position of the human body in the image), as well as identifying the connection relationship between all key points. This connection between key points is called a vector field or a human key point affinity field. The output of the model is paf (Part Affinity Fields partial affinity). This model simultaneously detects and connects human key points.

[0154] The idea of ​​OpenPose's single-person pose estimation algorithm is that the entire algorithm process is:

[0155] (1) One branch is used to predict confidence maps (S), and the other branch is used to predict partial affinities (PAFs).

[0156] (2) Use the Loss Function to ensure reliable convergence of the neural network.

[0157] (3) Use Gaussian distribution to calculate the heatmap and bone direction Vectormap.

[0158] (4) Perform skeletal joint connection and posture recognition prediction.

[0159] The specific algorithm idea of ​​OpenPose multi-person posture estimation. On the basis of the original data, the specific parameters are corrected through the previous ten-layer neural network and transmitted to the initial stage as the mapping F. PAFs, generally referred to as bone direction vectors in the present invention, represent the specific direction of the limbs in the skeleton. It is divided into two parts: blue and orange. Blue represents CNNs, which are generally used for engine reasoning and prediction. After repeated training, S is improved, which is generally referred to as confidence in the present invention. Orange represents the so-called convolution kernel, which can greatly reduce the workload of the present invention and improve the efficiency and number of layers of refinement. In other words, the confidence S mentioned above in the present invention is improved, that is, the specific position and posture behavior of the body are more accurately identified.

[0160] Formulas (2-1), (2-2), and (2-3) reveal the neural network posture recognition loss function. In the following formula, PAFs is represented by L*, and confidence is represented by S*. If a bone point is blocked or not detected, the point is removed and not calculated.

[0161]

[0162]

[0163]

[0164] Formula (2-4) reveals the normal peak value of the skeleton point detection annotation point. When the human skeleton point P of the present invention slowly approaches the standard point k, it will gradually reach the peak value as time goes by.

[0165]

[0166] The second step is to choose the network transmission technology:

[0167] After selecting the underlying device, the present invention needs to determine the data transmission layer technology. Since the amount of single data transmitted by the present invention is very small and the data encoding is not complicated, the stability and transmission rate of the transmission layer are particularly important. Therefore, the present invention chooses to use socket technology in the intranet for data transmission to complete zero-delay transmission from the edge device to the PC.

[0168] The third step is 3D reconstruction:

[0169] The 2D coordinates and depth value data of the 18 skeleton points of each frame of the human body are uploaded to the PC through the cloud for 3D reconstruction. Figure 6 , the human skeleton after 3D reconstruction is as follows Figure 7 The two-dimensional coordinates and depth value data of the 18 skeleton points are shown in the following table:

[0170] 2021-08-16 18:10:05: data-41: (270, 189, 2.70), (270, 216, 2.74), (248, 219, 2.83), (206, 223, 2.86), (169, 219, 2.80), (293, 216, 2.78), (330, 219, 2.87), (364, 216, 2.90), (255, 298, 2.79), (259, 351, 2.78), (263, 396, 2.90), (285, 298, 2.73), (289, 347, 2.87), (289, 392, 3.01), (263, 182, 2.69), (274, 182, 2.69), (259, 189, 2.72), (282, 186, 3.18),

[0171] With the two-dimensional coordinates and depth values ​​of the human skeleton points, the present invention can use the camera internal parameters camera_factor, camera_cx, camera_cy, camera_fx, camera_fy and other parameters to obtain the human skeleton points.

[0172] The hole imaging model formula (2-5) converts 2D coordinates into 3D coordinates x, y, and z.

[0173]

[0174] The fourth step is to use the random forest model to judge the human posture:

[0175] In the technical design of the present invention, the most important thing is the construction of a human posture recognition model. Figure 8 As shown in the figure, the human body's bone angle in three-dimensional space (such as the angle between the left shoulder and the left arm is called the left shoulder angle) is used as the basis for the random forest model prediction. Tens of thousands of data were collected and the random forest model was trained by the calculated bone angle. The trained model was deployed to the Huawei cloud, and the cloud was used to process the bone angle in three-dimensional space.

[0176] The fifth step is to run the pre-installed software on the PC that needs to control the PPT. The program can perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud to complete the corresponding PPT operations.

[0177] Step 6. When a teacher wants to register on the platform, he or she can stand in front of the camera and scan around to enter the facial and skeleton information. The system will prompt that the registration is successful and the teacher's identity information will be automatically entered into the system.

[0178] Whenever a teacher who has successfully registered in the system stands on the podium, the system will automatically log in as the teacher. The teacher can find the PPT he wants to explain in the files he has uploaded to the system, open it and start explaining the PPT. At the same time, students can watch the current page of the PPT being explained by the teacher in real time.

[0179] In a preferred embodiment of the present invention, in order to facilitate teachers to upload PPT in places without cameras (such as home), the system of the present invention also supports account and password login.

[0180] In addition, the present invention can also add the function of writing in the air. Since the present invention has realized the functions of controlling the fluorescent pen and the laser pen in the air by gesture, the function of using the marker pen in the air by gesture is achievable. Since the accuracy of the camera currently used in the present invention does not support the realization of the writing in the air function of the present invention, the possibility of realizing this function is very high.

[0181] At the same time, the camera in the present invention can also realize the human body tracking function. Since the present invention can realize the recognition of human identity through human face and human skeleton, and then by adding some electric control operations to the camera (such as up, down, left, right, etc.), the camera can be rotated to follow a specific person, so that the person is always in front of the camera. With this function, the accuracy of human posture recognition can be greatly improved. At the same time, taking the scene of using the present invention in the classroom as an example, with the camera tracking function, the teacher can walk in any corner of the classroom to explain the PPT without worrying about walking out of the shooting range of the camera.

[0182] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0183] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure should be limited by the appended claims.

Claims

1. An integrated gesture control PPT method based on cloud-edge collaboration technology, characterized in that: The integrated gesture control PPT method based on cloud-edge collaborative technology includes: Step 1: The edge device obtains the original data of human joint coordinates: Step 2: Selection of network transmission technology; Step 3: 3D reconstruction; Step 4: The random forest model is used to judge the human body posture; Step 5: On the PC that controls the PPT, run the preset software program, perform corresponding keyboard and mouse operations according to the data analysis results sent from the cloud, and complete the corresponding PPT operations; In the step 1, the two-dimensional coordinates of 18 bone points of the human body in the RGB image and the depth values ​​of the corresponding points are determined using the OpenPose neural network; the two-dimensional coordinates and depth values ​​of each joint point constitute the original data of the current joint point; The OpenPose network model is a bottom-up detection model that first identifies all possible key points in the image, outputs a colorful heat map to show the position of the human body in the image, and identifies the connection relationship between all key points; The OpenPose network model for single-person posture estimation method includes: (1) One branch of the OpenPose network model is used to predict the confidence maps (S), and one branch is used to predict partial affinity PAFs; (2) Use the Loss Function to ensure reliable convergence of the neural network; Among them, the Loss Function loss function is: ； When the human skeleton point Р slowly approaches the standard point k, it gradually reaches the peak value; The normal peak expression of the skeleton point detection annotation point is: ； (3) Use Gaussian distribution to calculate the heatmap and bone direction Vectormap; (4) Perform skeletal joint connection and posture recognition prediction; The step 2 selects to use socket technology in the intranet for data transmission, completing zero-delay transmission from the edge device to the PC; The three-dimensional reconstruction in step three includes: Using the camera intrinsic parameters camera_factor, camera_cx, camera_cy, camera_fx, camera_fy, the 2D coordinates are converted into 3D coordinates x, y, z through the pinhole imaging model formula; The pinhole imaging model is: ； The step 4 of using the random forest model to judge the human posture specifically includes: The bone angles of the human body in three-dimensional space are used as the basis for prediction of the random forest model. Multiple pieces of data are collected and the random forest model is trained by the calculated bone angles. The trained model is deployed to the cloud and the cloud is used to identify the bone angles in three-dimensional space.

2. An integrated gesture control PPT system based on cloud-edge collaboration technology that implements the integrated gesture control PPT method based on cloud-edge collaboration technology as described in claim 1, characterized in that: The integrated gesture control PPT system based on cloud-edge collaborative technology includes: Camera, used to realize the recognition of human body posture; Raspberry Pi, used to obtain the original data of human joint coordinates and perform 3D reconstruction; The Neural Compute Stick, which is installed on the Raspberry Pi, is used to judge human posture using the random forest model; The cloud is used to transmit data analysis results; The PC is used to perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud to complete the corresponding PPT operations.

3. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the integrated gesture control PPT method based on cloud-edge collaborative technology as described in claim 1.

4. A computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the integrated gesture control PPT method based on cloud-edge collaborative technology as described in claim 1.

5. A shadow-like teaching cloud platform, characterized in that: The shadow-like teaching cloud platform implements the integrated gesture control PPT method based on cloud-edge collaborative technology as described in claim 1, supports teachers to upload PPT, students to download PPT, and realizes simultaneous interpretation of teachers showing PPT; the specific steps are: In the first step, the edge device obtains the original data of human joint coordinates; The second step is to choose the network transmission technology; The third step is 3D reconstruction; The fourth step is to use the random forest model to judge the human body posture; Step 5: Run the pre-installed software on the PC that controls the PPT. The program will perform corresponding keyboard and mouse operations based on the data analysis results sent from the cloud to complete the corresponding PPT operations. Step 6: When the teacher registers on the platform, the camera scans the face and skeleton information and enters it. The system prompts that the registration is successful and the teacher's identity information is automatically entered into the system. When the system registration is successful, the system will automatically log in as the teacher and give PPT lectures.

Images