A computer vision-based intelligent interactive teaching method and system

Abstract

The application discloses an intelligent interactive teaching method and system based on computer vision, which mainly comprises a login verification module, a gesture teaching module, a classroom light source control module, a classroom fan control module, a log information module, a personnel management module, a personal information module and other modules; the gesture teaching module adopts a correlation function to collect and recognize information of a hand, and distinguishes a hand state through a hand state detection method, so as to select to start a mobile mouse program or a mouse clicking program; the computer vision technology is combined with a traditional teaching system, teaching functions are realized through different gestures in a gesture recognition mode; other media is not needed to realize operation, cost can be reduced, a program is logged in through face recognition, face image direction gradient histogram feature description information dataset vectors are acquired through camera shooting, and then face comparison is carried out, so that safety and operability are improved.

Description

Technical Field

[0001] This invention belongs to the field of facial recognition processing and human-computer interaction technology, and specifically relates to an intelligent interactive teaching method and system based on computer vision. Background Technology

[0002] In modern education, the use of computer-related applications in daily teaching has become increasingly common. However, due to the large amount of teaching content, teachers need to spend a lot of time interacting with teaching equipment, resulting in a heavy workload on multimedia devices, which leads to excessive distraction and reduced teaching efficiency.

[0003] With the rapid development of computer vision technology and smart terminal devices, vision-based gesture interaction has become a widely used human-computer interaction method due to its natural, user-friendly, and convenient advantages. The key technologies mainly include vision-based gesture recognition, estimation, and authentication. These technologies are closely related to each other and have long been a research hotspot in academia and industry.

[0004] Therefore, intelligent interactive teaching methods and systems that integrate increasingly sophisticated computer vision technology with traditional teaching systems have emerged. However, in gesture interaction, the hand area in images or videos is relatively small and the hand joints are highly flexible. Vision-based gesture recognition, estimation, and authentication face challenges in gesture image or video processing and analysis, such as low image resolution, diverse viewpoints, similarity and occlusion between fingers, and motion blur caused by rapid hand movements. These challenges pose significant hurdles to vision-based gesture interaction. Summary of the Invention

[0005] This invention provides an intelligent interactive teaching method and system based on computer vision, aiming to address the problem that while the use of computer-related applications in daily teaching has become increasingly common in modern education, the large amount of teaching content requires teachers to spend a significant amount of time interacting with teaching equipment, resulting in a heavy workload for multimedia devices, which leads to excessive distraction and reduced teaching efficiency.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A computer vision-based intelligent interactive teaching method includes the following steps:

[0008] S1: Preset a computer image processing vision framework, which is used to acquire camera gesture interaction video stream;

[0009] S2: Invoke the hand detection method in the computer image processing vision framework to detect the hand contour of the gesture interaction video stream, and pre-set the first parameter value as the minimum confidence value of the hand detection model;

[0010] S3: When the detection value exceeds the first parameter value, the hand detection is determined to be successful, and the 21 preset key information points of the hand are located; when the detection value does not exceed the first parameter value, the hand detection fails, and the computer image processing vision framework continues to perform hand detection on the next gesture interaction video stream.

[0011] S4: The hand key point mapping method in the computer image processing vision framework identifies the key information points of the hand and converts them into frame images in RGB format. It then connects the key information points of the hand, locates the coordinates of the key information points of the hand, defines the range of the hand (i.e., draws a rectangle), detects the coordinates of the rectangle range and its height and width, and then identifies whether the hand is the left or right hand according to the preset recognition rules.

[0012] S5: The hand state detection method in the cross-platform GUI automation toolkit of the computer image processing vision framework is called to detect hand information. After detection, a sequence of 5 elements is returned, each element corresponding to a finger. When the element is 1, it means that the finger is upright, and when the element is 0, it means that the finger is bent. When the second element in the sequence is 1 and the third element is 0, that is, the index finger is upright and the middle finger is bent, the mouse movement operation is enabled, and the mouse moves with the position of the index fingertip. When the second element in the sequence is 1 and the third element is 1, that is, the index finger and the middle finger are upright at the same time, the parameters (pt1, pt2, img) are substituted into the fingertip distance measurement method to collect hand information again, where img is the image information transmitted from the camera to the teaching computer, and pt1 and pt2 are the coordinates of the fingertips of the index finger and the middle finger, respectively. When the pixel distance between the fingertips of the index finger and the middle finger is less than 50, the mouse click operation is enabled.

[0013] Preferably, in step S5, the linear interpolation method np.interp(x,xp,fp) is used to map the width and height of the rectangle to the width and height of the computer screen, where x is the horizontal coordinate array mapped to the screen, xp is the horizontal coordinate array of the rectangle, and fp is the vertical coordinate array of the rectangle.

[0014] Preferably, the width and height dimensions of the computer screen are obtained, and then the mouse is moved to the preset coordinates.

[0015] Preferably, in step S5, during the mouse movement, the function cLocx=pLocx+(x3-p-Locx) / smooth is called to adjust the smoothness of mouse movement, where cLocx is the x-coordinate of the mouse position in the previous frame, pLocx is the x-coordinate of the mouse position in the frame before the previous frame, x3 is the x-coordinate of the current mouse position, and s-mooth is a customizable smoothness coefficient. By changing smoothness, the smoothness of mouse movement can be adjusted.

[0016] A computer vision-based intelligent interactive teaching system mainly includes a login verification module, a gesture teaching module, a classroom lighting control module, a classroom fan control module, a log information module, a personnel management module, a personal information module, and other modules. The gesture teaching module is used to execute the computer vision-based intelligent interactive teaching method as described in claims 1-4.

[0017] The login verification module further includes a face recognition unit and an account password verification unit. The face recognition unit is configured to perform the following steps:

[0018] S6: Convert the face video stream captured by the camera to grayscale;

[0019] S7: The input image is normalized in color space using the Gamma correction method, which adjusts the contrast of the image.

[0020] S8: Calculate the gradient of each pixel in the horizontal and vertical directions of the image, and calculate the gradient direction value of each pixel position accordingly to capture contour information;

[0021] S9: Divide the image into several small units, count the gradient histogram of each unit, that is, count the number of different gradients, and the feature description information of each unit can be formed.

[0022] S10: Combine several units into a block, and concatenate the feature description information of all units in a block to obtain the directional gradient histogram feature description information of the block;

[0023] S11: Concatenate the directional gradient histogram feature description information of all blocks in the image to obtain the directional gradient histogram feature description information dataset of the image.

[0024] S12: Collect the histogram of oriented gradients feature description information dataset, classify it according to different photos, and then store it in a CSV file. Calculate the Euclidean distance of the histogram of oriented gradients feature description information data in each CSV file, and take the dataset with the smallest Euclidean distance to recognize and match each face. If the two match, the face recognition is successful and the login is completed.

[0025] Preferably, in step S6, the grayscale conversion formula is:

[0026] ;

[0027] In the formula, Gray is the grayscale value, R is the red value, G is the green value, and B is the blue value.

[0028] Preferably, in step S8, the gradient transformation formula for any pixel (x, y) is:

[0029] ,

[0030] ;

[0031] In the formula, the partial derivative of G with respect to x represents the gradient in the horizontal direction at any pixel coordinate (x, y), and the partial derivative of G with respect to y represents the gradient in the vertical direction at any pixel coordinate (x, y); H is the pixel value of that pixel.

[0032] Preferably, in step S8, the gradient increase and gradient direction formulas at pixel (x,y) are as follows:

[0033] ,

[0034] .

[0035] Preferably, in step S12, the Euclidean distance calculation formula is:

[0036] , .

[0037] Preferably, in step S12, after face matching is completed, a face recognition box appears, which is marked with the function name, the number of faces captured, and the recognition box that moves with the face. In addition, the terminal interface will also display the number of users in the face collection database and the directional gradient histogram feature description information dataset of the recognizer.

[0038] Compared with the prior art, the present invention has the following technical effects:

[0039] 1. The present invention discloses an intelligent interactive teaching method and system based on computer vision, which integrates the increasingly sophisticated computer vision technology with the traditional teaching system. Through gesture recognition, teachers can perform functions such as PowerPoint presentation, video playback, volume adjustment, writing and drawing by using different gestures; no other media are needed to achieve the operation, thus reducing costs.

[0040] 2. The intelligent interactive teaching method and system based on computer vision described in this invention uses a computer image processing vision framework to construct data streams into modular graphs. By inputting data such as video or audio streams into the graphs, the graph models constructed by each module can be used to process the data and derive the final result from the graphs. This makes the processing in the whole process smoother, and therefore developers can deploy it on different devices and platforms and reproduce its effects.

[0041] 3. The intelligent interactive teaching method and system based on computer vision described in this invention uses the function cLocx=pLocx+(x3-pLocx) / smooth to adjust the smoothness of mouse movement, preventing inconvenience caused by excessively fast hand movement and mouse movement during daily teaching activities, thereby improving mouse stability.

[0042] 4. The intelligent interactive teaching method and system based on computer vision described in this invention allows the program to log in via face recognition, capture images using a camera to obtain a dataset of facial image orientation gradient histogram feature description information, and perform face comparison through Euclidean distance calculation to accurately identify faces, thereby improving security and operability. It is widely used in mobile web terminals, so it is easy to expand, has low cost, is easy to carry or port, and is flexibly applicable to various occasions.

[0043] 5. The intelligent interactive teaching method and system based on computer vision described in this invention utilizes gesture recognition to interact with the teaching system, which can avoid cross-contact between teachers and users during the use of the teaching system, thus providing effective protection for users. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating an intelligent interactive teaching method based on computer vision as described in this invention.

[0045] Figure 2 This is a schematic diagram of the left-hand recognition result in a computer vision-based intelligent interactive teaching method and system described in this invention.

[0046] Figure 3 This is a schematic diagram of the right-hand recognition result in a computer vision-based intelligent interactive teaching method and system described in this invention.

[0047] Figure 4 This is a schematic diagram of the face recognition box result in an intelligent interactive teaching method and system based on computer vision as described in this invention;

[0048] Figure 5 This is a gradient information graph of histogram statistical units of 9 binary files used in a specific embodiment example of an intelligent interactive teaching method and system based on computer vision described in this invention.

[0049] Figure 6 This is a schematic diagram of a unit-to-block form in a computer vision-based intelligent interactive teaching method and system described in this invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present application and with reference to the accompanying drawings.

[0051] Example 1

[0052] A computer vision-based intelligent interactive teaching method includes the following steps:

[0053] S1: Preset a computer image processing vision framework, which is used to acquire camera gesture interaction video stream;

[0054] S2: Invoke the hand detection method in the computer image processing vision framework to detect the hand contour of the gesture interaction video stream, and pre-set the first parameter value as the minimum confidence value of the hand detection model;

[0055] S3: When the detection value exceeds the first parameter value, the hand detection is determined to be successful, and the 21 preset key information points of the hand are located; when the detection value does not exceed the first parameter value, the hand detection fails, and the computer image processing vision framework continues to perform hand detection on the next gesture interaction video stream.

[0056] S4: The hand key point mapping method in the computer image processing vision framework identifies the key information points of the hand and converts them into frame images in RGB format. It then connects the key information points of the hand, locates the coordinates of the key information points of the hand, defines the range of the hand (i.e., draws a rectangle), detects the coordinates of the rectangle range and its height and width, and then identifies whether the hand is the left or right hand according to the preset recognition rules.

[0057] S5: The hand state detection method in the cross-platform GUI automation toolkit of the computer image processing vision framework is called to detect hand information. After detection, a sequence of 5 elements is returned, each element corresponding to a finger. When the element is 1, it means that the finger is upright, and when the element is 0, it means that the finger is bent. When the second element in the sequence is 1 and the third element is 0, that is, the index finger is upright and the middle finger is bent, the mouse movement operation is enabled, and the mouse moves with the position of the index fingertip. When the second element in the sequence is 1 and the third element is 1, that is, the index finger and the middle finger are upright at the same time, the parameters (pt1, pt2, img) are substituted into the fingertip distance measurement method to collect hand information again, where img is the image information transmitted from the camera to the teaching computer, and pt1 and pt2 are the coordinates of the fingertips of the index finger and the middle finger, respectively. When the pixel distance between the fingertips of the index finger and the middle finger is less than 50, the mouse click operation is enabled.

[0058] Preferably, in step S5, the linear interpolation method np.interp(x,xp,fp) is used to map the width and height of the rectangle to the width and height of the computer screen, where x is the horizontal coordinate array mapped to the screen, xp is the horizontal coordinate array of the rectangle, and fp is the vertical coordinate array of the rectangle.

[0059] Preferably, the width and height dimensions of the computer screen are obtained, and then the mouse is moved to the preset coordinates.

[0060] Preferably, in step S5, during the mouse movement, the function cLocx=pLocx+(x3-p-Locx) / smooth is called to adjust the smoothness of mouse movement, where cLocx is the x-coordinate of the mouse position in the previous frame, pLocx is the x-coordinate of the mouse position in the frame before the previous frame, x3 is the x-coordinate of the current mouse position, and smooth is a customizable smoothing coefficient. By changing smooth, the smoothness of mouse movement can be adjusted.

[0061] Example 2

[0062] A computer vision-based intelligent interactive teaching system mainly includes a login verification module, a gesture teaching module, a classroom lighting control module, a classroom fan control module, a log information module, a personnel management module, a personal information module, and other modules. The gesture teaching module is used to execute a computer vision-based intelligent interactive teaching method.

[0063] The login verification module further includes a face recognition unit and an account password verification unit. The face recognition unit is configured to perform the following steps:

[0064] S6: Convert the face video stream captured by the camera to grayscale;

[0065] S7: The input image is normalized in color space using the Gamma correction method, which adjusts the contrast of the image.

[0066] S8: Calculate the gradient of each pixel in the horizontal and vertical directions of the image, and calculate the gradient direction value of each pixel position accordingly to capture contour information;

[0067] S9: Divide the image into several small units, count the gradient histogram of each unit, that is, count the number of different gradients, and the feature description information of each unit can be formed.

[0068] S10: Combine several units into a block, and concatenate the feature description information of all units in a block to obtain the directional gradient histogram feature description information of the block;

[0069] S11: Concatenate the directional gradient histogram feature description information of all blocks in the image to obtain the directional gradient histogram feature description information dataset of the image.

[0070] S12: Collect the histogram of oriented gradients feature description information dataset, classify it according to different photos, and then store it in a CSV file. Calculate the Euclidean distance of the histogram of oriented gradients feature description information data in each CSV file, and take the dataset with the smallest Euclidean distance to recognize and match each face. If the two match, the face recognition is successful and the login is completed.

[0071] Preferably, in step S6, the grayscale conversion formula is:

[0072] ;

[0073] In the formula, Gray is the grayscale value, R is the red value, G is the green value, and B is the blue value.

[0074] Preferably, in step S8, the gradient transformation formula for any pixel (x, y) is:

[0075] ,

[0076] ;

[0077] In the formula, the partial derivative of G with respect to x represents the gradient in the horizontal direction at any pixel coordinate (x, y), and the partial derivative of G with respect to y represents the gradient in the vertical direction at any pixel coordinate (x, y); H is the pixel value of that pixel.

[0078] Preferably, in step S8, the gradient increase and gradient direction formulas at pixel (x,y) are as follows:

[0079] ,

[0080] .

[0081] Preferably, in step S12, the Euclidean distance calculation formula is:

[0082] , .

[0083] Preferably, in step S12, after face matching is completed, a face recognition box appears, which is marked with the function name, the number of faces captured, and the recognition box that moves with the face. In addition, the terminal interface will also display the number of users in the face collection database and the directional gradient histogram feature description information dataset of the recognizer.

[0084] Example 3

[0085] A computer vision-based intelligent interactive teaching method includes the following steps:

[0086] S1: Using MediaPipe in the computer image processing vision framework, the data stream is built into modular graphs. Data such as video or audio streams are input into the graph, and the graph model built by each module can be used to process the data and derive the final result from the graph. This makes the processing in the whole process smoother, and therefore developers can deploy it on different devices and platforms and reproduce its effect.

[0087] S2: The hand key point detection method in the computer image processing vision framework performs hand detection on the gesture interaction video stream. The hand key point detection method uses the function cvzone.HandTrackingModule.HandDetector(), and then pre-sets the first parameter value as the minimum confidence value of the hand detection model.

[0088] S3: If the detected value exceeds the first parameter value, the hand detection is determined to be successful, and the 21 preset key information points of the hand are located; if the detected value does not exceed the first parameter value, the hand detection fails, and the computer image processing vision framework continues to perform hand detection on the next gesture interaction video stream.

[0089] S4: The hand key point mapping method in the computer image processing vision framework identifies the key information points of the hand and converts them into frame images in RGB format. The hand key point mapping method uses the function cv-zone.HandTrackingModule.HandDetector.findHands() to connect the key information points of the hand, locate the coordinates of the key information points of the hand, define the range of the hand (i.e., draw a rectangle), detect the coordinates of the rectangle range and its height and width, and then identify whether the hand is the left hand or the right hand according to the preset recognition rules.

[0090] S5: The hand state detection method in the cross-platform GUI automation toolkit of the computer image processing vision framework is called to detect hand information. This hand state detection method uses the function `detector.fingersUp()`, which returns a sequence of 5 elements, each corresponding to a finger. This simulates the five fingers of a person; a 1 indicates the finger is upright, and a 0 indicates the finger is bent. When the second element in the sequence is 1 and the third element is 0 (i.e., the index finger is upright and the middle finger is bent), the teaching computer enables mouse movement, and the mouse moves with the fingertip of the index finger. When the third element is 1, meaning the index and middle fingers are raised simultaneously, the parameters (pt1, pt2, img) are substituted into the fingertip ranging method to collect hand information again, i.e., the detector.findD-istance(pt1, pt2, img) is calculated, where img is the image information transmitted from the camera to the teaching computer, and pt1 and pt2 are the coordinates of the fingertips of the index and middle fingers, respectively. When the pixel distance between the fingertips of the index and middle fingers is less than 50, the teaching computer enables mouse click operation, and the mouse click uses the cross-platform GUI automation toolkit function autopy.mouse.click().

[0091] Preferably, in step S5, the linear interpolation method np.interp(x,xp,fp) is used to map the width and height of the rectangle to the width and height of the computer screen, where x is the array of horizontal coordinates mapped to the screen, xp is the array of horizontal coordinates of the rectangle, and fp is the array of vertical coordinates of the rectangle. For example, np.interp(x1,(pt1[0],pt2[0]),(0,wScr)) maps the range of the x coordinates from the original pt1[0] to pt1[0]+w to the entire computer screen from 0 to wScr.

[0092] Preferably, the width and height of the computer screen are obtained using the function autopy.screen.size(), and then the mouse is moved to the position at coordinates (x,y) using the function autopy.mouse.move(x,y).

[0093] Preferably, in step S5, the smoothness of mouse movement is adjusted using the function cLocx=pLocx+(x3-pLocx) / smooth during mouse movement, where cLocx is the x-coordinate of the mouse position in the previous frame, pLocx is the x-coordinate of the mouse position in the frame before the previous frame, x3 is the x-coordinate of the current mouse position, and smooth is a customizable smoothing coefficient. By changing smooth, the smoothness of mouse movement can be adjusted. The larger the smoothness, the slower the mouse movement speed and the higher the smoothness.

[0094] Example 4

[0095] A computer vision-based intelligent interactive teaching system mainly includes a login verification module, a gesture teaching module, a classroom lighting control module, a classroom fan control module, a log information module, a personnel management module, a personal information module, and other modules. The gesture teaching module is used to execute a computer vision-based intelligent interactive teaching method.

[0096] The login verification module includes a face recognition module and an account password verification module. The face recognition module includes the following steps:

[0097] S6: Convert the face video stream captured by the camera to grayscale, that is, treat the image as a three-dimensional grayscale image of (x,y,z).

[0098] S7: The input image is normalized in color space using the Gamma correction method. The purpose is to adjust the contrast of the image, reduce the impact of local shadows and lighting changes, and suppress noise interference.

[0099] S8: Calculate the gradient of each pixel in the horizontal and vertical directions of the image, and calculate the gradient direction value of each pixel position accordingly to capture contour information and further reduce the interference of lighting.

[0100] S9: Divide the image into several small units, count the gradient histogram of each unit, that is, count the number of different gradients, and the feature description information of each unit can be formed.

[0101] S10: Combine several units into a block, and concatenate the feature description information of all units in a block to obtain the directional gradient histogram feature description information of the block;

[0102] S11: Concatenate the directional gradient histogram feature description information of all blocks in the image to obtain the directional gradient histogram feature description information dataset of the image.

[0103] S12: Collect the histogram of oriented gradients feature description information dataset, classify it according to different photos, and then store it in a CSV file. Calculate the Euclidean distance of the histogram of oriented gradients feature description information data in each CSV file, and take the dataset with the smallest Euclidean distance to recognize and match each face. If the two match, the face recognition is successful and the login is completed.

[0104] Preferably, in step S6, the grayscale conversion formula is:

[0105] .

[0106] In the formula, Gray is the grayscale value, R is the red value, G is the green value, and B is the blue value.

[0107] Preferably, in step S8, the gradient transformation formula for any pixel (x, y) is: ,

[0108] ;

[0109] In the formula, the partial derivative of G with respect to x represents the gradient in the horizontal direction at any pixel coordinate (x, y), and the partial derivative of G with respect to y represents the gradient in the vertical direction at any pixel coordinate (x, y); H is the pixel value of that pixel.

[0110] Preferably, in step S8, the gradient increase and gradient direction formulas at pixel (x,y) are as follows:

[0111] ,

[0112] .

[0113] Preferably, in step S12, the Euclidean distance calculation formula is:

[0114] , .

[0115] For example, for formulas After grayscale conversion, the image undergoes Gamma correction, i.e., normalization, to adjust the image contrast. Then, the gradients along the horizontal and vertical axes are calculated. The gradient transformation formula for any pixel (x, y) in the image is:

[0116]

[0117]

[0118] In the formula, the partial derivative of G with respect to x represents the gradient in the horizontal direction at any pixel coordinate (x, y), the partial derivative of G with respect to y represents the gradient in the vertical direction at any pixel coordinate (x, y), and H is the pixel value at this point.

[0119] Based on this, the gradient direction value at each pixel position is calculated. The formulas for the gradient increase and gradient direction at pixel (x,y) are shown below:

[0120]

[0121]

[0122] The image is divided into 6*6 pixels / units, and a gradient direction histogram is constructed for each unit. When the histograms of 9 binary files are used to count the gradient information of these 6*6 pixels, the gradient direction of the unit is divided into 9 directional blocks of 360 degrees.

[0123] Then, the 3x3 units are combined into a block. The feature vectors of all units within a block are concatenated to obtain the histogram of oriented gradients (HOG) feature description data for that block. These intervals overlap, meaning that the features of each unit will appear multiple times in the final feature vector with different results. This normalized block descriptor (vector) is called the HOG descriptor, which is the histogram of oriented gradients feature description data required for face recognition—the final feature vector usable for classification.

[0124] We collect the directional gradient histogram feature description information dataset vectors obtained through the above calculation method. Our approach is to store this data in C-SV files according to different photo classification principles, and calculate the Euclidean distance between the directional gradient histogram feature description information datasets in each C-SV file. The Euclidean distance formula is as follows:

[0125]

[0126]

[0127] The dataset with the smallest Euclidean distance is used to match each face. If a match is found, the face recognition is successful, and login is complete. The face recognition bounding box is labeled with the function name, the number of faces captured, and the recognition bounding box that follows the face. In addition, the terminal interface will also display the number of users in the face collection database and the directional gradient histogram feature description information dataset of the recognizer.

[0128] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A computer vision-based intelligent interactive teaching method, characterized in that, Includes the following steps: S1: Preset a computer image processing vision framework, which is used to acquire camera gesture interaction video stream; S2: Invoke the hand detection method in the computer image processing vision framework to detect the hand contour of the gesture interaction video stream, and pre-set the first parameter value as the minimum confidence value of the hand detection model; S3: When the detection value exceeds the first parameter value, the hand detection is determined to be successful, and the 21 preset key information points of the hand are located; when the detection value does not exceed the first parameter value, the hand detection fails, and the computer image processing vision framework continues to perform hand detection on the next gesture interaction video stream. S4: The hand key point mapping method in the computer image processing vision framework identifies the key information points of the hand and converts them into frame images in RGB format. It then connects the key information points of the hand, locates the coordinates of the key information points of the hand, defines the range of the hand (i.e., draws a rectangle), detects the coordinates of the rectangle range and its height and width, and then identifies whether the hand is the left or right hand according to the preset recognition rules. S5: The hand state detection method in the cross-platform GUI automation toolkit of the computer image processing vision framework is called to detect hand information. After detection, a sequence of 5 elements is returned, each element corresponding to a finger. When the element is 1, it means that the finger is upright, and when the element is 0, it means that the finger is bent. When the second element in the sequence is 1 and the third element is 0, that is, the index finger is upright and the middle finger is bent, the mouse movement operation is enabled, and the mouse moves with the position of the index fingertip. When the second element in the sequence is 1 and the third element is 1, that is, the index finger and the middle finger are upright at the same time, the parameters (pt1, pt2, img) are substituted into the fingertip distance measurement method to collect hand information again, where img is the image information transmitted from the camera to the teaching computer, and pt1 and pt2 are the coordinates of the fingertips of the index finger and the middle finger, respectively. When the pixel distance between the fingertips of the index finger and the middle finger is less than 50, the mouse click operation is enabled.

2. The intelligent interactive teaching method based on computer vision according to claim 1, characterized in that, In step S5, the linear interpolation method np.interp(x,xp,fp) is used to map the width and height of the rectangle to the width and height of the computer screen, where x is the horizontal coordinate array mapped to the screen, xp is the horizontal coordinate array of the rectangle, and fp is the vertical coordinate array of the rectangle.

3. The intelligent interactive teaching method based on computer vision according to claim 2, characterized in that, Obtain the width and height dimensions of the computer screen, then move the mouse to the preset coordinates.

4. The intelligent interactive teaching method based on computer vision according to claim 1, characterized in that, In step S5, during mouse movement, the function cLocx=pLocx+(x3-p-Locx) / smooth is called to adjust the smoothness of mouse movement. Here, cLocx is the x-coordinate of the mouse position in the previous frame, pLocx is the x-coordinate of the mouse position in the frame before the previous frame, x3 is the x-coordinate of the current mouse position, and s-mooth is a customizable smoothness coefficient. By changing smoothness, the smoothness of mouse movement can be adjusted.

5. A computer vision-based intelligent interactive teaching system, characterized in that, It mainly includes a login verification module, a gesture teaching module, a classroom lighting control module, a classroom fan control module, a log information module, a personnel management module, a personal information module, and other modules. The gesture teaching module is used to execute a computer vision-based intelligent interactive teaching method as described in any one of claims 1-4. The login verification module further includes a face recognition unit and an account password verification unit. The face recognition unit is configured to perform the following steps: S6: Convert the face video stream captured by the camera to grayscale; S7: The input image is normalized in color space using the Gamma correction method, which adjusts the contrast of the image. S8: Calculate the gradient of each pixel in the horizontal and vertical directions of the image, and calculate the gradient direction value of each pixel position accordingly to capture contour information; S9: Divide the image into several small units, count the gradient histogram of each unit, that is, count the number of different gradients, and the feature description information of each unit can be formed. S10: Combine several units into a block, and concatenate the feature description information of all units in a block to obtain the directional gradient histogram feature description information of the block; S11: Concatenate the directional gradient histogram feature description information of all blocks in the image to obtain the directional gradient histogram feature description information dataset of the image; S12: Collect the histogram of oriented gradients feature description information dataset, classify it according to different photos, and then store it in a CSV file. Calculate the Euclidean distance of the histogram of oriented gradients feature description information data in each CSV file, and take the dataset with the smallest Euclidean distance to recognize and match each face. If the two match, the face recognition is successful and the login is completed.

6. The intelligent interactive teaching system based on computer vision according to claim 5, characterized in that, In step S6, the grayscale conversion formula is: ； In the formula, Gray is the grayscale value, R is the red value, G is the green value, and B is the blue value.

7. The intelligent interactive teaching system based on computer vision according to claim 5, characterized in that, In step S8, the gradient transformation formula for any pixel (x, y) is: ， ； In the formula, the partial derivative of G with respect to x represents the gradient in the horizontal direction at any pixel coordinate (x, y), and the partial derivative of G with respect to y represents the gradient in the vertical direction at any pixel coordinate (x, y); H is the pixel value of that pixel.

8. The intelligent interactive teaching system based on computer vision according to claim 5, characterized in that, In step S8, the gradient increase and gradient direction formulas at pixel (x,y) are as follows: ， 。 9. The intelligent interactive teaching system based on computer vision according to claim 5, characterized in that, In step S12, the Euclidean distance calculation formula is as follows: ， 。 10. The intelligent interactive teaching system based on computer vision according to claim 5, characterized in that, In step S12, after face matching is completed, a face recognition box appears, which is marked with the function name, the number of faces captured, and the recognition box that moves with the face. In addition, the terminal interface will also display the number of users in the face collection database and the directional gradient histogram feature description information dataset of the recognizer.

Images