AI graphical programming method and system based on gesture recognition

Abstract

The invention discloses an AI graphical programming method and system based on gesture recognition. The method comprises the following steps: capturing a video frame in real time through a camera, preprocessing the video frame, and extracting a hand image from the preprocessed video frame; the hand images are classified through a convolutional neural network, corresponding control instructions are generated according to classification results, and the control instructions are used for controlling a graphical programming platform to achieve corresponding programming functions. The technical problem that an existing gesture recognition technology is insufficient in the aspects of accuracy, real-time performance, platform dependence and the like is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an AI graphical programming method and system based on gesture recognition. Background Art

[0002] In recent years, with the rapid development of computer vision and human-computer interaction technology, gesture recognition, as a natural and intuitive way of interaction, has received extensive attention and research. Gesture recognition technology can be applied in many fields such as smart home control, virtual reality, and game entertainment. It can capture user gestures through sensors such as cameras to control devices. However, existing gesture recognition technology has many shortcomings in terms of accuracy, real-time performance, and platform dependence, and urgently needs to be improved and optimized.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide an AI graphical programming method and system based on gesture recognition, so as to at least solve the technical problems that the existing gesture recognition technology has deficiencies in accuracy, real-time performance, and platform dependence.

[0005] According to one aspect of an embodiment of the present invention, there is provided an AI graphical programming method based on gesture recognition, comprising: capturing video frames in real time through a camera, preprocessing the video frames, and extracting hand images from the preprocessed video frames; classifying the hand images using a convolutional neural network, and generating corresponding control instructions based on the classification results, wherein the control instructions are used to control a graphical programming platform to implement corresponding programming functions.

[0006] According to another aspect of an embodiment of the present invention, there is also provided an AI graphical programming device based on gesture recognition, including: a gesture detection and extraction module, configured to capture video frames in real time through a camera, preprocess the video frames, and extract hand images from the preprocessed video frames; a gesture classification module, configured to classify the hand images using a convolutional neural network; and a control instruction generation module, configured to generate corresponding control instructions according to the classification results, wherein the control instructions are used to control the graphical programming platform to implement corresponding programming functions.

[0007] In the embodiment of the present invention, a video frame is captured in real time by a camera, the video frame is preprocessed, and a hand image is extracted from the preprocessed video frame; a convolutional neural network is used to classify the hand image, and a corresponding control instruction is generated according to the classification result, wherein the control instruction is used to control the graphical programming platform to implement the corresponding programming function. The above scheme solves the technical problems of the existing gesture recognition technology in terms of accuracy, real-time performance and platform dependence. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0009] Figure 1 is a flow chart of a graphical programming method based on gesture recognition according to an embodiment of the present invention;

[0010] Figure 2 is an architectural diagram of a gesture recognition system in a graphical programming platform based on a Web browser according to an embodiment of the present invention;

[0011] Figure 3 is a workflow diagram of a graphical programming platform based on a Web browser according to an embodiment of the present invention;

[0012] Figure 4 is a flow chart of a method for extracting a hand region according to an embodiment of the present invention;

[0013] Figure 5 4 is a structural diagram of a graphical programming device based on gesture recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0016] According to an embodiment of the present invention, a method embodiment of a graphical programming method based on gesture recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0017] Figure 1 is a graphical programming method based on gesture recognition according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0018] Step S102: capturing video frames in real time through a camera, preprocessing the video frames, and extracting hand images from the preprocessed video frames.

[0019] First, the video frame is captured in real time by a camera, and the video frame is preprocessed. For example, the video frame is grayed to obtain a gray image, and the gray image is smoothed by Gaussian blur to obtain a smoothed image; the smoothed image is binarized to obtain a binary image, and the binary image is used as the preprocessed video frame. The present application effectively removes noise and unnecessary details in the image and improves the clarity and quality of the image by preprocessing the video frame captured in real time, including graying, Gaussian blur smoothing and binarization. The preprocessed binary image can highlight the area of ​​interest, enhance the detection and extraction capabilities of key features, and provide more accurate and stable input data for subsequent gesture recognition and image analysis, thereby improving the recognition accuracy and processing efficiency of the entire system.

[0020] Secondly, extract the hand image from the preprocessed video frame. For example, use a skin color detection algorithm to extract the hand area from the preprocessed video frame, and use morphological operations to remove the noise in the hand area to obtain an enhanced hand area; use an edge detection method to extract the hand contour from the enhanced hand area, and normalize the hand contour to obtain the normalized hand contour as the hand image. The above steps improve the accuracy and standardization of the hand image, provide more reliable and consistent basic data for subsequent gesture recognition and classification, and thus enhance the overall performance and robustness of the system.

[0021] In some embodiments, a skin color detection algorithm is used to extract a hand region from a preprocessed video frame, including: using a skin color detection algorithm to convert the preprocessed video frame from an RGB image to a color space based on brightness and chroma; in the color space based on brightness and chroma, a skin color region is segmented based on a preset threshold to obtain the hand region. The present application accurately locates and separates the hand region by performing skin color region segmentation based on a preset threshold in the color space, thereby improving the extraction accuracy of the hand region, reducing background interference, making subsequent gesture recognition and analysis more accurate and efficient, and thereby improving the performance and stability of the system in complex environments.

[0022] In some embodiments, a morphological operation is used to remove noise from the hand region to obtain an enhanced hand region, including: performing an erosion operation on the hand region to remove noise points in the hand region; performing an expansion operation on the hand region after the noise points are removed to restore the structure of the hand or connect the parts disconnected by the erosion operation. The above steps effectively enhance the clarity and accuracy of the hand region, provide more accurate and reliable input data for subsequent gesture recognition, and thus improve the overall performance of the system.

[0023] Step S104, using a convolutional neural network to classify the hand image, and generating corresponding control instructions according to the classification results, wherein the control instructions are used to control the graphical programming platform to implement corresponding programming functions.

[0024] First, the hand image is classified using a convolutional neural network. For example, the convolutional layer of the convolutional neural network is used to perform a convolution operation on the hand image, and features are extracted from the hand image to obtain a feature map; the pooling layer of the convolutional neural network is used to downsample the features in the feature map to obtain a reduced-dimensional feature map; the fully connected layer of the convolutional neural network is used to map the reduced-dimensional feature map to a category space to classify the hand image and obtain a specific classification result. The classification result usually indicates a specific gesture, such as "waving", "clenching a fist" or "pointing with an index finger".

[0025] Secondly, the corresponding control instructions are generated according to the classification results. For example, the classification results of the gestures are mapped to the predefined gesture action list in the system, and each gesture action corresponds to a specific function or instruction. The classification results are mapped with the predefined gesture action list. For example, assuming that the classification result is "waving", the system will look for the control instruction corresponding to "waving". If the "waving" gesture is defined as the trigger condition of "page switching", the system will generate a control instruction for switching the current page. The control instruction is represented in the form of a data packet, which contains the type of instruction and the parameters for execution. For graphical programming platforms, these instructions can be in JSON format to ensure the standardization and compatibility of data transmission.

[0026] The above-mentioned graphical programming method based on gesture recognition uses a camera to capture video frames in real time, combined with skin color detection algorithms, morphological operations, edge detection, and convolutional neural network classification, to achieve accurate recognition and real-time response to user gestures. This solution effectively removes noise from the image, ensures accurate positioning and contour extraction of the hand area, and improves the accuracy of gesture recognition. The control instructions finally generated can be seamlessly integrated with the graphical programming platform to achieve intuitive user operation, significantly improve the interactive experience and ease of use of the system, while reducing dependence on traditional input devices, and are suitable for a variety of application scenarios.

[0027] Currently, most gesture recognition systems have performance problems in complex environments, especially in the case of light changes and background interference, the recognition accuracy is low. In addition, some gesture recognition algorithms have high computational complexity and are difficult to meet real-time requirements, thus affecting the user experience. At the same time, many systems are highly dependent on specific hardware platforms and lack cross-platform compatibility and flexibility.

[0028] In order to solve the above problems, an embodiment of the present invention provides a gesture recognition system in a graphical programming platform based on a Web browser. The system captures the user's gestures through an integrated camera and uses a convolutional neural network (CNN) to classify gestures, thereby achieving control of Web applications. Figure 2 As shown, the system includes a Web camera module 12 , a gesture detection and extraction module 14 , a gesture classification module 16 , a control instruction generation module 18 and a WEB application 20 .

[0029] The Web camera module calls the browser's camera interface through JavaScript to obtain the video stream, uses the tag in the HTML5 standard to display the real-time video stream, and obtains user permission to access the camera through the API.

[0030] The gesture detection and extraction module first uses image processing libraries such as OpenCV to preprocess the video frames, including grayscale, binarization, and contour extraction. Then, a skin color detection algorithm (such as threshold segmentation in YCrCb color space) is used to preliminarily extract the hand area. Next, morphological operations (such as corrosion and dilation) are applied to remove noise and enhance the hand contour. Finally, the hand area is normalized to adapt it to the input size of the CNN model.

[0031] The gesture classification module uses a convolutional neural network (CNN) to classify the extracted hand images. The CNN model consists of multiple convolutional layers, pooling layers, and fully connected layers to extract image features and perform classification.

[0032] Specifically, first, the convolution layer performs a convolution operation on the input image (I) and the convolution kernel (K) using the convolution operation formula. The convolution operation formula is:

[0033] Among them, I is the input image, K is the convolution kernel, and (I*K)(i,j) represents the value of the output image at position (i,j) after convolution. It means to sum each element K(m,n) in the convolution kernel K. I(im,jn) is the pixel value corresponding to the convolution kernel element K(m,n) in the input image.

[0034] Next, the pooling layer performs pooling operations. For the maximum pooling operation with a pooling window size of (p\times q), the value at the output position ((i,j)) is: O(i,j)=max 0≤m<p,0≤n<q I(i+m,j+n), where O(i,j) is the output image value after pooling and I(i+m,j+n) is the input image value within the pooling window.

[0035] Finally, the fully connected layer performs classification. The fully connected layer can integrate the features extracted by the convolutional layer and the pooling layer and output the final classification result. Specifically, the fully connected layer combines local features into global information, allowing the network to learn the complex relationship between categories. Then, through activation functions and linear transformations, feature learning is converted into clear classification decisions, and the output image belongs to the probability distribution of each category.

[0036] The control instruction generation module generates corresponding control instructions according to the classification results and sends them to the Web application through WebSocket or HTTP request. The control instructions are encapsulated in JSON format to ensure the standardization and compatibility of data transmission.

[0037] Web applications receive and parse control instructions to implement corresponding functions, such as page switching, element operations, etc., and use JavaScript event mechanisms to trigger corresponding DOM operations to achieve user interaction.

[0038] This application is based on the real-time gesture recognition system architecture design of the Web browser, which can realize real-time gesture recognition and control without relying on a specific hardware platform. The system uses convolutional neural network (CNN) to extract and classify hand images, thereby significantly improving the recognition accuracy and robustness. In addition, the system integrates gesture recognition function into the graphical programming platform in the Web browser, allowing users to realize complex human-computer interaction functions through simple drag and drop operations.

[0039] The following is a detailed description of the workflow of the Web browser-based graphical programming platform. Figure 3 As shown, the following steps are included:

[0040] Step S302, initializing the camera.

[0041] First, perform the initialization operation of the camera. Use JavaScript to call the browser's camera interface and request the user's permission to access the camera. After the user authorizes, the platform uses the HTML5 standard <video>The tag displays the real-time video stream, allowing users to see the image from the camera directly on the Web page.

[0042] After successfully obtaining permission and opening the camera, the platform sets the relevant parameters of the video stream, including resolution and frame rate, to ensure that the quality of the video stream can meet the requirements of subsequent image processing and gesture recognition. The platform also designs a corresponding error handling mechanism to deal with initialization failures. For example, when the user refuses authorization or the device does not support the camera, the system will trigger the error handling process, display an error prompt to the user, and may re-request permission or close the application.

[0043] Step S304, capturing video frames.

[0044] After the camera is initialized, the platform will start to continuously capture video frames and transmit them to the subsequent gesture detection and extraction module in real time.

[0045] Step S306, pre-processing the image.

[0046] The image noise is removed by Gaussian blur smoothing to improve the accuracy of edge detection. The generated Gaussian kernel is used to calculate the displacement value of each pixel in the image in combination with the spatial coordinates of each pixel and its gradient information in the horizontal and vertical directions. The displacement value is used to represent the grayscale change and texture characteristics of the image in the local area. In addition, in some embodiments, a noise intensity parameter can also be introduced as a regulating factor to control the sensitivity to image changes in the denoising process. Specifically, the width of the Gaussian distribution can be determined according to the standard deviation, and the center of the Gaussian kernel is constructed using the width, the coordinates of the pixel relative to the center of the kernel, and the size of the Gaussian kernel. Then, denoising is performed based on the pixel value of the original image at position (i, j), the horizontal and vertical offset relative to the center of the Gaussian kernel, the horizontal and vertical gradients of each pixel position, the gradient weight, and the noise intensity parameter used to adjust the sensitivity in the denoising process. In this process, the noise intensity parameter is determined according to the noise intensity, the pixel value of the image at position (i, j), and the change value of the pixel value after the offset. By using the generated Gaussian kernel and calculating the displacement value in combination with the spatial position and gradient information of each pixel, the local grayscale change and texture details of the image can be effectively reflected. At the same time, the introduction of the noise intensity parameter makes the denoising process more adaptive, which can effectively remove noise while retaining the detailed features of the image, thereby improving the quality and accuracy of image processing. Specifically, the Gaussian blur formula is:

[0047]

[0048] Among them, G(x,y) represents the position coordinates after Gaussian blur, σ 2 is the standard deviation of the Gaussian kernel.

[0049] Step S308, extracting the hand area.

[0050] like Figure 4 As shown, extracting the hand region includes the following steps:

[0051] Step S3082, skin color detection.

[0052] In the embodiment of the present invention, in order to accurately extract the user's hand area, a skin color detection algorithm based on color space conversion is used. Skin color detection is a key step in the gesture recognition process. Its purpose is to segment the skin area that may contain the hand from the video frame so that the subsequent morphological operation and edge detection can focus on the target area more accurately. The following are the detailed steps of the skin color detection algorithm:

[0053] 1) Color space conversion.

[0054] This application first converts the input RGB image to the YCrCb color space. The RGB (red, green, and blue) color space is a common color space for display devices, but for skin color detection, there is a large correlation between the three channels of the RGB space, making the segmentation of the skin color area unstable. The YCrCb color space consists of brightness (Y) and two chromaticity components (Cr and Cb), which can better separate the chromaticity and brightness information, thereby more stably segmenting the skin color area.

[0055] The conversion formula is as follows:

[0056] Y = 0.299R + 0.587G + 0.114B;

[0057] Cr = R–Y;

[0058] Cb = B–Y;

[0059] Among them, R, G, B represent the red, green, and blue channel values ​​in the image, Y represents the brightness component, and Cr and Cb represent the two chromaticity components. Through the above conversion formula, the image is converted from RGB space to YCrCb space to obtain three new channels: brightness Y, chromaticity red difference Cr, and chromaticity blue difference Cb.

[0060] 2) Threshold segmentation.

[0061] After completing the color space conversion, the algorithm applies a preset threshold range to the Cr and Cb channels of the YCrCb image to segment the skin color area. Typical skin colors will appear in a certain range on the Cr and Cb channels, for example:

[0062] Cr min ≤Cr≤Cr max

[0063] Cb min ≤Cb≤Cb max

[0064] By selecting these threshold ranges, the system can filter out pixels that meet the conditions and mark them as skin color areas. All pixels outside this range will be considered not to belong to the skin and will be excluded from the skin color area.

[0065] 3) Optimization and enhancement.

[0066] The preliminary results of skin color detection may contain noise and non-hand skin color areas. To improve the accuracy of detection, the system can apply some additional image processing techniques, such as using median filtering to smooth the image and reduce the impact of noise on the detection results. At the same time, the detection stability and continuity of the hand area can be enhanced by combining the detection results of multiple frames (inter-frame information).

[0067] Through the above steps, the skin color detection algorithm effectively segments the hand area, providing a clear initial input for subsequent morphological operations and edge detection. This skin color detection method based on color space conversion and threshold segmentation has the advantages of simple calculation, strong real-time performance, and stable performance under various lighting conditions. It is suitable for application in real-time gesture recognition systems based on Web browsers.

[0068] Step S3084, morphological operation.

[0069] Apply erosion and dilation operations to remove noise. The erosion operation is used to remove small noise points, while the dilation operation restores the object structure. The erosion formula is: The expansion formula is: Among them, I is the input image and K is the structural element.

[0070] Step S3086, edge detection.

[0071] Use the Canny edge detection algorithm to accurately detect the hand contour. The Canny edge detection steps include:

[0072] 1) Gaussian blur.

[0073] Apply a Gaussian blur to the input image. Gaussian blur reduces noise and fine details by smoothing the image, which can prevent them from being mistaken for edges. Gaussian blur is achieved by using a Gaussian kernel function, which assigns weights based on the distance of pixels. The farther the distance, the lower the weight, thus eliminating most of the random noise.

[0074] 2) Calculate the gradient.

[0075] Calculate the gradient strength and direction of each pixel in the image. The gradient represents the direction and magnitude of the grayscale change in the image. By calculating the horizontal and vertical gradients of the image, find the area with the most significant change in the image, that is, the potential edge. The greater the intensity, the more obvious the change, and the more likely it is an edge.

[0076] 3) Non-maximum suppression.

[0077] Check each pixel and compare its gradient strength with the two adjacent pixels in the gradient direction. If the gradient strength of the current pixel is not the largest of the three, it is considered not to be an edge and the strength value will be set to zero. In this way, the width of the edge can be reduced, making the edge clearer and more precise.

[0078] 4) Dual threshold detection.

[0079] Two thresholds are used to distinguish strong edges, weak edges, and non-edges. Pixels with gradient strengths higher than the high threshold are considered strong edges, i.e., true edges. Pixels with gradient strengths between the low and high thresholds are considered weak edges, which may be actual edges or noise. Pixels with gradient strengths lower than the low threshold are considered non-edges and are directly ignored. Through dual threshold detection, the algorithm can effectively separate actual edges and noise in the image.

[0080] 5) Edge connection.

[0081] Starting from the strong edge pixel, connect all the weak edge pixels connected to it. If a weak edge pixel is connected to a strong edge pixel, it is also considered an edge. If it is not connected to a strong edge pixel, it is ignored. Through the above method, the actual edge can be accurately identified and false detection can be reduced.

[0082] Step S310, classifying gestures.

[0083] The system analyzes and classifies the hand regions extracted from the image to identify the user's specific gestures. Pre-trained machine learning models can be used, especially deep learning algorithms such as convolutional neural networks (CNNs). Deep learning models are trained with a large amount of gesture data and are able to recognize different gestures from images, such as waving, tapping, zooming, etc.

[0084] Specifically, the system inputs the preprocessed hand image into a deep learning-based classification model. The model extracts and analyzes the features of the image through a multi-layer neural network. These features may include the shape of the hand, the relative position of the fingers, the direction of the palm, etc. Then, the model outputs a classification result based on the learned pattern, indicating the most likely gesture category. For example, if the user makes a waving gesture, the model will output the "waving" category. The classification result usually includes a confidence score to indicate the accuracy of the gesture recognition result. When training the classification model, the gradient is first scrambled based on the noise term to obtain the gradient component after adding noise; then, based on the gradient component after adding noise, the learning rate, and the model parameters at the current moment, the model parameters at the next moment are updated; after the model training is completed, the model parameters are finally injected with noise based on the variance of the model parameter noise to protect the privacy of the training data.

[0085] Step S312, generating a control instruction.

[0086] Generate corresponding control instructions based on the classification results. Based on different gesture classification results, the system will generate different control instructions to drive the graphical code. For example, in the graphical coding platform, users can drag the "gesture recognition" instruction building block to the code area and combine other building block instructions to generate control instructions: "If gesture V is recognized, the character rotates 30 degrees clockwise."

[0087] Step S314, execute the instruction.

[0088] According to the generated control instructions, the action of the character in the stage area is controlled, for example, the character is controlled to rotate 30 degrees clockwise.

[0089] This system has good real-time performance by optimizing algorithms and using efficient image processing technology, which can meet real-time requirements and provide users with a smooth interactive experience. At the same time, the system is implemented based on a Web browser, does not rely on a specific hardware platform, has excellent cross-platform compatibility, and is suitable for a variety of devices and operating systems. In addition, the system integrates a graphical programming platform, so users can achieve gesture control without writing complex codes, which greatly reduces the threshold for use and improves ease of use.

[0090] The present application also provides a graphical programming device based on gesture recognition, such as Figure 5 As shown, it includes: a processing module 52, which is configured to capture video frames in real time through a camera, pre-process the video frames, and extract hand images from the pre-processed video frames; a classification module 54, which is configured to classify the hand images using a convolutional neural network; a generation module 56, which is configured to generate corresponding control instructions according to the classification results, wherein the control instructions are used to control the graphical programming platform to implement corresponding programming functions

[0091] It should be noted that the graphical programming device based on gesture recognition provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the graphical programming device based on gesture recognition provided in the above embodiment and the graphical programming method embodiment based on gesture recognition belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0092] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.< / video>

Claims

1. An AI graphical programming method based on gesture recognition, characterized in that: include: Capturing video frames in real time through a camera, preprocessing the video frames, and extracting hand images from the preprocessed video frames; The hand image is classified using a convolutional neural network, and corresponding control instructions are generated according to the classification results, wherein the control instructions are used to control the graphical programming platform to implement corresponding programming functions.

2. The method according to claim 1, characterized in that Preprocessing the video frame includes: Performing grayscale processing on the video frame to obtain a grayscale image, and performing Gaussian blur smoothing processing on the grayscale image to obtain a smoothed image; The smoothed image is binarized to obtain a binary image, and the binary image is used as the preprocessed video frame.

3. The method according to claim 1, characterized in that Extract hand images from preprocessed video frames, including: A hand region is extracted from the preprocessed video frame using a skin color detection algorithm, and a morphological operation is used to remove noise from the hand region to obtain an enhanced hand region; The hand contour is extracted from the enhanced hand region using an edge detection method, and the hand contour is normalized to obtain the normalized hand contour as the hand image.

4. The method according to claim 3, characterized in that Extract the hand region from the preprocessed video frames using a skin color detection algorithm, including: Converting the preprocessed video frame from an RGB image to a color space based on brightness and chrominance using a skin color detection algorithm; In the color space based on brightness and chromaticity, a skin color area is segmented based on a preset threshold to obtain the hand area.

5. The method according to claim 3, characterized in that: A morphological operation is used to remove noise from the hand region to obtain an enhanced hand region, including: Performing an erosion operation on the hand region to remove noise points in the hand region; The hand region after noise points are removed is expanded to restore the structure of the hand or connect the parts disconnected by the corrosion operation.

6. The method according to claim 1, characterized in that The hand image is classified using a convolutional neural network, including: Using the convolution layer of the convolutional neural network to perform a convolution operation on the hand image, extracting features from the hand image to obtain a feature map; Down-sampling the features in the feature map using the pooling layer of the convolutional neural network to obtain a feature map after dimensionality reduction; The fully connected layer of the convolutional neural network is used to map the reduced-dimensional feature map to a category space to classify the hand image.

7. An AI graphical programming device based on gesture recognition, characterized in that: include: A gesture detection and extraction module is configured to capture video frames in real time through a camera, pre-process the video frames, and extract hand images from the pre-processed video frames; a gesture classification module configured to classify the hand image using a convolutional neural network; The control instruction generation module is configured to generate corresponding control instructions according to the classification results, wherein the control instructions are used to control the graphical programming platform to implement corresponding programming functions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images