Method for imitating non-heritage handicraft based on visual gesture recognition and contrastive learning

Abstract

The embodiment of the application relates to the technical field of machine vision, and discloses a method for imitating non-heritage handicraft through gesture recognition and contrast learning based on vision, which comprises the following steps: generating a training set based on collected videos of non-heritage handicraft manufacturing processes, and performing label annotation on training samples; inputting the training samples into an initial model, iteratively training the initial model based on labels and a preset loss function until convergence, and obtaining a trained visual gesture recognition model; inputting a user's practice video into the visual gesture recognition model, so that the visual gesture recognition model can refer to a teaching video to perform gesture recognition and contrast, and obtain a discrimination result of the user's gesture action; if the discrimination result is not standard, generating a correction suggestion based on the gap between the user's gesture action and a standard gesture action in the teaching video, and feeding back the correction suggestion to the user, so that the efficiency of learning non-heritage handicraft of the learner is effectively improved, and an outstanding contribution is made for protecting and inheriting intangible cultural heritage.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a method for learning intangible cultural heritage handicrafts based on vision-based gesture recognition, imitation, and comparison. Background Technology

[0002] Among intangible cultural heritage items, intangible cultural heritage handicrafts boast a long history of inheritance, unique techniques, and local characteristics. Their production often involves multiple complex steps and meticulous manual operations, demanding that artisans possess superb skills and experience, as well as an extreme pursuit of detail. Strict quality control is required at every stage, from material selection and processing to the final product. One characteristic of intangible cultural heritage handicrafts is their intricate procedures. These crafts typically require artisans to undergo extensive learning and practice, mastering various complex techniques and methods. Each step demands careful execution, thus requiring a significant investment of time and energy. Most of the time, the transmission of intangible cultural heritage handicrafts relies primarily on oral and hands-on instruction. However, most inheritors are scattered in rural and mountainous areas, making it difficult for them to encounter young people in real life, thus hindering young people's enthusiasm for learning intangible cultural heritage handicrafts. Summary of the Invention

[0003] The purpose of this application is to provide a method for learning intangible cultural heritage handicrafts based on visual gesture recognition and comparative learning. It uses visual gesture recognition technology and comparative learning technology as support, breaks the time and space limitations between intangible cultural heritage inheritors and learners, effectively improves the learning efficiency and learning effect of learners on intangible cultural heritage handicrafts, and makes an outstanding contribution to the protection and inheritance of intangible cultural heritage.

[0004] To address the aforementioned technical problems, embodiments of this application provide a vision-based method for gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts. The method includes the following steps: generating a training set based on high-definition videos of several intangible cultural heritage handicraft production processes, and labeling the training samples in the training set; wherein the labeled tags include at least hand shape, hand size, and hand texture, whereby the hand shape characterizes the hand's position and gesture, the hand size characterizes the force and amplitude of the gesture, and the hand texture characterizes the contact and interaction between the hand and the material; and inputting the training samples from the training set into an RNN-based network. An initial model (constructed by a recurrent neural network) is iteratively trained until convergence based on the labels and a preset loss function to obtain a trained visual gesture recognition model. The user's practice video is input into the visual gesture recognition model, which then performs gesture recognition and comparison with the instructional video, obtaining the model's output judgment result on the user's gesture actions. The judgment result is either standard or non-standard. If the judgment result is non-standard, a correction suggestion is generated based on the difference between the user's gesture actions and the standard gesture actions in the instructional video, and fed back to the user.

[0005] This application also provides a vision-based gesture recognition and imitation contrast learning system for intangible cultural heritage handicrafts. The system includes: a training sample collection module, a labeling module, a model building module, a model training module, a contrast recognition module, and a feedback module. The training sample collection module is used to collect several high-definition videos of the production process of intangible cultural heritage handicrafts to generate a training set. The labeling module is used to label the training samples in the training set. The labeled tags include at least hand shape, hand size, and hand texture. The hand shape represents the position and gesture of the hand, the hand size represents the force and amplitude of the gesture, and the hand texture represents the contact and interaction between the hand and the material. The model building module is used to construct an initial model based on an RNN network. The model training module is used to input training samples from the training set into the initial model, and iteratively train the initial model until convergence based on the labels and a preset loss function to obtain a trained visual gesture recognition model; the comparison and recognition module is used to input the user's practice video into the visual gesture recognition model, so that the visual gesture recognition model can refer to the teaching video to perform gesture recognition and comparison, and obtain the judgment result of the visual gesture recognition model on the user's gesture action; wherein, the judgment result is standard or non-standard; the feedback module is used to generate correction suggestions based on the difference between the user's gesture action and the standard gesture action in the teaching video when the judgment result of the user's gesture action is determined to be non-standard, and feed them back to the user.

[0006] Embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for vision-based gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts.

[0007] Embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for vision-based gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts.

[0008] The embodiments of this application provide a vision-based gesture recognition and imitation comparison learning method for intangible cultural heritage handicrafts. Based on machine vision and deep learning technologies, it sequentially performs video capture of the intangible cultural heritage handicraft production process, feature extraction and labeling, construction and training of a visual gesture recognition model, user-recorded practice videos for gesture recognition and comparison, and correction feedback for non-standard gestures. The labeling process must at least label the hand shape representing the hand's position and gesture, the hand size representing the force and amplitude of the gesture, and the hand texture representing the contact and interaction between the hand and the material, enabling the visual gesture recognition model to fully learn the details of the intangible cultural heritage handicraft production process. During the learning process, users input recorded videos of their intangible cultural heritage (ICH) handcraft making process into a visual gesture recognition model. The model then judges the accuracy of the user's gestures. For inaccurate gestures, a feedback correction mechanism generates suggestions based on the discrepancy between the user's gestures and the standard gestures in the instructional video. This allows users to understand the problems they encounter during the process and gradually master the skills of ICH handcraft making through continuous practice and correction. This approach breaks down the time and space limitations between ICH inheritors and learners, effectively improving the efficiency and effectiveness of learners in learning ICH handcrafts, and making an outstanding contribution to the protection and inheritance of intangible cultural heritage.

[0009] In some optional embodiments, the step of generating a training set based on high-definition videos of several intangible cultural heritage handicraft production processes and labeling the training samples in the training set includes: generating a training set using the collected high-definition videos of intangible cultural heritage handicraft production processes as training samples, and segmenting the training samples into frames to obtain several frame images corresponding to the training samples; using a preset feature extraction algorithm to extract features of the hand shape, hand size, and hand texture from the frame images to obtain feature vectors of the frame images and the position coordinates of the hand in the frame images; and labeling the training samples according to the feature vectors, hand position coordinates, and timestamps corresponding to each frame image to obtain the labels corresponding to the training samples. Gestures are a continuous process, so when labeling, the training samples need to be segmented into frames. Feature extraction and labeling are performed on each frame image. Finally, a label group is formed for the training samples, which includes feature vectors, hand position coordinates and timestamps. Such a label group can help the visual gesture recognition model learn the production steps or techniques related to each feature, thereby more accurately understanding the gestures in the production process of intangible cultural heritage handicrafts.

[0010] In some optional embodiments, the training samples in the training set are input into an initial model built based on an RNN network, and the initial model is iteratively trained until convergence based on the labels and a preset loss function to obtain a trained visual gesture recognition model, which is expressed by the following formula: ; in, This represents the model parameters of the initial model before this iteration of training. This represents the operation of finding the minimum value. This represents the total number of training samples input into the initial model. Indicates the first training samples, Indicates the first Labels of each training sample This indicates that the initial model is related to the first... The predicted output of each training sample. This represents the preset loss function. This represents the preset regularization coefficient. Represents the regularization function. This represents the optimal model parameters, i.e., the model parameters of the trained visual gesture recognition model.

[0011] In some optional embodiments, after receiving a user's practice video, the visual gesture recognition model performs gesture recognition and comparison with a teaching video through the following steps, and outputs a judgment result of the user's gesture actions: Features are extracted and classified from the user's practice video to determine the gesture actions in the practice video; a practice feature sequence is generated based on the feature vectors of the gesture actions in the practice video in chronological order, and a distance matrix is ​​calculated between the practice feature sequence and the teaching feature sequence; wherein, the teaching feature sequence is pre-generated by the visual gesture recognition model based on the teaching video; using a dynamic time warping algorithm, based on preset boundary conditions, the optimal path from the upper left corner to the lower right corner is planned in the distance matrix to obtain the path matrix corresponding to the distance matrix; based on the path matrix, the similarity between the user's gesture actions and the standard gesture actions in the teaching video is calculated; if the similarity is greater than a preset threshold, the judgment result of the user's gesture actions is determined to be standard and output; otherwise, the judgment result of the user's gesture actions is determined to be non-standard and output. This application is based on the idea of ​​dynamic time warping algorithm to compare the difference between the user's gesture and standard gesture, thereby identifying the user's non-standard gesture and providing correction suggestions to the user. Dynamic time warping algorithm can effectively measure the similarity between two time series and make up for the difference in length between time series, and find the optimal alignment.

[0012] In some optional embodiments, the practice feature sequence and the teaching feature sequence are expressed by the following formulas: ; ; in, This represents the sequence of practical features. Indicates the length of the practical feature sequence. Represents the first in the sequence of practical features 1 eigenvector This represents the teaching feature sequence. Indicates the length of the teaching feature sequence. Represents the first in the teaching feature sequence 1 eigenvector; The distance matrix between the practice feature sequence and the teaching feature sequence is calculated using the following formula: ; in, Represents the first in the sequence of practical features The th eigenvector of the th feature vector 1 eigenvalue, Represents the first in the teaching feature sequence The th eigenvector of the th feature vector 1 eigenvalue, Represents the distance matrix in which the first... Line 1 The distance values ​​of the columns, the size of the distance matrix is .

[0013] In some optional embodiments, the optimal path from the top left corner to the bottom right corner is planned step-by-step in the distance matrix using the following formula, based on preset boundary conditions, to obtain the path matrix corresponding to the distance matrix: ; , ; ; ; ; in, , and This represents the preset boundary conditions. Represents the path matrix of the first... Line 1 Column elements, This indicates the operation of finding the minimum value.

[0014] In some optional embodiments, the similarity between the user's gesture and the standard gesture in the instructional video is calculated based on the path matrix using the following formula: ; in, This represents the length of the optimal path. Represents the path matrix of the first... Line 1 The elements of the column correspond to the distance values ​​in the distance matrix. This represents the summation operation. This indicates the similarity between the user's gesture and the standard gesture in the instructional video. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0016] Figure 1 This is a flowchart of a vision-based gesture recognition, imitation, and comparative learning method for intangible cultural heritage handicrafts, provided in one embodiment of this application. Figure 2 This is a flowchart provided in one embodiment of the present application, which describes how to generate a training set based on high-definition videos of several intangible cultural heritage handicraft production processes, and how to label the training samples in the training set. Figure 3 This is a schematic diagram of a gesture recognition method provided in one embodiment of this application. Figure 1 ; Figure 4 This is a schematic diagram of a gesture recognition method provided in one embodiment of this application. Figure 2 ; Figure 5 This is a schematic diagram of a gesture recognition method provided in one embodiment of this application. Figure 3 ; Figure 6 This is a schematic diagram of a gesture recognition method provided in one embodiment of this application. Figure 4 ; Figure 7 This is a flowchart provided in one embodiment of the present application, in which a visual gesture recognition model performs gesture recognition and comparison with reference to a teaching video, and outputs the judgment result of the user's gesture action; Figure 8 This is a schematic diagram of the structure of a vision-based gesture recognition, imitation, and contrast learning system for intangible cultural heritage handicrafts, provided in another embodiment of this application. Figure 9 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0018] One embodiment of this application relates to a method for learning intangible cultural heritage handicrafts through visual gesture recognition and comparison, applied to an electronic device. The electronic device can be a terminal or a server; in this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the method for learning intangible cultural heritage handicrafts through visual gesture recognition and comparison in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0019] The specific process of the vision-based gesture recognition, imitation, and comparative learning method for intangible cultural heritage handicrafts proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Generate a training set based on the collected high-definition videos of several intangible cultural heritage handicraft production processes, and label the training samples in the training set.

[0020] In its implementation, the server can crawl and collect a massive amount of high-definition videos of the production process of intangible cultural heritage handicrafts from self-media platforms and short video platforms to generate a training set, and then label the training samples in the training set. The high-definition videos of the production process of intangible cultural heritage handicrafts collected by the server include the production process of various intangible cultural heritage handicrafts such as velvet flowers, embroidery, and paper cutting. These videos not only demonstrate the production process of intangible cultural heritage handicrafts, but also capture every delicate movement and unique skill of the inheritors of intangible cultural heritage handicrafts.

[0021] In practical implementation, the labeled information includes at least hand shape, hand size, and hand texture. Hand shape represents the hand's position and gestures, including the degree of finger flexion, the spacing between fingers, and their relative positions. This information helps the model accurately recognize the shapes of different gestures, such as splitting threads, combing, rubbing, and tapping. Hand size represents the force and amplitude of the gestures. The essence of hand size is the relative proportion of the hand, which helps the model accurately understand the force and amplitude of the gestures. For example, in the process of making velvet flowers, the hand size is larger when pressing the petals than when unfolding them. This change in hand size is crucial to the techniques of velvet flower making. Hand textures are used to characterize the contact and interaction between the hand and materials. When making velvet flowers, the hand will come into contact with yarn, scissors, tweezers, glass beads, splints or other materials. The texture and feel of these materials are different from the texture of the skin on the hand. The maker will use their fingers to pick up the yarn and perform actions such as twisting or pressing. These actions will cause different texture changes on the skin of the hand, such as friction, folding or indentation. By annotating the hand texture, the model can better identify and understand the contact and interaction between the hand and materials, such as capturing the contact and manipulation between the hand and the yarn.

[0022] In one example, the server generates a training set based on high-definition videos of several intangible cultural heritage handicraft production processes, and labels the training samples in the training set. This can be done through methods such as... Figure 2 The steps shown are implemented as follows: Step 1011: The collected high-definition videos of the intangible cultural heritage handicraft production process are used as training samples to generate a training set, and the training samples are segmented into frames to obtain several frame images corresponding to the training samples.

[0023] In the specific implementation, the server uses the collected high-definition videos of the intangible cultural heritage handicraft production process as training samples to generate a training set, and then segments the training samples frame by frame to obtain several frame images corresponding to the training samples. Assume that a total of [number missing] high-definition videos of the intangible cultural heritage handicraft production process are collected in the training set. The training sample, the first training sample in the training set The training samples were divided into The first frame image, the... Each frame image can be denoted as .

[0024] Step 1012: Using a preset feature extraction algorithm, extract features of hand shape, hand size and hand texture from the frame image to obtain the feature vector of the frame image and the position coordinates of the hand in the frame image.

[0025] In the specific implementation, after the server completes the segmentation of the training samples, it can use a preset feature extraction algorithm to extract features about hand shape, hand size, and hand texture from the frame image, obtaining the feature vector of the frame image and the position coordinates of the hand in the frame image. Among them, the preset feature extraction algorithms include an edge detection algorithm (suitable for feature extraction about hand shape), a principal component analysis algorithm (suitable for feature extraction about hand size), and an oriented gradient histogram algorithm (suitable for texture extraction about hand size).

[0026] In one example, the feature extraction algorithm can be used Indicates a frame image The eigenvectors can be denoted as Frame image The position coordinates of the middle hand can be denoted as: .

[0027] Step 1013: Label the training samples according to the feature vectors, hand position coordinates and timestamps corresponding to each frame image, and obtain the labels corresponding to the training samples.

[0028] In the specific implementation, after the server obtains the feature vector of the frame image and the position coordinates of the hand in the frame image, it can label the training samples according to the feature vector, hand position coordinates and timestamp of each frame image to obtain the label of the training sample.

[0029] It is understandable that gestures are a continuous process. Therefore, when labeling, the training samples need to be segmented into frames. Feature extraction and labeling are performed on each frame image. Finally, a label group is formed for the training samples, which includes feature vectors, hand position coordinates and timestamps. Such a label group can help the visual gesture recognition model learn the production steps or techniques related to each feature vector, thereby more accurately understanding the gestures in the production process of intangible cultural heritage handicrafts.

[0030] In one example, a frame image The labeled information can be remembered as , The training samples in the training set The labels (label groups) of a training sample can be denoted as: , The label library corresponding to the training set can be denoted as , .

[0031] Step 102: Input the training samples in the training set into the initial model built based on the RNN network, and iteratively train the initial model until convergence based on the labels and the preset loss function to obtain the trained visual gesture recognition model.

[0032] In its implementation, the server pre-builds an initial model based on an RNN network. During model training, training samples from the training set are input into the initial model, and the initial model is iteratively trained until convergence based on labels and a pre-defined loss function, resulting in a trained visual gesture recognition model. Compared to ordinary CNN networks (Convolutional Neural Networks), RNN networks are particularly suitable for processing sequential data because they consider the influence of the previous state on the current state. This "memory" ability enables visual gesture recognition models built on RNN networks to understand the continuous movements in the production process of intangible cultural heritage handicrafts.

[0033] This model construction and training method links each frame of the intangible cultural heritage (ICH) handicraft production process with its corresponding action tags, forming a complete annotation system for the ICH handicraft production process and gestures. This system includes not only various hand movements and gestures during the production process, but also various techniques and tips. Taking velvet flower making as an example, the visual gesture recognition model can identify that "the action of pinching silk thread with fingers" corresponds to "splitting silk," "the action of holding scissors" corresponds to "sharpening," and "both hands grasping copper wire" corresponds to "attaching copper wire." This model provides learners with rich and precise teaching resources. Learners can not only learn velvet flower making by imitating the actions in the videos, but also correct their movements through system feedback, improve their skills, and thus better learn and inherit ICH handicrafts.

[0034] In one example, the server inputs training samples from the training set into an initial model built on an RNN network. Based on labels and a pre-defined loss function, the initial model is iteratively trained until convergence, resulting in a trained visual gesture recognition model. This can be achieved using the following formula: ; In the formula, This represents the model parameters of the initial model before this iteration of training. This represents the operation of finding the minimum value. This represents the total number of training samples input into the initial model. Indicates the first training samples, Indicates the first Labels of each training sample Indicates the initial model for the first... The predicted output of each training sample. This represents the preset loss function. This represents the preset regularization coefficient. Represents the regularization function. This represents the optimal model parameters, i.e., the model parameters of the trained visual gesture recognition model.

[0035] Step 103: Input the user's practice video into the visual gesture recognition model, so that the visual gesture recognition model can refer to the teaching video to perform gesture recognition and comparison, and obtain the judgment result of the visual gesture recognition model on the user's gesture action, which is either standard or non-standard.

[0036] In practice, self-media platforms and short video platforms offer numerous videos (instructional videos) documenting the production processes of intangible cultural heritage handicrafts. Users can choose handicrafts of interest to watch and learn, such as velvet flower making, embroidery, and paper cutting. While watching the videos, users carefully observe the gestures demonstrated by the inheritors, including the shape, position, and posture of their hands. They then attempt to imitate the gestures in the videos and record their own practice using a camera (such as a mobile phone camera), creating a user's practice video. The server can then access these practice videos and input them into a visual gesture recognition model. The model can then refer to the instructional videos for gesture recognition and comparison. Figure 3 , Figure 4 , Figure 5 and Figure 6 The diagram illustrates some gesture recognition methods. The server obtains the judgment results of the user's gesture actions from the visual gesture recognition model, and then compares the differences between the user's gestures and those of non-inheritors of traditional culture to determine errors or inaccuracies in the user's gesture practice. The judgment results of the user's gesture actions output by the visual gesture recognition model are categorized as standard or non-standard.

[0037] In one example, after receiving a video of a user's gesture, the visual gesture recognition model can, for example,... Figure 7 The steps shown refer to the instructional video for gesture recognition and comparison, and output the judgment result of the user's gesture action, specifically including: Step 1031: Extract and classify features from the user's practice video to determine the hand gestures in the practice video.

[0038] In a practical implementation, the visual gesture recognition model can be understood as consisting of a recognition and classification subnetwork and a contrastive discriminative subnetwork. After the user's time video is input into the visual gesture recognition model, the model first extracts and classifies the features of the user's action video through the recognition and classification subnetwork to determine the gesture actions in the action video.

[0039] Step 1032: Based on the feature vectors of gestures in the practice video, generate a practice feature sequence according to the time sequence, and calculate the distance matrix between the practice feature sequence and the teaching feature sequence.

[0040] In its implementation, after identifying the gestures in the practice video, the visual gesture recognition model can generate a practice feature sequence based on the feature vectors of the gestures in the practice video, following a chronological order. The visual gesture recognition model also needs to pre-generate a teaching feature sequence based on the instructional video, using a similar method, which will not be elaborated here. After obtaining the practice feature sequence and the instructional feature sequence, the visual gesture recognition model can calculate the distance matrix (Euclidean distance matrix) between them.

[0041] In one example, the practice feature sequence and the teaching feature sequence can be expressed by the following formula: ; ; In the formula, Represents a sequence of practical characteristics. Indicates the length of the practical feature sequence. Represents the first in the sequence of practical characteristics 1 eigenvector Represents a sequence of teaching characteristics. Indicates the length of the teaching feature sequence. Represents the first in the sequence of teaching features 1 eigenvector.

[0042] In one example, a visual gesture recognition model can calculate the distance matrix between the practical feature sequence and the teaching feature sequence using the following formula: ; In the formula, Represents the first in the sequence of practical characteristics The th eigenvector of the th feature vector 1 eigenvalue, Represents the first in the sequence of teaching features The th eigenvector of the th feature vector 1 eigenvalue, Represents the distance matrix in which the first... Line 1 The distance values ​​of the columns, the size of the distance matrix is .

[0043] Step 1033: Using the dynamic time warping algorithm, based on the preset boundary conditions, plan the optimal path from the top left corner to the bottom right corner in the distance matrix, and obtain the path matrix corresponding to the distance matrix.

[0044] In its implementation, after calculating the distance matrix between the practical feature sequence and the teaching feature sequence, the visual gesture recognition model can use a dynamic time warping algorithm to plan the optimal path from the top left corner to the bottom right corner within the distance matrix, based on preset boundary conditions, thus obtaining the path matrix corresponding to the distance matrix. This process is also known as dynamic path planning, and its purpose is to find an optimal path within the distance matrix that minimizes the difference between the practical and teaching feature sequences. This approach considers the asynchrony of time and the inconsistency in feature sequence lengths, matching the two feature sequences by selecting the best alignment method, and also employing the idea of ​​minimizing cumulative error to ensure optimal path selection.

[0045] In one example, the distance matrix calculated by the visual gesture recognition model The size is , It is the length of the practical feature sequence. The length of the teaching feature sequence is determined by dynamic path planning through a recursive approach, starting from the distance matrix. Starting from the top left corner (starting point), the optimal path to the bottom right corner (ending point) is calculated step by step, ultimately forming a path of size [value missing]. Path matrix Path matrix Used to record the optimal path for each location.

[0046] In one example, a visual gesture recognition model can use the dynamic time warping algorithm, based on preset boundary conditions, to plan the optimal path from the top left corner to the bottom right corner in the distance matrix, thus obtaining the path matrix corresponding to the distance matrix: ; , ; ; ; ; In the formula, , and This indicates the preset boundary conditions. Represents the path matrix of the first... Line 1 Column elements, This represents the operation of finding the minimum value. In other words, Represents the coordinates of the current position. Indicates a path originating from above. This indicates a path originating from the top left. This represents the path originating from the left. Ultimately, the visual gesture recognition model uses the path matrix... Starting from the bottom right corner (the finish line), according to The value continuously backtracks to the top left corner (the starting point), forming the optimal path.

[0047] Step 1034: Based on the path matrix, calculate the similarity between the user's gesture and the standard gesture in the teaching video. If the similarity is greater than a preset threshold, the user's gesture is determined to be standard and output. Otherwise, the user's gesture is determined to be non-standard and output.

[0048] In its implementation, the visual gesture recognition model obtains the path matrix after finding the optimal path. Then, it can be based on the path matrix. The system calculates the similarity between a user's gesture and a standard gesture in a teaching video. If the calculated similarity is greater than a preset threshold, the user's gesture is deemed standard and output. If the calculated similarity is less than or equal to the preset threshold, the user's gesture is deemed non-standard and output. The preset threshold can be set by those skilled in the art according to actual needs, and this embodiment does not specifically limit it.

[0049] In one example, a visual gesture recognition model can calculate the similarity between a user's gesture and a standard gesture in an instructional video, based on a path matrix, using the following formula: ; In the formula, Indicates the length of the optimal path. Represents the path matrix of the first... Line 1 The elements of a column correspond to their distance values ​​in the distance matrix. This represents the summation operation. This indicates the similarity between a user's gesture and the standard gesture in the instructional video.

[0050] Understandably, visual gesture recognition models are mainly based on the idea of ​​dynamic time warping algorithm to compare the differences between the user's gesture and standard gesture, thereby judging the user's non-standard gesture and providing correction suggestions to the user. Dynamic time warping algorithm can effectively measure the similarity between two time series and make up for the difference in length between time series, finding the optimal alignment.

[0051] Step 104: If the judgment result is non-standard, then based on the difference between the user's gesture and the standard gesture in the teaching video, a correction suggestion is generated and fed back to the user.

[0052] In practical implementation, if the visual gesture recognition model determines that a user's gesture is non-standard, it can generate correction suggestions based on the difference between the user's gesture and the standard gesture in the instructional video, and then provide feedback to the user. The feedback of these correction suggestions can be visually annotated in the video or visualization interface, or textually annotated in the text or data. By annotating the erroneous parts, users can understand the errors in their gestures and make targeted improvements.

[0053] Taking velvet flower making as an example, after a user uploads a video of their velvet flower making practice, the visual gesture recognition model will display the correct velvet flower making process and gesture demonstrations for the user's mistakes, for reference and imitation. Regarding velvet flower making, an intangible cultural heritage craft, the following correction suggestions are made: (1) Adjust the shape of the velvet flower. Based on the markings of the incorrect parts, if the shape of the user's velvet flower is inconsistent with that of the intangible cultural heritage inheritor, the user can be reminded to pay attention to the hand gestures of the intangible cultural heritage inheritor. In the practice video, if the outline of the velvet flower is not round enough, the user can be advised to change the shape of the hand gesture to make the velvet flower look more rounded.

[0054] (2) Adjusting the size of the velvet flower: If the size of the velvet flower made by the user does not match that of the inheritor of intangible cultural heritage, it can be suggested that the user adjust the strength or tightness of the hand gesture to change the size of the velvet flower. In the user's practice, it is common for users to make velvet flowers that are too large. Therefore, it can be suggested that the user reduce the strength of the hand gesture to make the velvet flower smaller.

[0055] (3) Adjust the layers of the velvet flowers. The layers of the velvet flowers refer to the layering effect of the velvet flowers. If the user's sense of layering of the velvet flowers is not obvious enough, the user can be reminded to increase the number of layers of the gesture. For example, when making velvet flowers, the user can add more yarn to each layer to enhance the sense of layering.

[0056] (4) Prompt the user to adjust the position of the gesture. Based on the markings of the incorrect parts, if the user's gesture position is inconsistent with that of the intangible cultural heritage inheritor, the user can be advised to adjust the position of the gesture. For example, if the user's gesture position is off-center from the center of the velvet flower, the user can be reminded to place the gesture in the correct position.

[0057] Through the above suggestions, users can understand the problems they encounter in making velvet flowers and receive specific improvement advice. These suggestions address aspects such as the shape, size, layering, and placement of the flowers, helping users refine their techniques and methods to improve the accuracy and aesthetics of their creations. Through continuous practice and refinement, users can gradually master the correct techniques for making velvet flowers.

[0058] In this embodiment, based on machine vision and deep learning technologies, the process of making intangible cultural heritage handicrafts is sequentially performed, including video acquisition, feature extraction and labeling, construction and training of a visual gesture recognition model, user-recorded practice videos for gesture recognition and comparison, and correction feedback for non-standard gestures. The labeling process must at least label the hand's position and shape representing the gesture, the hand's size representing the force and amplitude of the gesture, and the hand texture representing the contact and interaction between the hand and the material, enabling the visual gesture recognition model to fully learn the details of the intangible cultural heritage handicraft making process. During the learning process, users input recorded videos of their intangible cultural heritage (ICH) handcraft making process into a visual gesture recognition model. The model then identifies and judges the accuracy of the user's gestures. For inaccurate gestures, a feedback correction mechanism generates correction suggestions based on the difference between the user's gestures and standard gestures. This allows users to understand the problems they encounter during the making process and gradually master the skills of ICH handcraft making through continuous practice and correction. This breaks down the time and space limitations between ICH handcraft inheritors and learners, effectively improving the learning outcomes of ICH handcrafts and making an outstanding contribution to the protection and inheritance of intangible cultural heritage.

[0059] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0060] Another embodiment of this application proposes a system for learning intangible cultural heritage handicrafts based on visual gesture recognition and imitation comparison. The implementation details of the system for learning intangible cultural heritage handicrafts based on visual gesture recognition and imitation comparison in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0061] The structural diagram of the vision-based gesture recognition, imitation, and contrast learning system for intangible cultural heritage handicrafts proposed in this embodiment can be seen as follows: Figure 8As shown, it includes: training sample collection module 201, label annotation module 202, model building module 203, model training module 204, practice video receiving module 205, comparison and recognition module 206, and feedback module 207.

[0062] The training sample collection module 201 is used to collect high-definition videos of several intangible cultural heritage handicraft production processes to generate a training set.

[0063] The labeling module 202 is used to label the training samples in the training set. The labels include at least hand shape, hand size and hand texture. Hand shape is used to represent the position and gesture of the hand, hand size is used to represent the force and amplitude of the gesture, and hand texture is used to represent the contact and interaction between the hand and the material.

[0064] Model building module 203 is used to build an initial model based on an RNN network.

[0065] The model training module 204 is used to input training samples from the training set into the initial model, and iteratively train the initial model until convergence based on the labels and the preset loss function to obtain the trained visual gesture recognition model.

[0066] The practical video receiving module 205 is used to receive practical videos input by the user.

[0067] The comparison and recognition module 206 is used to input the practice video into the visual gesture recognition model, so that the visual gesture recognition model can refer to the teaching video to perform gesture recognition and comparison, and obtain the judgment result of the visual gesture recognition model on the user's gesture action, wherein the judgment result is standard or non-standard.

[0068] The feedback module 207 is used to generate correction suggestions and provide feedback to the user when the user's gesture is judged to be non-standard, based on the difference between the user's gesture and the standard gesture in the teaching video.

[0069] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0070] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0071] Another embodiment of this application relates to an electronic device, such as... Figure 9 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions executable by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to perform the vision-based gesture recognition imitation comparison learning method for intangible cultural heritage handicrafts in the above embodiments.

[0072] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0073] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0074] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0075] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for learning intangible cultural heritage handicrafts through visual gesture recognition, imitation, and comparison, characterized in that... include: A training set was generated based on high-definition videos of several intangible cultural heritage handicraft production processes, and the training samples in the training set were labeled. The labels included at least hand shape, hand size, and hand texture. The hand shape was used to characterize the position and gesture of the hand, the hand size was used to characterize the force and amplitude of the gesture, and the hand texture was used to characterize the contact and interaction between the hand and the material. The training samples in the training set are input into the initial model built based on the RNN network. The initial model is iteratively trained until convergence based on the labels and the preset loss function to obtain the trained visual gesture recognition model. The user's practice video is input into the visual gesture recognition model, which then refers to the instructional video to perform gesture recognition and comparison, and obtains the judgment result of the visual gesture recognition model on the user's gesture action; wherein, the judgment result is standard or non-standard. If the judgment result is non-standard, then based on the difference between the user's gesture and the standard gesture in the teaching video, a correction suggestion is generated and fed back to the user; After receiving a user's practice video, the visual gesture recognition model performs gesture recognition and comparison with the instructional video through the following steps, and outputs the judgment result of the user's gesture action: Feature extraction and classification are performed on the user's practice videos to determine the hand gestures in the practice videos: Based on the feature vectors of gestures in the practice video, a practice feature sequence is generated in chronological order, and the distance matrix between the practice feature sequence and the teaching feature sequence is calculated; wherein, the teaching feature sequence is generated in advance by the visual gesture recognition model based on the teaching video; Using the dynamic time warping algorithm, based on preset boundary conditions, the optimal path from the top left corner to the bottom right corner is planned in the distance matrix, and the path matrix corresponding to the distance matrix is ​​obtained. Based on the path matrix, the similarity between the user's gesture and the standard gesture in the teaching video is calculated. If the similarity is greater than a preset threshold, the user's gesture is determined to be standard and output. Otherwise, the user's gesture is determined to be non-standard and output.

2. The method for visual gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts according to claim 1, characterized in that, The training set is generated based on high-definition videos of several intangible cultural heritage handicraft production processes, and the training samples in the training set are labeled, including: High-definition videos of the production process of intangible cultural heritage handicrafts are collected as training samples to generate a training set, and the training samples are segmented into frames to obtain several frame images corresponding to the training samples. Using a preset feature extraction algorithm, features related to the hand shape, hand size, and hand texture are extracted from the frame image to obtain the feature vector of the frame image and the position coordinates of the hand in the frame image; The training samples are labeled according to the feature vectors, hand position coordinates and timestamps corresponding to each frame image, and the labels corresponding to the training samples are obtained.

3. The method for visual gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts according to claim 2, characterized in that, The training samples from the training set are input into an initial model built on an RNN network. The initial model is iteratively trained until convergence based on the labels and a preset loss function to obtain a trained visual gesture recognition model, which is expressed by the following formula: ； in, This represents the model parameters of the initial model before this iteration of training. This represents the operation of finding the minimum value. This represents the total number of training samples input into the initial model. Indicates the first training samples, Indicates the first Labels of each training sample This indicates that the initial model is related to the first... The predicted output of each training sample. This represents the preset loss function. This represents the preset regularization coefficient. Represents the regularization function. This represents the optimal model parameters, i.e., the model parameters of the trained visual gesture recognition model.

4. The method for visual gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts according to claim 1, characterized in that, The practice feature sequence and the teaching feature sequence are expressed by the following formula: ； ； in, This represents the sequence of practical features. Indicates the length of the practical feature sequence. Represents the first in the sequence of practical features 1 eigenvector This represents the teaching feature sequence. Indicates the length of the teaching feature sequence. Represents the first in the teaching feature sequence 1 eigenvector; The distance matrix between the practice feature sequence and the teaching feature sequence is calculated using the following formula: ； in, Represents the first in the sequence of practical features The th eigenvector of the th feature vector 1 eigenvalue, Represents the first in the teaching feature sequence The th eigenvector of the th feature vector 1 eigenvalue, Represents the distance matrix in which the first... Line number The distance values ​​of the columns, the size of the distance matrix is .

5. The method for visual gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts according to claim 4, characterized in that, Using the following formula and the dynamic time warping algorithm, based on preset boundary conditions, the optimal path from the top left corner to the bottom right corner is calculated step by step in the distance matrix, resulting in the path matrix corresponding to the distance matrix: ； ， ； ； ； ； in, , and This represents the preset boundary conditions. Represents the path matrix of the first... Line number Column elements, This indicates the operation of finding the minimum value.

6. The method for vision-based gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts according to claim 5, characterized in that, The similarity between the user's gesture and the standard gesture in the instructional video is calculated using the following formula, based on the path matrix: ； in, This represents the length of the optimal path. Represents the path matrix of the first... Line number The elements of the column correspond to the distance values ​​in the distance matrix. This represents the summation operation. This indicates the similarity between the user's gesture and the standard gesture in the instructional video.

7. A system for vision-based gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts, used to implement the method for vision-based gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts as described in any one of claims 1 to 6, characterized in that, include: The system includes a training sample collection module, a label annotation module, a model building module, a model training module, a practical video receiving module, a comparison and recognition module, and a feedback module. The training sample collection module is used to collect high-definition videos of several intangible cultural heritage handicraft production processes to generate a training set; The labeling module is used to label the training samples in the training set; wherein the labeled labels include at least hand shape, hand size and hand texture, the hand shape is used to characterize the position and gesture of the hand, the hand size is used to characterize the force and amplitude of the gesture, and the hand texture is used to characterize the contact and interaction between the hand and the material. The model building module is used to build an initial model based on an RNN network; The model training module is used to input the training samples in the training set into the initial model, and iteratively train the initial model until convergence based on the labels and the preset loss function to obtain the trained visual gesture recognition model. The practice video receiving module is used to receive practice videos input by the user; The comparison and recognition module is used to input the practice video into the visual gesture recognition model, so that the visual gesture recognition model can refer to the teaching video to perform gesture recognition and comparison, and obtain the judgment result of the visual gesture recognition model on the user's gesture action; wherein, the judgment result is standard or non-standard; The feedback module is used to generate correction suggestions and provide feedback to the user when the user's gesture is determined to be non-standard, based on the difference between the user's gesture and the standard gesture in the teaching video.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the vision-based gesture recognition, imitation, and comparative learning method for intangible cultural heritage handicrafts as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for visual-based gesture recognition, imitation, and comparative learning of intangible cultural heritage handicrafts as described in any one of claims 1 to 6.

Images