Hand direction recognition method based on behavior recognition, electronic device, and storage medium
By using a behavior recognition method based on C3D networks, training a model with handicraft videos, and then visualizing it, the problems of recognition accuracy and real-time performance in the inheritance of intangible cultural heritage handicrafts were solved, and efficient inheritance of intangible cultural heritage handicrafts was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2023-10-31
- Publication Date
- 2026-07-24
AI Technical Summary
Existing gesture recognition technology has low accuracy and poor real-time performance in the inheritance of intangible cultural heritage handicrafts, resulting in poor inheritance effects.
A behavior recognition method based on C3D network is adopted. By acquiring videos of different handicraft categories and artisans' handicrafts as samples, preprocessing, feature extraction and model training are performed. Combined with the visualization of hand direction, the C3D network is used to learn and model time series information to improve the recognition accuracy.
This has enabled the efficient and accurate transmission of intangible cultural heritage handicrafts, allowing users to learn the steps involved in making the handicrafts more clearly, thus improving the effectiveness of the transmission and the user experience.
Smart Images
Figure CN117373130B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of behavior recognition technology, and in particular to a hand orientation recognition method, electronic device, and storage medium based on behavior recognition. Background Technology
[0002] With the rapid development of mobile internet, logging into short video platforms daily has become an integral part of people's lives. As more and more intangible cultural heritage inheritors join these platforms, they are gradually becoming a new channel for the transmission of intangible cultural heritage. An increasing number of intangible cultural heritage craftspeople and inheritors are choosing to disseminate and showcase their skills on short video platforms. In the process of transmitting intangible cultural heritage crafts, the direction and techniques of hand movements are crucial. However, in the traditional transmission of crafts, many skills rely on the master's personal demonstration, with apprentices learning through observation and imitation. This process is time-consuming, inefficient, and the effectiveness of transmission is limited by the apprentice's observation and comprehension abilities, potentially leading to misunderstandings.
[0003] With the development of modern technology, people have gradually realized that integrating intangible cultural heritage handicrafts with modern technology can effectively promote the inheritance of intangible cultural heritage and facilitate its revitalization. Gesture recognition technology is an important practice that combines intangible cultural heritage handicrafts with modern technology. Gesture recognition technology is a technology that converts human hand movements into computer input. While it can be implemented based on sensors, its accuracy and real-time performance are not high, resulting in unsatisfactory outcomes in the inheritance of intangible cultural heritage. Summary of the Invention
[0004] The purpose of this application is to provide a hand orientation recognition method, electronic device, and storage medium based on behavior recognition, which can realize hand orientation recognition in video in real time and accurately, and visualize the recognized direction and action, so that users can learn intangible cultural heritage handicrafts more carefully and conveniently, and ensure that the inheritance of intangible cultural heritage handicrafts will never be interrupted.
[0005] To address the aforementioned technical problems, embodiments of this application provide a hand orientation recognition method based on behavior recognition, comprising the following steps: acquiring several videos of handicraft production by different craftspeople and different craftspeople as sample videos; dividing the sample videos into several video segments, extracting keyframes from each video segment, calculating the similarity between each keyframe, and storing the video segments corresponding to keyframes with similarity greater than a first threshold in the same folder; extracting feature vectors from each video segment in the folder and calculating a first average value, labeling the first average value as a tag for each video segment in the folder, and training a preset recognition model based on the video segments and their tags; inputting the video to be recognized into the trained recognition model to obtain the hand orientation in the video to be recognized, and visually displaying it in the video to be recognized.
[0006] Embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described hand orientation recognition method based on behavior recognition.
[0007] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described hand orientation recognition method based on behavior recognition.
[0008] The embodiments of this application provide a hand orientation recognition method, electronic device, and storage medium based on behavior recognition. These methods train a recognition model built on a C3D network. The C3D network can learn and model time-series information, and compared to traditional two-dimensional convolutional neural networks, the recognition model built on the C3D network has a much higher recognition accuracy. When selecting training samples, crawling videos of different handicraft categories and craftspeople's production processes can improve the generalization ability and universality of the trained recognition model, making it applicable to different recognition scenarios. The hand orientation recognized from the video is combined with hand movements for visualization, allowing users to see more clearly, carefully, and accurately the detailed steps taken by intangible cultural heritage inheritors when making handicrafts. This makes it easier for users to learn and practice, and the sense of accomplishment and satisfaction encourages them to continuously learn and create, thus becoming new handicraft makers and disseminators, ensuring the uninterrupted transmission of intangible cultural heritage.
[0009] In some optional embodiments, acquiring several handicraft production videos of different handicraft categories and artisans as sample videos includes: preprocessing the acquired handicraft production videos of different handicraft categories and artisans, and using the preprocessed handicraft production videos as sample videos; the preprocessing includes at least: denoising and smoothing the handicraft production videos; normalizing the screen size of each handicraft production video, so that the normalized screen size of each handicraft production video is the same; extracting the three-dimensional coordinate data of hand movements in the handicraft production videos and performing time alignment. Preprocessing can remove interference information in the sample videos, so that the sample information is converted into a unified format, which is convenient for training the recognition model and improves the accuracy and stability of the recognition effect of the trained recognition model.
[0010] In some optional embodiments, calculating the similarity between the keyframes includes: for a first keyframe and a second keyframe, establishing a first window and a second window of the same size at the same initial position of the first keyframe and the second keyframe, respectively; calculating a local structural similarity index (SSIM) based on the pixel values of each pixel in the first window and the pixel values of each pixel in the second window; synchronously sliding the first window and the second window according to a preset step size to calculate each local SSIM; averaging all the calculated local SSIMs to obtain the global SSIM, and using the global SSIM as the similarity between the first keyframe and the second keyframe. The SSIM index can effectively measure the similarity between two images. Since hand movements are relatively subtle and the hand contains many areas, directly calculating the global SSIM is not accurate and may dilute the differences between some local areas. Therefore, this application calculates each local SSIM and then averages them to obtain the global SSIM, which can better determine the similarity between keyframes.
[0011] In some optional embodiments, the local SSIM is calculated based on the pixel values of each pixel in the first window and the pixel values of each pixel in the second window using the following formula: ; ; ; ; ; in, Characterizing the first keyframe, Characterizes the second keyframe, This is the average pixel value of all pixels within the first window. This is the average pixel value of all pixels within the second window. This represents the variance of the pixel values of each pixel within the first window. This represents the variance of the pixel values of each pixel within the second window. This is the covariance between the pixel values of each pixel in the first window and the pixel values of each pixel in the second window. Characterizes the dynamic range of pixel values. In some optional embodiments, the step of extracting feature vectors from each video segment in the folder and calculating a first average value, and labeling the first average value as a tag for each video segment in the folder, includes: extracting feature vectors from each video segment in the folder and calculating a first average value, using the first average value as a representative vector of the folder; calculating the distance between the representative vector and five preset standard vectors, labeling the standard vector with the smallest distance to the representative vector as a first tag for each video segment in the folder, and labeling the category to which the standard vector with the smallest distance to the representative vector belongs as a second tag for each video segment in the folder; wherein, the five standard vectors include: a first standard vector representing hand movement from top to bottom, a second standard vector representing hand movement from bottom to top, a third standard vector representing hand rotation clockwise, a fourth standard vector representing hand rotation counterclockwise, and a fifth standard vector representing two hands rotating in opposite directions. Dividing the tags into a first tag representing feature vectors and a second tag representing categories can further improve the training accuracy of the recognition model.
[0012] In some optional embodiments, training the pre-built recognition model based on the C3D network using the video clips and their labels includes: training the pre-defined C3D network to convergence using the video clips and their first labels to obtain a trained C3D network; training the pre-defined classifier to convergence using the video clips and their second labels to obtain a trained classifier; wherein the classifier is a Support Vector Machine (SVM); and concatenating the trained C3D network and the trained classifier to obtain a trained recognition model. Training the C3D network and the classifier separately can independently optimize and improve the performance of feature extraction and classification. Concatenating the two trained parts to obtain the trained recognition model effectively improves training speed and training effect.
[0013] In some optional embodiments, the visualization of the video to be identified includes: adding arrow markers and text markers at corresponding positions in the video to indicate the identified hand direction; wherein, the corresponding position is the location where the hand action corresponding to the hand direction occurs. Arrow markers are vivid and clear, and text markers are accurate and reliable. Combining the two to visualize the hand direction allows users to better learn the hand actions in the video.
[0014] In some optional embodiments, the step of inputting the video to be identified into a trained recognition model to obtain the hand direction in the video to be identified includes: dividing the video to be identified into several video segments to be identified, inputting each video segment to be identified into the trained recognition model to obtain the hand direction in each video segment to be identified; after obtaining the hand direction in the video to be identified, the method further includes: saving the video to be identified, segment by segment, after recording attribute information, to a preset database; wherein, the attribute information includes: the video file name of the video to be identified to which the video segment belongs, the number of the video segment to which it belongs, the feature vector extracted from the video segment, the hand direction identified from the video segment, the recognition time corresponding to the video segment, the name of the recognition model used to identify the hand direction in the video segment, and the data source of the video to which the video segment belongs. The videos that identify hand movements are stored in a database as video clips, and attribute information is recorded. This allows users to easily retrieve and review the videos at any time, thereby further improving the inheritance of intangible cultural heritage handicrafts. Attached Figure Description
[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.
[0016] Figure 1 This is a flowchart of a hand orientation recognition method based on behavior recognition provided in one embodiment of this application; Figure 2 This is a schematic diagram illustrating a visual representation of hand orientation provided in one embodiment of this application; Figure 3 This is a flowchart illustrating the preprocessing of a handicraft production video in one embodiment of this application; Figure 4 This is a flowchart illustrating the calculation of similarity between keyframes in one embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.
[0018] One embodiment of this application relates to a hand orientation recognition method based on behavior recognition, applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the hand orientation recognition method based on behavior recognition in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.
[0019] The specific process of the hand direction recognition method based on behavior recognition in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Obtain several videos of handicraft production from different categories of handicrafts and different artisans as sample videos.
[0020] In practice, short video platforms possess a rich library of videos showcasing handicrafts. Servers can use packet sniffing software, such as Fiddler, to crawl videos from various platforms, capturing different categories of handicrafts and craftspeople as sample videos. The more categories of handicraft videos and the more diverse the sources of craftspeople, the stronger the generalization ability and universality of the trained recognition model will be.
[0021] In some examples, handicraft production videos of different categories can include videos of making velvet flowers, paper cutting, embroidery, etc.
[0022] Step 102: Divide the sample video into several video segments, extract keyframes from each video segment, calculate the similarity between each keyframe, and store the video segments corresponding to keyframes with similarity greater than the first threshold in the same folder.
[0023] In the specific implementation, after the server obtains the sample video, it can use the boundary partitioning method to divide the sample into several video segments. Each video segment contains only one stage action in the handicraft production process. These stage actions are mainly divided into several categories in terms of directional features. Therefore, the server can calculate the similarity between each key frame and regard the video segments corresponding to the key frames with similarity greater than the first threshold as video segments with the same directional features. These video segments with the same directional features are stored in the same folder.
[0024] In some cases, the server can divide the sample video into several video segments of equal length, each video segment consisting of 16 frames, and there is frame overlap between two temporally adjacent video segments. The server can extract the 8th frame of each video segment as the keyframe of each video segment.
[0025] In some cases, servers can use image similarity metrics such as SSIM (Structural Similarity Index Measurement), MSE (Mean Squared Error), and PSNR (Peak Signal to Noise Ratio) to calculate the similarity between two keyframes.
[0026] Step 103: Extract the feature vectors of each video segment in the folder and calculate the first average value. Label the first average value as the tag of each video segment in the folder. Train the recognition model pre-built based on the C3D network based on the video segments and their tags.
[0027] In the implementation, the server traverses each folder, extracting feature vectors from each video segment within each folder. For the feature vectors of each video segment within the same folder, the server calculates their first average value and labels this average value as the tag for each video segment in that folder. The server uses all labeled video segments as training samples and iteratively trains the pre-built recognition model based on the C3D network based on these video segments and their tags until the recognition model converges, resulting in a trained recognition model. The C3D network can learn and model time-series information, and compared to traditional two-dimensional convolutional neural networks, the recognition accuracy of the recognition model built based on the C3D network is much higher.
[0028] In some examples, the server extracts feature vectors from each video segment in a folder and calculates a first average value. This first average value is used as the representative vector for that folder. Then, the distance between the representative vector and five predefined standard vectors is calculated. The standard vector with the smallest distance to the representative vector is labeled as the first label for each video segment in that folder, and the category to which the standard vector with the smallest distance belongs is labeled as the second label for each video segment in that folder. Dividing the labels into a first label representing the feature vector and a second label representing the category can further improve the training accuracy of the recognition model.
[0029] In some examples, the five standard vectors include: a first standard vector representing hand movement from top to bottom, a second standard vector representing hand movement from bottom to top, a third standard vector representing hand rotation clockwise, a fourth standard vector representing hand rotation counterclockwise, and a fifth standard vector representing rotation of both hands in opposite directions. These five standard vectors are obtained by feature extraction of direction based on video optical flow.
[0030] In some examples, when training the recognition model, the server first trains a pre-defined C3D network to convergence based on video clips and their first labels, resulting in a trained C3D network. Then, it trains a pre-defined classifier to convergence based on video clips and their second labels, resulting in a trained classifier. Finally, the trained C3D network and the trained classifier are concatenated to obtain the trained recognition model. Training the C3D network and classifier separately can independently optimize and improve the performance of feature extraction and classification. Concatenating the two trained parts to obtain the trained recognition model effectively improves both training speed and training effectiveness.
[0031] In some cases, the recognition model can use SVM (Support Vector Machines) as the classifier. The training principle of SVM can be expressed by the following formula: ; In the formula, Indicates the test sample. This represents the i-th training sample. Let represent the label of the i-th training sample. Indicates the first Lagrange multipliers of training samples, Represents the kernel function. This indicates the bias term.
[0032] Step 104: Input the video to be recognized into the trained recognition model to obtain the hand direction in the video to be recognized, and then visualize it in the video to be recognized.
[0033] In practical implementation, after obtaining the trained recognition model, the server can input the video to be recognized into the model to obtain the hand direction in the video. This is then visualized within the video by adding arrow and text markers at corresponding locations to indicate the recognized hand direction; these locations represent the positions where the hand movements occur. The arrow markers are vivid and clear, while the text markers are accurate and reliable. Combining these two methods to visualize hand directions allows users to better learn the hand movements in the video.
[0034] In some examples, adding arrow markers and text labels to handicraft videos can be like... Figure 2 As shown.
[0035] In some examples, the server inputs the video to be recognized into a trained recognition model to obtain the hand direction within the video. Specifically, the video to be recognized is divided into several segments, and each segment is input into the trained recognition model to obtain the hand direction within each segment. After obtaining the hand direction, the server can also save the video to be recognized, segment by segment, after recording attribute information, to a pre-defined database. The attribute information includes: the video filename of the video segment to which it belongs (Video Name), the segment number within the video segment (SegmentNumber), the feature vector extracted from the segment, the hand direction identified from the segment, the recognition time, the name of the recognition model used to identify the hand direction, and the data source of the video segment. The videos that identify hand movements are stored in a database as video clips, and attribute information is recorded. This allows users to easily retrieve and review the videos at any time, thereby further improving the inheritance of intangible cultural heritage handicrafts.
[0036] In this embodiment, a recognition model based on a C3D network is trained. The C3D network can learn and model time-series information, and compared to traditional two-dimensional convolutional neural networks, the recognition model based on a C3D network has a much higher recognition accuracy. When selecting training samples, videos of different handicraft categories and craftspeople's production processes are crawled as sample videos. This improves the generalization ability and universality of the trained recognition model, making it applicable to different recognition scenarios. The hand directions and hand movements identified from the videos are visualized, allowing users to see more clearly, carefully, and accurately the detailed steps taken by inheritors of intangible cultural heritage in making their handicrafts. This makes it easier for users to learn and practice, and the sense of accomplishment and satisfaction encourages them to continue learning, thus turning users into new handicraft makers and disseminators, ensuring the uninterrupted transmission of intangible cultural heritage.
[0037] In some embodiments, after the server obtains several videos of handicraft production from different categories and artisans as sample videos, it also needs to preprocess these videos and use the preprocessed videos as sample videos. The server's preprocessing of the handicraft videos can be performed as follows: Figure 3 The steps shown are implemented as follows: Step 201: Denoise and smooth the handicraft production video.
[0038] In practice, after the server crawls the handicraft videos, it first performs noise reduction and smoothing on the handicraft production videos, that is, removes noise and unnecessary information from the videos, and only retains the information of the hands.
[0039] Step 202: Normalize the screen size of each handicraft production video. After normalization, the screen size of each handicraft production video is the same.
[0040] In practice, the scraped videos of handicrafts vary in size and format, which hinders the stable training of the recognition model. The server normalizes the screen size of each handicraft video, ensuring that all videos have the same screen size after normalization. This improves the accuracy and stability of the trained recognition model.
[0041] Step 203: Extract the three-dimensional coordinate data of hand movements in the handicraft making video and perform time alignment.
[0042] In practice, to facilitate subsequent feature extraction, the server can extract the 3D coordinate data of hand movements in the handicraft making video and perform time alignment during the preprocessing stage, thereby shortening the model training time.
[0043] In some cases, the server calculates the similarity between keyframes, which can be done through methods such as... Figure 4 The steps shown are implemented as follows: Step 301: For the first keyframe and the second keyframe, create a first window and a second window of the same size at the same initial position of the first keyframe and the second keyframe, respectively.
[0044] In the specific implementation, for the first keyframe and the second keyframe, the server creates a first window and a second window of the same size at the same initial position of the first keyframe and the second keyframe, respectively. For example, a first window and a second window of size N×N are created at the top left corner of the first keyframe and the second keyframe, respectively.
[0045] Step 302: Calculate the local SSIM based on the pixel values of each pixel in the first window and the pixel values of each pixel in the second window.
[0046] In the specific implementation, for the first window and the second window, the server can calculate the local SSIM based on the pixel values of each pixel in the first window and the pixel values of each pixel in the second window using the following formula:
[0047] ; ; ; ; In the formula, Representing the first keyframe, Characterizes the second keyframe. This is the average pixel value of all pixels within the first window. This is the average pixel value of all pixels within the second window. This represents the variance of the pixel values of all pixels within the first window. This represents the variance of the pixel values of each pixel within the second window. This is the covariance between the pixel values of each pixel in the first window and the pixel values of each pixel in the second window. Characterizing the dynamic range of pixel values, The value is usually 255.
[0048] Step 303: Slide the first and second windows synchronously according to the preset step size to calculate each local SSIM.
[0049] In the specific implementation, the server can synchronously slide the first and second windows according to a preset step size to calculate each local SSIM. For example, the size of the first and second windows is both... The preset step size is Then, when the server slides the first and second windows, it can slide to the right. 1 pixel.
[0050] Step 304: Calculate the average value of all the calculated local SSIMs to obtain the global SSIM, and use the global SSIM as the similarity between the first keyframe and the second keyframe.
[0051] In practice, the SSIM metric can effectively measure the similarity between two images. Since hand movements are subtle and the hand contains many regions, directly calculating the global SSIM is not accurate and may dilute the differences between some local regions. Therefore, this application calculates the SSIM of each local region and then averages them to obtain the global SSIM, which can better determine the similarity between the first keyframe and the second keyframe.
[0052] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.
[0053] Another embodiment of this application relates to an electronic device, such as... Figure 5 As shown, it includes: at least one processor 401; and a memory 402 communicatively connected to the at least one processor 401; wherein the memory 402 stores instructions executable by the at least one processor 401, the instructions being executed by the at least one processor 401 to enable the at least one processor 401 to execute the hand orientation recognition method based on behavior recognition in the above embodiments.
[0054] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0055] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0056] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.
[0057] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0058] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.
Claims
1. A hand orientation recognition method based on behavior recognition, characterized in that, include: Obtain several videos of handicraft production from different categories of handicrafts and different artisans as sample videos; The sample video is divided into several video segments, and keyframes are extracted from each video segment. The similarity between each keyframe is calculated, and the video segments corresponding to keyframes with similarity greater than a first threshold are stored in the same folder. Extract the feature vectors of each video segment in the folder and calculate the first average value. Label the first average value as the tag of each video segment in the folder. Train the recognition model pre-built based on the C3D network based on the video segments and their tags. The video to be identified is input into the trained recognition model to obtain the hand direction in the video to be identified, and then the image is visualized in the video to be identified. The calculation of the similarity between the keyframes includes: For the first keyframe and the second keyframe, a first window and a second window of the same size are created at the same initial position of the first keyframe and the second keyframe, respectively. Calculate the local structural similarity index SSIM based on the pixel values of each pixel in the first window and the pixel values of each pixel in the second window; The first and second windows are slid synchronously according to a preset step size to calculate each local SSIM. The average value of all calculated local SSIMs is obtained to get the global SSIM, and the global SSIM is used as the similarity between the first keyframe and the second keyframe. The step of extracting feature vectors from each video segment in the folder and calculating a first average value, and then labeling the first average value as a tag for each video segment in the folder, includes: Extract the feature vectors of each video segment in the folder and calculate the first average value, then use the first average value as the representative vector of the folder; Calculate the distance between the representative vector and the five preset standard vectors, label the standard vector with the smallest distance to the representative vector as the first label of each video segment in the folder, and label the category to which the standard vector with the smallest distance to the representative vector belongs as the second label of each video segment in the folder; The five types of standard vectors include: a first standard vector representing the hand moving from top to bottom, a second standard vector representing the hand moving from bottom to top, a third standard vector representing the hand rotating clockwise, a fourth standard vector representing the hand rotating counterclockwise, and a fifth standard vector representing the two hands rotating in opposite directions. The training of the recognition model pre-built based on the C3D network based on the video clips and their labels includes: Based on the video clip and its first label, the preset C3D network is trained until convergence, resulting in a trained C3D network; Based on the video segment and its second label, a preset classifier is trained until convergence to obtain a trained classifier; wherein, the classifier is a support vector machine (SVM); The trained C3D network and the trained classifier are concatenated to obtain the trained recognition model.
2. The hand orientation recognition method based on behavior recognition according to claim 1, characterized in that, The acquisition of several videos of handicraft production from different categories and artisans as sample videos includes: Several different handicraft categories and different artisans' handicraft production videos were preprocessed, and the preprocessed handicraft production videos were used as sample videos. The preprocessing includes at least: The video of the handicraft making process was denoised and smoothed. The screen size of each of the aforementioned handicraft production videos is normalized, and the screen size of each of the aforementioned handicraft production videos is the same after normalization; Extract the three-dimensional coordinate data of hand movements from the handicraft making video and align them in time.
3. The hand orientation recognition method based on behavior recognition according to claim 1, characterized in that, The local SSIM is calculated using the following formula, based on the pixel values of each pixel within the first window and the pixel values of each pixel within the second window: ； ； ； ； ； in, Characterizing the first keyframe, Characterizes the second keyframe, This is the average pixel value of all pixels within the first window. This is the average pixel value of all pixels within the second window. This represents the variance of the pixel values of each pixel within the first window. This represents the variance of the pixel values of each pixel within the second window. This is the covariance between the pixel values of each pixel in the first window and the pixel values of each pixel in the second window. Characterizes the dynamic range of pixel values.
4. The hand orientation recognition method based on behavior recognition according to any one of claims 1 to 3, characterized in that, The visualization of the video to be identified includes: Arrow markers and text markers are added at corresponding positions in the video to indicate the identified hand direction; wherein, the corresponding position is the location where the hand action corresponding to the hand direction occurs.
5. The hand orientation recognition method based on behavior recognition according to any one of claims 1 to 3, characterized in that, The step of inputting the video to be identified into the trained recognition model to obtain the hand orientation in the video to be identified includes: The video to be identified is divided into several video segments to be identified, and each video segment to be identified is input into the trained recognition model to obtain the hand direction in each video segment to be identified. After obtaining the hand orientation in the video to be identified, the method further includes: The video to be identified is divided into video segments, and after recording attribute information, it is saved to a preset database. The attribute information includes: the video file name of the video to be identified to which the video segment belongs, the number of the video segment to which it belongs, the feature vector extracted from the video segment, the hand direction identified from the video segment, the recognition time corresponding to the video segment, the name of the recognition model used to identify the hand direction in the video segment, and the data source of the video to which the video segment belongs.
6. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the hand orientation recognition method based on behavior recognition as described in any one of claims 1 to 5.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the hand orientation recognition method based on behavior recognition as described in any one of claims 1 to 5.