Sign language recognition system based on dynamic time warping algorithm

Abstract

The invention discloses a sign language recognition system based on a dynamic time warping algorithm, and the system comprises a data collection module which is used for collecting gesture data in real time through a camera and a depth sensor; the feature extraction and time sequence conversion module is used for converting the gesture data into a time sequence; the dynamic time warping matching module is used for matching the input gesture with the gesture in the template library by utilizing a dynamic time warping algorithm; the matching result selection and template classification module is used for selecting the most matched template according to the similarity and determining the gesture category; the embedded platform and hardware acceleration module is used for executing a sign language recognition algorithm on the embedded platform and improving the recognition speed and real-time performance through hardware acceleration; and the character and voice output and real-time feedback optimization module is used for converting the recognition result into characters and voice output. Through hardware acceleration and data compression optimization, the sign language recognition precision, real-time performance and adaptability are improved, and the sign language recognition method is suitable for an embedded system with limited resources.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a sign language recognition system based on a dynamic time warping algorithm. Background Art

[0002] With the rapid development of artificial intelligence, computer vision and natural language processing technologies, gesture recognition, as an important branch of human-computer interaction, has gradually attracted widespread attention. Gesture recognition technology aims to automatically recognize human gestures through computer vision and deep learning methods, and is widely used in sign language translation, virtual reality, augmented reality, smart home control and other fields. In these applications, gesture recognition technology can effectively improve the interactive experience, especially in the communication and life of deaf-mute people, the technical advantages of sign language recognition are particularly important.

[0003] Existing gesture recognition methods are mainly based on image processing, machine learning and deep learning technologies. They usually collect hand movement images or video data through cameras, depth sensors and other devices, and convert gestures into digital data that can be processed by computers after image preprocessing, feature extraction and other steps. Then, various algorithms are used for pattern matching and classification to achieve gesture recognition. With the rise of deep learning, more and more gesture recognition methods have begun to use models such as deep neural networks (DNNs) and convolutional neural networks (CNNs) for automatic learning and recognition of gestures, and have achieved remarkable results.

[0004] However, although existing gesture recognition methods have achieved good results in some scenarios, they still face some technical bottlenecks and challenges. First, most gesture recognition algorithms in existing methods rely on static images or video frames for processing, which makes the accuracy of gesture recognition more problematic when facing dynamic and complex gesture movements. For example, factors such as gesture speed, posture changes, interference from ambient light, and background noise can affect recognition accuracy. These factors make it difficult for traditional image-based gesture recognition methods to achieve high-precision, high-real-time recognition, especially in complex environments, where existing methods are difficult to guarantee recognition stability.

[0005] Secondly, existing gesture recognition methods usually require training with a large amount of annotated data, which places high demands on the construction and training process of the gesture template library. In practical applications, the construction process of the gesture library is both time-consuming and laborious, and for the expansion and update of gesture categories, existing methods usually need to re-train the entire library, which poses a great challenge to the real-time and flexibility of application scenarios. Therefore, when faced with new gesture types or gesture changes, existing methods require a large amount of data and computing resources for retraining and tuning, which cannot meet the high requirements for real-time and adaptability in practical applications.

[0006] In addition, the real-time problem in existing gesture recognition technology also needs to be solved urgently. Gesture recognition requires not only high accuracy, but also the ability to respond to user gestures in real time. However, the computational complexity of existing methods is high, especially when processing high-dimensional image data, the amount of calculation is huge, resulting in a lack of real-time performance. Although deep learning methods have improved accuracy, compared with traditional gesture recognition methods, deep learning models usually require higher computing resources, which makes these methods difficult to run efficiently on embedded systems with limited computing power.

[0007] In order to solve the above problems, scholars and engineers have proposed a gesture recognition method based on the dynamic time warping (DTW) algorithm. The dynamic time warping algorithm is a classic algorithm for time series alignment. It can handle nonlinear deformations between time series and is particularly suitable for processing sequence data with inconsistent timing, speed changes, and unequal lengths. In gesture recognition, the dynamic time warping algorithm can effectively recognize complex gestures by matching the input gesture time series with the gesture sequence in the template library without requiring too many computing resources. Compared with traditional image processing methods, the dynamic time warping algorithm can directly match the time series data of gestures, thereby avoiding the complex image processing process and improving the response speed and computing efficiency of the system.

[0008] Although the dynamic time warping algorithm has shown good performance in gesture recognition, the existing gesture recognition methods based on the dynamic time warping algorithm still have some technical defects. First, the traditional dynamic time warping algorithm still has the problem of high computational complexity when processing large-scale data. Although the dynamic time warping algorithm is very effective in matching time series data, for large-scale gesture template libraries, it still needs to match and calculate each template, which leads to a sharp increase in the amount of calculation when the system faces a diverse gesture library, affecting the real-time and efficiency of recognition.

[0009] Secondly, although the dynamic time warping algorithm has good nonlinear alignment capabilities, it still faces challenges in feature selection and matching accuracy when processing high-dimensional feature data. Existing methods usually rely on traditional feature extraction techniques, which often have difficulty in processing complex gestures and have poor adaptability to individual differences of different users. In gesture recognition, the motion trajectory, position, shape, and angle of gestures are affected by the movement habits, physical conditions, and environmental factors of different users. Traditional feature extraction methods have poor adaptability in these situations.

[0010] Therefore, how to provide a sign language recognition system based on a dynamic time warping algorithm is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0011] One purpose of the present invention is to propose a sign language recognition system based on a dynamic time warping algorithm. The present invention makes full use of computer vision technology, dynamic time warping (DTW) algorithm, hardware acceleration technology and data compression optimization method, and describes in detail the matching of gesture time series data through the dynamic time warping algorithm, so as to achieve efficient, real-time and high-precision sign language recognition. The calculation process is optimized through hardware acceleration, and the model parameters are adjusted in combination with real-time feedback to improve the accuracy and adaptability of gesture recognition. The method has the advantages of high real-time performance, strong accuracy, good adaptability, high computing efficiency and the ability to run efficiently in embedded systems.

[0012] A sign language recognition system based on a dynamic time warping algorithm according to an embodiment of the present invention includes:

[0013] A data acquisition module, used to collect hand motion images or gesture data in real time through a camera and a depth sensor;

[0014] Feature extraction and time series conversion module, used to convert gesture data from static images into time series data;

[0015] A dynamic time warping matching module is used to match the input gesture time sequence with the gesture template stored in the database using a dynamic time warping algorithm;

[0016] The matching result selection and template classification module is used to select the template that best matches the input gesture based on the similarity calculated by the dynamic time warping algorithm, and determine the gesture category corresponding to the template;

[0017] Embedded platform and hardware acceleration module, used to execute sign language recognition algorithm on embedded platform and improve recognition speed and real-time performance through hardware acceleration;

[0018] The text-speech output and real-time feedback optimization module is used to convert recognition results into text and speech output for real-time user interaction.

[0019] Optionally, modules can be connected via the following methods:

[0020] S1, collect hand motion images and gesture data through cameras and depth sensors;

[0021] S2, preprocessing the collected hand motion images and gesture data, extracting key features of the gesture's motion trajectory, position, shape, and angle, and converting the extracted features into time series data;

[0022] S3, using a dynamic time warping algorithm, matching the time series data with the time series data in a pre-stored gesture template library, and calculating the similarity between each gesture template;

[0023] S4. Selecting a template that best matches the input gesture based on the similarity between each gesture template, and determining the gesture category corresponding to the template;

[0024] S5, converting the gesture category corresponding to the matched gesture template into corresponding text and voice information;

[0025] S6. According to the gesture recognition results fed back in real time, adjust the parameters of the recognition model to optimize the gesture recognition accuracy;

[0026] S7. In the adjusted recognition model, hardware acceleration technology is used to accelerate the calculation of the dynamic time warping algorithm;

[0027] S8. Optimize the recognition results, use compression algorithms to optimize data transmission and storage, and use real-time operating systems to ensure that the system completes the task within the specified time.

[0028] Optionally, S2 includes the following specific steps:

[0029] S21, performing background removal and noise elimination on the collected gesture image;

[0030] S22, using an edge detection algorithm to extract contour information of the gesture to identify the basic shape of the gesture, and further performing morphological operations to enhance gesture features;

[0031] S23, extracting the position features of the gesture by calculating the center positions of each part in the gesture image, and tracking the dynamic gesture in the image to generate a position change trajectory;

[0032] S24, extracting angle features of the gesture through angle calculation;

[0033] S25, using speed and acceleration to calculate and extract motion trajectory features of the gesture, including speed, direction, and change curve of the hand movement;

[0034] S26, converting the extracted gesture position, angle, speed and other features into time series data.

[0035] Optionally, S3 includes the following specific steps:

[0036] S31. Construct a cost matrix, where each element of the cost matrix represents the distance between a data point in the input gesture time series and a data point in the pre-stored gesture template time series:

[0037]

[0038] Among them, d ij represents the Euclidean distance, c i represents the i-th point in the input gesture time series, q jrepresents the jth point in the time series of the pre-stored gesture template;

[0039] S32. According to the boundary conditions, continuity and monotonicity of the dynamic time warping algorithm, the minimum cumulative distance of each position in the cost matrix is ​​calculated step by step through the dynamic programming algorithm:

[0040] γ(i,j)=d ij +min(γ(i-1,j),γ(i,j-1),γ(i-1,j-1));

[0041] Among them, γ(i,j) represents the minimum cumulative distance from the starting point (1,1) to the current position (i,j), γ(i-1,j) represents the minimum cumulative distance from the upper adjacent point (i-1,j) to the current position (i,j), γ(i,j-1) represents the minimum cumulative distance from the left adjacent point (i,j-1) to the current position (i,j), and γ(i-1,j-1) represents the minimum cumulative distance from the upper left adjacent point (i-1,j-1) to the current position (i,j);

[0042] S33, searching for a minimum cost path from the starting point to the end point in the cost matrix, wherein the path corresponds to the best match between the input gesture and the template gesture:

[0043] W={(i1,j1),(i2,j2),...,(i t ,j t )};

[0044] Where W represents the minimum cost path selected from the cost matrix, i t represents the position of the tth point on the path in the input gesture time series, j t Indicates the position of the t-th point on the path in the pre-stored gesture template time series;

[0045] S34. Calculate the similarity between the input gesture and the template gesture based on the selected minimum cost path.

[0046] Optionally, S4 includes the following specific steps:

[0047] S41, constructing a matching scoring model, and calculating a matching score between the input gesture and each template gesture based on the matching result of the dynamic time warping algorithm;

[0048] S42, normalize the matching score, convert the matching score into a similarity value between 0 and 1, and remove the template gestures below a preset similarity threshold:

[0049]

[0050] Among them, S(x,y) is the normalized similarity value, and D(x,y) represents the total matching score between the input gesture and the template gesture;

[0051] S43, sorting the normalized similarities, and arranging the matching templates in descending order;

[0052] S44, when the similarity values ​​of multiple template gestures are the same, the dynamic gesture template is preferentially selected according to the gesture type priority rule in the template library;

[0053] S45, modifying the initially selected gesture template according to the dynamic characteristics of the input gesture and the user's personalized parameters, and re-determining the category of the matching gesture:

[0054] S ′ (x,T)=S(x,T)+w u ·P u +w m ·P m ;

[0055] Among them, S ′ (x,T) represents the corrected similarity value, P u Contribution value for user personalized parameters, P m is the matching contribution value of the dynamic feature, w u and w m is the weight coefficient;

[0056] S46. Update the confirmed gesture category and its matching result to the template library through the user feedback mechanism, and adjust the priority weight of the relevant template in the template library.

[0057] Optionally, S5 includes the following specific steps:

[0058] S51, obtaining the gesture category corresponding to the template according to the selected best matching template;

[0059] S52, converting the category of the selected gesture template into corresponding text and voice information, wherein the text information includes a description of the selected gesture, and the voice information is generated by speech synthesis technology;

[0060] S53, judging the specific meaning of the input gesture according to the category of the gesture template, and selecting text or voice information matching the category for output;

[0061] S54, if the category of the selected template is a dynamic gesture category, outputting the dynamic meaning of the gesture through speech synthesis technology;

[0062] S55: If the category of the selected template is a static gesture category, convert the gesture category into corresponding text, and output corresponding text information for display or interaction by the user.

[0063] Optionally, S6 includes the following specific steps:

[0064] S61, adjusting the weights and biases in the neural network through a back propagation algorithm according to the gesture recognition result fed back in real time;

[0065] S62. During the adjustment process, the error of the current model is calculated, and the gradient of each weight and bias is calculated by the error back propagation algorithm:

[0066]

[0067] Among them, ∈ represents the error function, y i is the true label, is the predicted value, n represents the total number of samples in the data set;

[0068] S63. Update weights and biases in neural networks using the gradient descent algorithm:

[0069]

[0070] Among them, w new Represents the updated weight value, w old represents the current weight value, η represents the learning rate, represents the partial derivative of the error function with respect to the weight, represents the partial derivative of the error function with respect to the bias, b new represents the updated bias value, b old To indicate the current bias value;

[0071] S64. Use the updated weights and biases to continue optimizing the neural network.

[0072] Optionally, S7 includes the following specific steps:

[0073] S71, constructing a hardware acceleration model of the dynamic time warping algorithm, decomposing the cost matrix calculation task into multiple independent subtasks, and assigning them to the hardware acceleration unit for parallel processing;

[0074] S72, transferring the computing core of the dynamic time warping algorithm to the hardware accelerator, optimizing the computing process based on the pipeline structure, and reducing the computing delay by calculating each row of the cost matrix layer by layer;

[0075] S73, using the shared storage mechanism of the hardware acceleration unit to cache intermediate calculation results in the dynamic time warping algorithm, thereby reducing repeated calculations and improving overall calculation efficiency;

[0076] S74. Improving the path search efficiency of the dynamic time warping algorithm by using the parallel computing resources in the hardware acceleration module:

[0077]

[0078] Among them, T CPU is the CPU-based algorithm calculation time, T Accelerator is the computing time after hardware acceleration;

[0079] S75. Design an adaptive task scheduling algorithm for different hardware architectures to dynamically allocate computing tasks to hardware resources to optimize resource utilization and computing efficiency;

[0080] S76, combining the hardware acceleration module and the main control unit, using a distributed computing method to divide and distribute the computing tasks of the dynamic time warping algorithm to multiple hardware modules for parallel processing;

[0081] S77. Based on hardware acceleration technology, the minimum path search process in the dynamic time warping algorithm is optimized to reduce the time complexity required for path search.

[0082] Optionally, S8 includes the following specific steps:

[0083] S81, compressing the gesture data using the LZ77 algorithm compression algorithm to reduce the storage space and bandwidth requirements of the data;

[0084] S82. Model the frequency of gesture data using the Huffman coding method, generate a set of coding trees, and calculate the average length of the coding so that frequently occurring data uses shorter coding:

[0085]

[0086] Among them, L avg is the average length of the code, p i is the probability of symbol i, l i is the code length corresponding to symbol i, and n is the total number of symbols;

[0087] S83, compressing the input data using the LZ77 algorithm, replacing identical byte sequences with pointers pointing to previously occurring sequences, and generating a dictionary of pointers and lengths;

[0088] S84. During data transmission, the compressed data is encoded using a transmission protocol, and error detection and correction techniques are used to ensure the integrity of data transmission;

[0089] S85. In embedded systems, a real-time operating system is used to schedule the compression and transmission processes.

[0090] The beneficial effects of the present invention are:

[0091] The beneficial effect of the present invention is that the accuracy, real-time performance and adaptability of the sign language recognition system are significantly improved through a sign language recognition method based on a dynamic time warping (DTW) algorithm. By using a dynamic time warping algorithm, the method can effectively handle the nonlinear alignment of gesture time series, and solve the recognition difficulties of traditional gesture recognition methods when faced with gestures with speed changes, inconsistent timing and unequal lengths. Compared with traditional image processing-based methods, the present invention reduces the computational complexity, directly processes time series data, avoids errors that may occur in image preprocessing and feature extraction, and thus improves the accuracy of gesture recognition.

[0092] On this basis, the present invention uses hardware acceleration technology to optimize the calculation process of the dynamic time warping algorithm, significantly improving the processing speed of the system. Hardware acceleration enables the cost matrix calculation and the solution of the minimum cost path to be completed quickly in gesture recognition tasks with high computing density, solving the bottleneck of existing methods in terms of real-time performance. Through parallel computing and optimized dynamic planning processes, the system can maintain efficient real-time response in complex gesture recognition tasks and meet the real-time requirements of embedded platforms.

[0093] In addition, the present invention optimizes data transmission and storage through data compression technology, reduces the bandwidth requirements and storage space of the system, and improves the efficiency and stability of data processing. This optimization measure enables the system to operate efficiently in a resource-constrained environment, and is particularly suitable for embedded devices, smart watches and other mobile devices, further improving the practicality and flexibility of sign language recognition.

[0094] In summary, the present invention not only solves the problems of high computational complexity, poor real-time performance, and insufficient adaptability in the prior art, but also provides an efficient, real-time, and accurate sign language recognition solution through hardware acceleration, data compression optimization, and real-time feedback mechanism, which has stronger practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0096] Figure 1 This is a flow chart of a sign language recognition system based on a dynamic time warping algorithm proposed by the present invention;

[0097] Figure 2 This is a schematic diagram of the hardware acceleration process of a sign language recognition system based on a dynamic time warping algorithm proposed by the present invention. DETAILED DESCRIPTION

[0098] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0099] refer to Figure 1-2 , a sign language recognition system based on dynamic time warping algorithm, comprising:

[0100] A data acquisition module, used to collect hand motion images or gesture data in real time through a camera and a depth sensor;

[0101] Feature extraction and time series conversion module, used to convert gesture data from static images into time series data;

[0102] A dynamic time warping matching module is used to match the input gesture time sequence with the gesture template stored in the database using a dynamic time warping algorithm;

[0103] The matching result selection and template classification module is used to select the template that best matches the input gesture based on the similarity calculated by the dynamic time warping algorithm, and determine the gesture category corresponding to the template;

[0104] Embedded platform and hardware acceleration module, used to execute sign language recognition algorithm on embedded platform and improve recognition speed and real-time performance through hardware acceleration;

[0105] The text-speech output and real-time feedback optimization module is used to convert recognition results into text and speech output for real-time user interaction.

[0106] In this implementation, the modules are implemented by the following method:

[0107] S1, collect hand motion images and gesture data through cameras and depth sensors;

[0108] S2, preprocessing the collected hand motion images and gesture data, extracting key features of the gesture's motion trajectory, position, shape, and angle, and converting the extracted features into time series data;

[0109] S3, using a dynamic time warping algorithm, matching the time series data with the time series data in a pre-stored gesture template library, and calculating the similarity between each gesture template;

[0110] S4. Selecting a template that best matches the input gesture based on the similarity between each gesture template, and determining the gesture category corresponding to the template;

[0111] S5, converting the gesture category corresponding to the matched gesture template into corresponding text and voice information;

[0112] S6. According to the gesture recognition results fed back in real time, adjust the parameters of the recognition model to optimize the gesture recognition accuracy;

[0113] S7. In the adjusted recognition model, hardware acceleration technology is used to accelerate the calculation of the dynamic time warping algorithm;

[0114] S8. Optimize the recognition results, use compression algorithms to optimize data transmission and storage, and use real-time operating systems to ensure that the system completes the task within the specified time.

[0115] In this implementation, S2 includes the following specific steps:

[0116] S21, performing background removal and noise elimination on the collected gesture image;

[0117] S22, using an edge detection algorithm to extract contour information of the gesture to identify the basic shape of the gesture, and further performing morphological operations to enhance gesture features;

[0118] S23, extracting the position features of the gesture by calculating the center positions of each part in the gesture image, and tracking the dynamic gesture in the image to generate a position change trajectory;

[0119] S24, extracting angle features of the gesture through angle calculation;

[0120] S25, using speed and acceleration to calculate and extract motion trajectory features of the gesture, including speed, direction, and change curve of the hand movement;

[0121] S26, converting the extracted gesture position, angle, speed and other features into time series data.

[0122] In this implementation, S3 includes the following specific steps:

[0123] S31. Construct a cost matrix, where each element of the cost matrix represents the distance between a data point in the input gesture time series and a data point in the pre-stored gesture template time series:

[0124]

[0125] Among them, d ij represents the Euclidean distance, c i represents the i-th point in the input gesture time series, q j represents the jth point in the time series of the pre-stored gesture template;

[0126] S32. According to the boundary conditions, continuity and monotonicity of the dynamic time warping algorithm, the minimum cumulative distance of each position in the cost matrix is ​​calculated step by step through the dynamic programming algorithm:

[0127] γ(i,j)=d ij +min(γ(i-1,j),γ(i,j-1),γ(i-1,j-1));

[0128] Among them, γ(i,j) represents the minimum cumulative distance from the starting point (1,1) to the current position (i,j), γ(i-1,j) represents the minimum cumulative distance from the upper adjacent point (i-1,j) to the current position (i,j), γ(i,j-1) represents the minimum cumulative distance from the left adjacent point (i,j-1) to the current position (i,j), and γ(i-1,j-1) represents the minimum cumulative distance from the upper left adjacent point (i-1,j-1) to the current position (i,j);

[0129] S33, searching for a minimum cost path from the starting point to the end point in the cost matrix, wherein the path corresponds to the best match between the input gesture and the template gesture:

[0130] W={(i1,j1),(i2,j2),...,(i t ,j t )};

[0131] Where W represents the minimum cost path selected from the cost matrix, i t represents the position of the tth point on the path in the input gesture time series, j t Indicates the position of the t-th point on the path in the pre-stored gesture template time series;

[0132] S34. Calculate the similarity between the input gesture and the template gesture based on the selected minimum cost path.

[0133] In this implementation, S4 includes the following specific steps:

[0134] S41, constructing a matching scoring model, and calculating a matching score between the input gesture and each template gesture based on the matching result of the dynamic time warping algorithm;

[0135] S42, normalize the matching score, convert the matching score into a similarity value between 0 and 1, and remove the template gestures below a preset similarity threshold:

[0136]

[0137] Among them, S(x,y) is the normalized similarity value, and D(x,y) represents the total matching score between the input gesture and the template gesture;

[0138] S43, sorting the normalized similarities, and arranging the matching templates in descending order;

[0139] S44, when the similarity values ​​of multiple template gestures are the same, the dynamic gesture template is preferentially selected according to the gesture type priority rule in the template library;

[0140] S45, modifying the initially selected gesture template according to the dynamic characteristics of the input gesture and the user's personalized parameters, and re-determining the category of the matching gesture:

[0141] S ′ (x,T)=S(x,T)+w u ·P u +w m ·P m ;

[0142] Among them, S ′ (x,T) represents the corrected similarity value, P u Contribution value for user personalized parameters, P m is the matching contribution value of the dynamic feature, w u and w m is the weight coefficient;

[0143] S46. Update the confirmed gesture category and its matching result to the template library through the user feedback mechanism, and adjust the priority weight of the relevant template in the template library.

[0144] In this implementation, S5 includes the following specific steps:

[0145] S51, obtaining the gesture category corresponding to the template according to the selected best matching template;

[0146] S52, converting the category of the selected gesture template into corresponding text and voice information, wherein the text information includes a description of the selected gesture, and the voice information is generated by speech synthesis technology;

[0147] S53, judging the specific meaning of the input gesture according to the category of the gesture template, and selecting text or voice information matching the category for output;

[0148] S54, if the category of the selected template is a dynamic gesture category, outputting the dynamic meaning of the gesture through speech synthesis technology;

[0149] S55: If the category of the selected template is a static gesture category, convert the gesture category into corresponding text, and output corresponding text information for display or interaction by the user.

[0150] In this implementation, S6 includes the following specific steps:

[0151] S61, adjusting the weights and biases in the neural network through a back propagation algorithm according to the gesture recognition result fed back in real time;

[0152] S62. During the adjustment process, the error of the current model is calculated, and the gradient of each weight and bias is calculated by the error back propagation algorithm:

[0153]

[0154] Among them, ∈ represents the error function, y i is the true label, is the predicted value, n represents the total number of samples in the data set;

[0155] S63. Update weights and biases in neural networks using the gradient descent algorithm:

[0156]

[0157] Among them, w new Represents the updated weight value, w old represents the current weight value, η represents the learning rate, represents the partial derivative of the error function with respect to the weight, represents the partial derivative of the error function with respect to the bias, b new represents the updated bias value, b old To indicate the current bias value;

[0158] S64. Use the updated weights and biases to continue optimizing the neural network.

[0159] In this implementation, S7 includes the following specific steps:

[0160] S71, constructing a hardware acceleration model of the dynamic time warping algorithm, decomposing the cost matrix calculation task into multiple independent subtasks, and assigning them to the hardware acceleration unit for parallel processing;

[0161] S72, transferring the computing core of the dynamic time warping algorithm to the hardware accelerator, optimizing the computing process based on the pipeline structure, and reducing the computing delay by calculating each row of the cost matrix layer by layer;

[0162] S73, using the shared storage mechanism of the hardware acceleration unit to cache intermediate calculation results in the dynamic time warping algorithm, thereby reducing repeated calculations and improving overall calculation efficiency;

[0163] S74. Improving the path search efficiency of the dynamic time warping algorithm by using the parallel computing resources in the hardware acceleration module:

[0164]

[0165] Among them, T CPU is the CPU-based algorithm calculation time, T Acceleratoris the computing time after hardware acceleration;

[0166] S75. Design an adaptive task scheduling algorithm for different hardware architectures to dynamically allocate computing tasks to hardware resources to optimize resource utilization and computing efficiency;

[0167] S76, combining the hardware acceleration module and the main control unit, using a distributed computing method to divide and distribute the computing tasks of the dynamic time warping algorithm to multiple hardware modules for parallel processing;

[0168] S77. Based on hardware acceleration technology, the minimum path search process in the dynamic time warping algorithm is optimized to reduce the time complexity required for path search.

[0169] In this implementation, S8 includes the following specific steps:

[0170] S81, compressing the gesture data using the LZ77 algorithm compression algorithm to reduce the storage space and bandwidth requirements of the data;

[0171] S82. Model the frequency of gesture data using the Huffman coding method, generate a set of coding trees, and calculate the average length of the coding so that frequently occurring data uses shorter coding:

[0172]

[0173] Among them, L avg is the average length of the code, p i is the probability of symbol i, l i is the code length corresponding to symbol i, and n is the total number of symbols;

[0174] S83, compressing the input data using the LZ77 algorithm, replacing identical byte sequences with pointers pointing to previously occurring sequences, and generating a dictionary of pointers and lengths;

[0175] S84. During data transmission, the compressed data is encoded using a transmission protocol, and error detection and correction techniques are used to ensure the integrity of data transmission;

[0176] S85. In embedded systems, a real-time operating system is used to schedule the compression and transmission processes.

[0177] Embodiment 1:

[0178] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a smart watch. The purpose is to provide a more convenient sign language recognition service for the deaf-mute. In daily life, the deaf-mute often face the difficulty of communicating with non-sign language users. Traditional sign language recognition methods mostly rely on computer vision and large-scale gesture template libraries, but these methods have poor recognition accuracy in dynamic environments, high computational complexity, poor real-time performance, and are difficult to run in embedded devices with limited resources. To this end, we designed a sign language recognition method based on a dynamic time warping algorithm. By optimizing the calculation process, hardware acceleration and data compression technology, sign language recognition can be efficiently run on embedded devices such as smart watches.

[0179] In this embodiment, the smart watch collects gesture image data through the built-in camera and depth sensor, and converts the data into a time series. Then, the gesture data is matched with the pre-stored gesture template through the dynamic time warping algorithm, and the recognition results are displayed in real time on the watch screen or output to non-sign language users through voice. The whole process needs to be completed in a short time, so how to improve recognition accuracy and computing efficiency becomes the key.

[0180] In actual use, the deaf-mute person only needs to make a gesture in front of the smart watch, and the watch will start collecting gesture data. After the gesture image is preprocessed, the features such as the motion trajectory, position, shape and angle of the gesture are extracted and converted into time series data. Then, the dynamic time warping algorithm matches the gestures in the template library, selects the most similar gesture template, determines the gesture category, and outputs it through voice or text.

[0181] Specifically, the dynamic time warping algorithm calculates the similarity between each pair of input gestures and templates, and finds the optimal matching path through dynamic programming. Due to the limited computing resources in embedded systems, we use hardware acceleration technology to accelerate the cost matrix calculation, thereby greatly improving the processing speed. The data compression algorithm is used to reduce the storage and transmission requirements of gesture data, ensuring that the system can run stably under limited bandwidth and storage space.

[0182] After training and optimization, the watch can complete gesture recognition within 1 second and accurately identify common gestures. The gesture recognition accuracy is as high as over 95%, and it can adapt to the speed changes and shape changes of the user's gestures. In order to verify the beneficial effects of the present invention, we collected multiple data points in actual applications and compared them with traditional gesture recognition methods. The smart watch uses an ARM architecture-based processor, equipped with GPU and FPGA hardware acceleration. The gesture recognition system is carried out in an indoor environment, collecting various common sign language gestures, and collecting gesture data from 10 deaf-mute people. Each user demonstrated 10 different gestures, with a data volume of 1,000 gestures.

[0183] Table 1 Gesture recognition performance comparison experimental results

[0184]

[0185] As can be seen from the table, the present invention is significantly superior to traditional methods in terms of recognition accuracy, computing efficiency, real-time performance, adaptability and storage optimization. Traditional gesture recognition methods usually rely on image processing and machine learning algorithms, which leads to bottlenecks in real-time performance and computing efficiency. The present invention uses dynamic time warping algorithms, hardware acceleration technology and data compression technology to greatly improve recognition speed and adaptability while ensuring high accuracy. In particular, when running on embedded devices such as smart watches, it can effectively utilize limited computing resources to meet the needs of real-time response.

[0186] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A sign language recognition system based on a dynamic time warping algorithm, characterized in that: include: A data acquisition module, used to collect hand motion images or gesture data in real time through a camera and a depth sensor; Feature extraction and time series conversion module, used to convert gesture data from static images into time series data; A dynamic time warping matching module is used to match the input gesture time sequence with the gesture template stored in the database using a dynamic time warping algorithm; The matching result selection and template classification module is used to select the template that best matches the input gesture based on the similarity calculated by the dynamic time warping algorithm, and determine the gesture category corresponding to the template; Embedded platform and hardware acceleration module, used to execute sign language recognition algorithm on embedded platform and improve recognition speed and real-time performance through hardware acceleration; The text-speech output and real-time feedback optimization module is used to convert recognition results into text and speech output for real-time user interaction.

2. The sign language recognition system based on dynamic time warping algorithm according to claim 1, characterized in that: The modules are implemented in the following ways: S1, collect hand motion images and gesture data through cameras and depth sensors; S2, preprocessing the collected hand motion images and gesture data, extracting key features of the gesture's motion trajectory, position, shape, and angle, and converting the extracted features into time series data; S3, using a dynamic time warping algorithm, matching the time series data with the time series data in a pre-stored gesture template library, and calculating the similarity between each gesture template; S4. Selecting a template that best matches the input gesture based on the similarity between each gesture template, and determining the gesture category corresponding to the template; S5, converting the gesture category corresponding to the matched gesture template into corresponding text and voice information; S6. According to the gesture recognition results fed back in real time, adjust the parameters of the recognition model to optimize the gesture recognition accuracy; S7. In the adjusted recognition model, hardware acceleration technology is used to accelerate the calculation of the dynamic time warping algorithm; S8. Optimize the recognition results, use compression algorithms to optimize data transmission and storage, and use real-time operating systems to ensure that the system completes the task within the specified time.

3. The sign language recognition system based on dynamic time warping algorithm according to claim 2, characterized in that: The S2 comprises the following specific steps: S21, performing background removal and noise elimination on the collected gesture image; S22, using an edge detection algorithm to extract contour information of the gesture to identify the basic shape of the gesture, and further performing morphological operations to enhance gesture features; S23, extracting the position features of the gesture by calculating the center positions of each part in the gesture image, and tracking the dynamic gesture in the image to generate a position change trajectory; S24, extracting angle features of the gesture through angle calculation; S25, using speed and acceleration to calculate and extract motion trajectory features of the gesture, including speed, direction, and change curve of the hand movement; S26, converting the extracted gesture position, angle, speed and other features into time series data.

4. The sign language recognition system based on dynamic time warping algorithm according to claim 2, characterized in that: The S3 includes the following specific steps: S31. Construct a cost matrix, where each element of the cost matrix represents the distance between a data point in the input gesture time series and a data point in the pre-stored gesture template time series: Among them, d ij represents the Euclidean distance, c i represents the i-th point in the input gesture time series, q j represents the jth point in the time series of the pre-stored gesture template; S32. According to the boundary conditions, continuity and monotonicity of the dynamic time warping algorithm, the minimum cumulative distance of each position in the cost matrix is ​​calculated step by step through the dynamic programming algorithm: γ(i,j)=d ij +min(γ(i-1,j),γ(i,j-1),γ(i-1,j-1)); Among them, γ(i,j) represents the minimum cumulative distance from the starting point (1,1) to the current position (i,j), γ(i-1,j) represents the minimum cumulative distance from the upper adjacent point (i-1,j) to the current position (i,j), γ(i,j-1) represents the minimum cumulative distance from the left adjacent point (i,j-1) to the current position (i,j), and γ(i-1,j-1) represents the minimum cumulative distance from the upper left adjacent point (i-1,j-1) to the current position (i,j); S33, searching for a minimum cost path from the starting point to the end point in the cost matrix, wherein the path corresponds to the best match between the input gesture and the template gesture: W={(i1,j1),(i2,j2),...,(i t ,j t )}; Where W represents the minimum cost path selected from the cost matrix, i t represents the position of the tth point on the path in the input gesture time series, j t Indicates the position of the t-th point on the path in the pre-stored gesture template time series; S34. Calculate the similarity between the input gesture and the template gesture based on the selected minimum cost path.

5. The sign language recognition system based on dynamic time warping algorithm according to claim 2, characterized in that: The S4 comprises the following specific steps: S41, constructing a matching scoring model, and calculating a matching score between the input gesture and each template gesture based on the matching result of the dynamic time warping algorithm; S42, normalize the matching score, convert the matching score into a similarity value between 0 and 1, and remove the template gestures below a preset similarity threshold: Among them, S(x,y) is the normalized similarity value, and D(x,y) represents the total matching score between the input gesture and the template gesture; S43, sorting the normalized similarities, and arranging the matching templates in descending order; S44, when the similarity values ​​of multiple template gestures are the same, the dynamic gesture template is preferentially selected according to the gesture type priority rule in the template library; S45, modifying the initially selected gesture template according to the dynamic characteristics of the input gesture and the user's personalized parameters, and re-determining the category of the matching gesture: S ′ (x,T)=S(x,T)+w u ·P u +w m ·P m ; Among them, S ′ (x,T) represents the corrected similarity value, P u Contribution value for user personalized parameters, P m is the matching contribution value of the dynamic feature, w u and w m is the weight coefficient; S46. Update the confirmed gesture category and its matching result to the template library through the user feedback mechanism, and adjust the priority weight of the relevant template in the template library.

6. The sign language recognition system based on dynamic time warping algorithm according to claim 2, characterized in that: The S5 comprises the following specific steps: S51, obtaining the gesture category corresponding to the template according to the selected best matching template; S52, converting the category of the selected gesture template into corresponding text and voice information, wherein the text information includes a description of the selected gesture, and the voice information is generated by speech synthesis technology; S53, judging the specific meaning of the input gesture according to the category of the gesture template, and selecting text or voice information matching the category for output; S54, if the category of the selected template is a dynamic gesture category, outputting the dynamic meaning of the gesture through speech synthesis technology; S55: If the category of the selected template is a static gesture category, convert the gesture category into corresponding text, and output corresponding text information for display or interaction by the user.

7. The sign language recognition system based on dynamic time warping algorithm according to claim 2, characterized in that: The S6 comprises the following specific steps: S61, adjusting the weights and biases in the neural network through a back propagation algorithm according to the gesture recognition result fed back in real time; S62. During the adjustment process, the error of the current model is calculated, and the gradient of each weight and bias is calculated by the error back propagation algorithm: Among them, ∈ represents the error function, y i is the true label, is the predicted value, n represents the total number of samples in the data set; S63. Update weights and biases in neural networks using the gradient descent algorithm: Among them, w new Represents the updated weight value, w old represents the current weight value, η represents the learning rate, represents the partial derivative of the error function with respect to the weight, represents the partial derivative of the error function with respect to the bias, b new represents the updated bias value, b old To indicate the current bias value; S64. Use the updated weights and biases to continue optimizing the neural network.

8. The sign language recognition system based on dynamic time warping algorithm according to claim 2, characterized in that: The S7 comprises the following specific steps: S71, constructing a hardware acceleration model of the dynamic time warping algorithm, decomposing the cost matrix calculation task into multiple independent subtasks, and assigning them to the hardware acceleration unit for parallel processing; S72, transferring the computing core of the dynamic time warping algorithm to the hardware accelerator, optimizing the computing process based on the pipeline structure, and reducing the computing delay by calculating each row of the cost matrix layer by layer; S73, using the shared storage mechanism of the hardware acceleration unit to cache intermediate calculation results in the dynamic time warping algorithm, thereby reducing repeated calculations and improving overall calculation efficiency; S74. Improving the path search efficiency of the dynamic time warping algorithm by using the parallel computing resources in the hardware acceleration module: Among them, T CPU is the CPU-based algorithm calculation time, T Accelerator is the computing time after hardware acceleration; S75. Design an adaptive task scheduling algorithm for different hardware architectures to dynamically allocate computing tasks to hardware resources to optimize resource utilization and computing efficiency; S76, combining the hardware acceleration module and the main control unit, using a distributed computing method to divide and distribute the computing tasks of the dynamic time warping algorithm to multiple hardware modules for parallel processing; S77. Based on hardware acceleration technology, the minimum path search process in the dynamic time warping algorithm is optimized to reduce the time complexity required for path search.

9. The sign language recognition system based on dynamic time warping algorithm according to claim 2, characterized in that: The S8 comprises the following specific steps: S81, compressing the gesture data using the LZ77 algorithm compression algorithm to reduce the storage space and bandwidth requirements of the data; S82. Model the frequency of gesture data using the Huffman coding method, generate a set of coding trees, and calculate the average length of the coding so that frequently occurring data uses shorter coding: Among them, L avg is the average length of the code, p i is the probability of symbol i, l i is the code length corresponding to symbol i, and n is the total number of symbols; S83, compressing the input data using the LZ77 algorithm, replacing identical byte sequences with pointers pointing to previously occurring sequences, and generating a dictionary of pointers and lengths; S84. During data transmission, the compressed data is encoded using a transmission protocol, and error detection and correction techniques are used to ensure the integrity of data transmission; S85. In embedded systems, a real-time operating system is used to schedule the compression and transmission processes.

Images