HOW THE BIBLE-SIDED TRANSLATION SYSTEM WORKS
The bidirectional translation system addresses inefficiencies in sign language translation by using advanced computer vision and speech recognition techniques to provide real-time, accurate, and interactive sign-to-speech conversion with enhanced visual feedback and learning features.
Patent Information
- Application Number
- TR202512543
- Authority / Receiving Office
- TR · TR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-09-02
- Publication Date
- 2026-10-21
AI Technical Summary
Current sign language translation technologies are limited by one-way operation, real-time speed and stability issues, lack of visual feedback, limited language support, missing time series comparison, hand matching and data skipping operations, and lack of caching and data quality filtering, making them inefficient for daily communication and learning.
A bidirectional translation system using computer vision, speech recognition, and visual gesture generation, employing MediaPipe Holistic for key point detection, PyTorch for tensor processing, Soft-DTW for time series alignment, and caching for high-quality data management, to provide real-time visual and audio feedback of sign language gestures.
Enables efficient, real-time, and accurate two-way translation of sign language into spoken language, with improved visual feedback and learning capabilities, reducing reliance on interpreters and expanding language support.
Abstract
Description
HOW THE BIBLE-SIDED TRANSLATION SYSTEM WORKS TECHNICAL FIELD The invention translates sign language gestures into spoken language for presentation in audio and / or written form. It is a bidirectional system that translates spoken language data, received in audio and / or written form, into sign language. The translation system is related to its working method. STATE OF THE ART In the current state of technology, communication via sign language is particularly relevant during live video streaming. Translations are used. However, in daily life, direct communication is only possible with people who know sign language. It is possible to establish this. People who communicate using sign language can go to the bank, get an examination, etc. extremely simple for everyday life such as shopping, ordering, and asking for directions. Even finding someone to deal with during the process is extremely challenging. It is emerging. Furthermore, the reading perception of hearing-impaired individuals (due to the lack of internal sounds, grammatical structure and suffixes / roots) is also affected. (low) their understanding of the rules, not high, only a very small percentage of the texts they read This does not lead to them understanding. Another problem with the use of sign language is that each just like spoken languages / dialects / regional variations, there are differences across countries and geographical areas. It is a demonstration. It is estimated that there are more than 130 different types of sign languages in the world. To solve this problem, applications have been developed that translate various sign languages into text. There are companies on the market that recognize sign language and / or convert signs to text / speech. The disadvantages of these applications can be listed as follows: 1. One-Way Operation: They only work in the "sign → text / sound" direction; "sound / text → There is generally no reverse translation towards "visualized sign". 2. Real-Time Speed and Stability Issues: Hand tracking in live video streaming Recognition in cases of disappearance, overlap, or signaling at different speeds. It is unstable. 3. Lack of Visual Feedback: The sign lacks a 3D avatar, 2D animation, or video of a person. It lacks a visual representation, which reduces its visual appeal and ease of learning. 1 4. Limited Language / Alphabet Support: It generally focuses on a single set of characters and is a reference. It is not possible to easily expand the database. 5. Lack of Specific Time Series Comparison: Most solutions lack features like DTW / Soft-DTW. Algorithms are not used; therefore, it is done at different speeds or with different movement paths. The matching of signals will be weak. 6. Missing Hand Matching and Data Skipping Operations: Usually based on the "last known good" logic. This is not applicable; instances where different hands (left / right) are used in the recording / reference will not be processed. 7. Lack of Caching and Data Quality Filtering: This is mostly due to embeddings. eliminating low-quality records such as those requiring caching or a handheld visibility threshold They lack mechanisms; this both slows down the process and reduces accuracy. The invention described in document number TR 2022 / 000427 in the known state of the art is for the hearing impaired. to help individuals solve problems they experience in their social lives and to connect those who do not know sign language with others. to all platforms and advanced technologies that do not offer sign language options at the time Texts / sounds that can be integrated into animations, live streams An instant sign language translation method for converting movements / videos into text / audio. It is related to. In another document in the known field of technology, document number TR 2015 / 02053, spoken language is discussed. Converting (voice) to sign language and converting sign language to spoken language (voice) from the device It is mentioned. Patent application number TR 2015 / 13151 describes "Instant communication for visually and hearing impaired individuals." The "translation system" is described. The invention is for individuals with visual and / or both visual and hearing impairments. It relates to an instant translation system. The system is a place where the audio data to be translated is recorded. a voice recording unit, a speech-to-text transcription unit that converts recorded voice data into text, text a translation unit that translates the converted data into the desired language, the translated data A Braille conversion unit that converts to Braille characters, and Braille characters themselves. It is characterized by containing a Braille device that enables its display. (As mentioned above) The patent application described outlines a device for interpreting hearing-impaired individuals. The product provides speech-to-text and text-to-speech conversion for the hearing impaired. (For the hearing impaired) It does not perform an image transformation. Document number US2022327309 (A1) describes communication systems, methods and non-temporary communication. Machine-readable storage media are described here. A communication system, Real-time communication within a single communication device or between multiple communication devices. during the session sign language content and any content indicators associated with the video stream It may include a communication device configured to receive a video stream. communication system 2 Additionally, assistance from a human sign language interpreter during the real-time communication session. It is configured to automatically translate sign language content into spoken word content without needing to. It may also include a translation engine. Furthermore, the communication system can handle real-time communication sessions during the session. The word content translation can be configured to be output to a communication device. Invention KR20220126305 (A) relates to a sign language translation device. More specifically... Thus, the present invention is a sign used by a person with hearing impairment and speech disorder. a language that recognizes the movement of the tongue and enables communication with a person who does not know sign language. A sign language translation system that makes sign language education possible by translating and providing a translated sentence. It relates to the device. The sign language translation device consists of a sign language recognition glove; a camera; and a sign. It includes a language translation terminal. THE PURPOSE OF THE INVENTION The main purpose of the invention is to present gestures in sign language in an audible and / or written form. It converts spoken language data, received in spoken and / or written form, into sign language. The goal is to establish a working method for a two-way translation system. Another aim of the invention is to provide translation that will make daily life easier for individuals with speech impairments. The goal is to establish the system's working method. Another aim of the invention is to provide real-time visual feedback to those with speech impairments. a translation system working method that will facilitate communication with individuals to place. Another aim of the invention is to integrate sign language with its interactive, visual, and personalized methodology. The goal is to develop a translation system and working method that will facilitate learning. Another aim of the invention is to reduce reliance on video and live interpreters. The goal is to establish a working method for a translation system. Another purpose of the invention is to create 3D avatars, 2D animations, or videos of real people using sign language. The goal is to establish a translation system and working method that allows for presentation in this manner. DETAILED DESCRIPTION OF THE INVENTION This section is intended solely to help to better understand the subject. Explanations are being provided. 3 The invention involves computer vision technologies, speech recognition, voice synthesis, and visual gesture generation. a human-computer interaction systems that includes adaptive learning and digital pedagogy It relates to the method of enabling two-way communication between sign language users and hearing individuals. and enables users to learn sign language in an interactive, visual and personalized way. to make it possible. Sign language gestures are spoken in a way that is presented orally and / or in written and / or visual form. The language it receives, in the form of spoken language data, in audio and / or written and / or visual form, is called signification. The working method of a bidirectional translation system that translates into another language involves the following steps: It includes: Presenting gestures in sign language in an audio and / or written and / or visual form for, With the help of that camera, real-time video of the hands and the person's movements. recorded as, By processing frames from the video through the Mediapipe Holistic application. the predefined key on the right and left hands Identifying the points, Using MediaPipe Holistic, key hand points are captured and their invariant... A system containing modules for converting features into feature vectors has been developed. utils / mediapipe_utils.py: This file contains helper functions for working with MediaPipe Holistic. Includes: mediapipe_detection(image, model): An image and a MediaPipe model (for example, It accepts mp.solutions.holistic.Holistic. It converts the image from BGR to RGB. (As required by MediaPipe), it works with the model, disables writing, and then again. Converts to BGR. Returns the processed image and MediaPipe results (markers). draw_landmarks(image, results): Hand key points according to MediaPipe results and It uses mp.solutions.drawing_utils to manipulate these connections on the image. It draws the key points and connections of the hands on the image. This module is a fundamental component for extracting visual data from video streams. If those hands and / or key points cannot be identified, the relevant Processing the last determined location data of the hand, those identified key point movements are tracked through the module by Pytorch conversion to tensors, 4 by combining the tensors created separately for the right hand and the left hand, each Obtaining a single large tensor for the category, utils / landmark_utils.py: This file is responsible for extracting and saving keypoint data. is responsible. extract_landmarks_torch(results, device='cpu'): Position, left and right from MediaPipe results. It extracts the key points and converts them into PyTorch tensors. If key points are missing... (e.g., if the hand is not visible), a zero-filled tensor is important for consistent sequence lengths. It rotates. Tensors are moved to the specified device. save_landmarks_from_video(video_name, target_device_array: str = None): with video_name Opens the specified video file. mp.solutions.holistic.Holistic is used to process each frame of the video. is used. For each frame, to obtain the pose, left hand and right hand tensors. The `extract_landmarks_torch` command is called. The tensors of the pose, left hand, and right hand keypoints are obtained. It adds these tensors to the corresponding list (landmarks_sequences). When the video is fully processed, By combining the pose, left-hand, and right-hand tensors separately, a single large tensor is obtained for each category. (e.g., all left-hand tensors). These latter tensors are in a structured index. They are saved as pickle files in (data / dataset / {sign_name} / {video_name} / ). save_array(tensor_arr: torch.Tensor, path: str) and load_array(path: str, device: str = 'cpu'): To save and load PyTorch tensors (or NumPy arrays) into pickle files. Helper functions. `load_array` can also move data to the specified device. This module generates raw gesture data that will later be used in the feature extraction phase. converting the obtained tensors into invariant feature vectors, the absence of sequences within those immutable feature vectors In this case, the empty elements of the vectors are filled with zero data. using the cosine similarity method between those invariant feature vectors calculating similarity, by combining all cosine similarities into a single vector, the hands are separated. Determining the separate entity, location, orientation and size, by calculating the angles between those invariant feature vectors 3 Determining three-dimensional coordinates, models / hand_model.py: This file describes how to convert the key points of the hand into a feature vector. It defines that it will be converted. HandModelTorch(torch.nn.Module): Hand switch points containing 21 points and 3D coordinates. It takes the tensor and calculates an attribute vector. Among the vectors created by hand connections... It uses cosine analogy. Connections (for example, connections between finger joints) Defined by mp.solutions.holistic.HAND_CONNECTIONS. Each pair of connection vectors The cosine similarity is calculated for. All cosine similarities are a single smoothed feature. It is combined into a vector. This approach is invariant to the scale, position, and rotation of the hand. and is critical for robust gesture recognition HandModel(object): An older or alternative method for calculating angles between connection vectors. It is an application. For each pair of linking vectors, the angle between them is calculated. These angles are the same. It creates a feature vector in time. Our code in SignModel uses a tensor instead of a list. It uses the HandModelTorch, which is preferred because it rotates the torch and is more efficient. HandModelTorch extracts semantically meaningful features from raw 3D keypoint coordinates. It plays an important role in its extraction. that video is distorted or has a quality below a predetermined standard. deletion of data If those hands and / or key points cannot be identified, the result is low. Improving image quality by filtering high-quality data, To speed up data processing, the reference gesture found in the database Caching array data, The quality of the reference gesture sequence data stored in the cache is improved by filtering it. increasing utils / dataset_utils.py: This file manages the loading and preprocessing of video data and It also creates a reference motion dataset. self_heal_cleanup(video_name: str): Cleans files related to a video in case of processing errors. It is a helpful function for cleaning up the video file and its associated keypoints / embeddings directory. It deletes them. This is critical for maintaining data integrity. get_video_length(video_path: str): A simple way to get the number of frames in a video. It is a function. load_dataset(processor_string: str = None): Data / videos for new video files (.mp4) It scans the directory. It retrieves data / dataset for previously processed keypoint files (pose_*.pickle). It scans. For videos where key points haven't been extracted yet, it identifies the videos to be processed and... Adds to the videos_to_process list. Uses tqdm to show the progress. It iterates through videos_to_process. 6 For each video, use `save_landmarks_from_video` to extract and save key points. It is called. It loads the saved keypoint data for the left and right hands. hand_visibility_ratio (The non-zero frame rate of the hands visible) This is used to filter out low-quality recordings. This is an important rule for hand_visibility_ratio, MIN_HAND_VISIBILITY_RATIO (0.70). If it's below that level, the video quality is considered low and the data is deleted (self_heal_cleanup). Only High-quality videos are added to videos_for_reference. It returns a list of videos_for_reference. load_reference_signs(videos: list, device: str = 'cuda'): gets the list of video_names (usually (Returned by load_dataset). Iterate over each video_name to reference it. It loads the signals. For each video, pre-calculated embeds (embeddings_{video_name}.pickle) It attempts to load it. This is a significant optimization because the embeddings have already been calculated. There is no need to recalculate each time. Cached placements. If it cannot be found or is corrupted, pickle the raw keypoint data for the left and right hands. It loads from its files. Using raw bookmarks or cached placements. Creates a SignModel object for the reference movement. If the embeds are newly calculated. (If not loaded from cache), it is saved for future reuse. The beacon name is... The SignModel object and data with an initial distance of 0.0 are stored in reference_signs_data. It collects data in the DataFrame. It returns the value pd.DataFrame(reference_signs_data). This module handles motion database processes, including preprocessing, filtering, and caching. It acts as a backend for management, significantly improving performance. The hardware unit that will be used for comparison based on data size is the CPU. Whether it will be a GPU or not is determined via PyTorch, models / sign_model.py: This class represents a single gesture that is both recorded and referenced. It encapsulates. SignModel: Initialized with tensor lists for the left and right hand (key points for each frame). (array). It can take cached_embeddings (pre-calculated feature vectors), which in turn loads. It speeds it up. If there is no cache, it iterates through the frames for each hand (if any), and It uses HandModelTorch to extract the feature vector for each frame. These vectors are then used for each hand. Creates a temporally ordered embedding sequence for (lh_embedding, rh_embedding). The `has_left_hand` and `has_right_hand` attributes indicate whether hands were detected in the recording. SignModel displays hand-placement sequences made ready for comparison using DTW. It combines. 7 those converted invariant feature vectors are found in the database Soft DTW (Dynamic Time Distortion) with reference gesture sequence data. Thanks to its algorithm, they are processed in groups, resulting in different movements and speeds. Even if they possess the same qualities, a separate comparison should be made for both the right and left hands. Dynamic Time Distortion (DTW) is the similarity between two time series that may have different speeds. It is an algorithm used to measure. For example, whether the same movement is faster or slower. This is feasible, and DTW allows for the "distortion" of the time axes of these sequences, It allows you to find the optimum alignment between the points. Soft-DTW is a derivative (smooth) version of classic DTW. Standard DTW, While Soft-DTW uses the non-differentiable minimum operation, it uses the soft-minimum operation. This changes Soft-DTW to be suitable for use in deep learning where backpropagation is necessary. It brings. utils / soft_dtw_batch.py: This module is optimized for batch processing using PyTorch and Numba. Includes the modified Soft-DTW application. compute_softdtw_batch(D, gamma, M_real_lens): with jit(nopython=True) decorator It is a coded Numba function. This allows Python code to be translated into machine code at runtime. By compiling, it significantly speeds up numerical calculations, especially in loops. Parameters: D: The distance tensor between all pairs of points in the aggregate work. D is of the form (B, N, M), where B The size of the batch is N, the length of the first sequence, and M is the maximum length of the second sequence in the batch. gamma: Correction parameter for Soft-DTW. M_real_lens: A tensor containing the real lengths of every second array in the batch process. This This is necessary because arrays can be filled with zeros for batch processing, and DTW is only true to life. It must be calculated down to their lengths. The function is cumulative for Soft-DTW throughout the entire batch. The program applies dynamic programming to compute the matrix R. Inside the loop, `soft-min`: The formula 𝐷[𝑘,𝑖−1,𝑗−1]+(−𝛾𝑙𝑜𝑔(∑𝑒−𝑅 / 𝛾)). soft-min is used. For each pair in the batch job, Soft- Returns the `final_distances` tensor, which contains the DTW distances. This tensor is used for each pair in the batch. Includes Soft-DTW distance. SoftDTW_Batch(Function): A class that inherits from torch.autograd.Function. PyTorch's special forward and backward transitions compatible with automatic differentiation It allows for its identification. 8 forward(ctx, D, gamma, M_real_lens): In the forward pass, it moves the tensors to the CPU and into NumPy arrays. after conversion (as required by Numba), optimized by Numba. It calls the `compute_softdtw_batch` function. The result is then converted back into a PyTorch tensor. It is converted and returned to the original device (CPU / GPU). backward(ctx, grad_output): In our code, the backward transition returns the value None, None, None. It returns; this means that the gradients are not calculated. This is because Soft-DTW It is used solely for similarity assessment and for training the embedded model. It is not used. SoftDTW(nn.Module): This is the main PyTorch module class that uses SoftDTW_Batch. calc_distance_matrix(self, x, y): The even distance between the squares of two sequences of moves in a batch operation. Calculates the quadratic Euclidean distance matrix. The x and y tensors represent the motion placement sequences. does. • forward(self, x, y, y_lens): Basic forward method. Distance D_xy using _calc_distance_matrix. It calculates the matrix and then uses this to calculate the actual Soft-DTW distances. It passes the command to self.func_dtw (instance of _SoftDTW_Batch). For Compute_softdtw_batch, use Numba and a custom one for PyTorch integration. Using `torch.autograd.Function` speeds up DTW calculations on the CPU and batch processing. It is a highly efficient approach to managing operations. utils / dtw.py: This file contains the DTW (Direct Transaction Tracking) function between a saved transaction and reference transactions. It includes the basic logic for calculating distances. soft_dtw_calculator = SoftDTW(gamma=1.0): Initializes the Soft-DTW instance with gamma=1.0. MINI_BATCH_SIZE = 8: Mini batch size for batch processing of reference transactions. This defines efficient GPU usage and memory management when processing a large number of reference movements. This is of critical importance. dtw_distances(...): Parameters: recorded_sign: The SignModel object of the current gesture. reference_signs_df: DataFrame containing reference gestures. 9 Initializes the calculated_distances as a tensor populated with inf on the selected device. The embedded placements of the left (rec_lh_tensor) and right (rec_rh_tensor) hands from the recorded signal. It removes. Separate processing of the hands: The code processes the left and right hands separately. This means that only when both sequences (recorded and reference) use the same hand... It allows for comparison when included. Left hand: Filters reference_signs_df to find gestures involving the left hand (ref_has_lh). Filtered It collects left hand embeddings from reference gestures (ref_lh_list). On ref_lh_list It iterates through batches of size MINI_BATCH_SIZE. For each batch, the batch operation... It uses `pad_sequence` to align the sequence lengths within it and to the maximum length. It fills it with zeros. This is necessary for batch processing in Soft-DTW. Batch comparison To accomplish this, it copies the rec_lh_tensor (the recorded motion) for each item in the batch. The distances between the left hands of the reference movements in the recorded left hand and the left hands in the batch process. It calls soft_dtw_calculator to perform the calculation. For the relevant reference transactions. Updates the calculated distances. The right hand is held similarly to the left hand. Distance summing: If both hands are present in both the recorded and reference movements, DTW The distances are added together. If a hand is missing in the reference but is present in the recorded movement, its distance is calculated. (Infinity) is ignored in the first instance. If both are present, the distances are simply added. Processing hand numbness: Hand configuration (left / right) with recorded movement. A mask called mismatched_hands_mask is used to mark reference movements that don't match. These discrepancies are created. For such discrepancies, the distance is set to float('inf'), which takes them into account. It effectively removes it from being taken. This mask separates the hand set in the recorded gesture from the reference gesture. It indicates situations where the (left / right) sides do not match. For example, the recorded gesture might only involve the left hand, but the reference gesture might involve both the left and right hands. If it contains (or vice versa), these gestures are considered incongruent. These incongruent gestures The distance value is assigned as float('inf'), meaning these are excluded from comparison. The function retrieves the reference_signs_df DataFrame with an added distance column and the distance. It returns them sorted accordingly. This is the central module that performs the actual motion comparisons. Collective and manually separated. Processing significantly increases efficiency. Instead of DTW, HMM, CNN+LSTM, and Transformer-based models can also be used. Thanks to the PyTorch application, matching gestures can be detected, For comparison of recorded gesture sequences with reference examples, Numba was used for accelerated processing, and An optimized DTW (Dynamic Time Warp) suitable for batch processing with PyTorch. The (warping) algorithm is used. The system also handles data management and caching of embedded representations. It includes logic for acquiring and predicting gestures in real time. sign_recorder.py: This file is used to record new actions in real time and for reference. It manages the process of comparing actions. SignRecorder: reference_signs (a DataFrame of reference transactions), seq_len (for analysis) It is initiated with (sequence length in frames) and device. record(): Changes the recording status (is_recording), recorded_results, and current_prediction. It cleans the buffers. It also resets the distances in reference_signs and last_left_hand and last_right_hand cleans the bumpers. process_results(results, threshold=0.5): Current MediaPipe results It adds to self.recorded_results. When recorded_results reaches the self.seq_len value (or recording When it stops, compute_distances is called. It returns the current estimate and logging status. compute_distances(threshold=0.5): This is the basic method for comparing recorded movements. This is the method. It iterates through recorded_results (i.e., all frames of the recorded sequence). Left and right hand landmarks are extracted using extract_landmarks_torch. "Last known good" It uses the logic. If there is no hand data in the current framework (zero tensor), it uses data from the previous valid framework. `self.last_left_hand` or `self.last_right_hand` is used. This is for intermittent tracking in MediaPipe. It helps to cope with loss. sm_lh_activity to assess hand visibility. and calculates sm_rh_activity (total hand "activity"). It sums the left and right hand tensors. It creates a SignModel object (recorded_sign) for the recorded action. It compares the recorded_sign with all reference_signs by calling `dtw_distances` from `utils.dtw`. The result (an ordered Data Frame with distances) updates self.reference_signs. Total hand If the activity is high enough, it calls `_get_sign_predicted` to get a prediction. Otherwise... In that case, it clears current_prediction. For the next window or log, it clears recorded_results. It cleans the bumper. _get_sign_predicted(batch_size=5, threshold=0.5): closest from self.reference_signs It retrieves the maximum batch_size (default 5) of the reference movements (already based on distance). (listed). Filters those with a distance equal to f. These are the most frequent valid signals. 11 Collections.Counter is used to count the name of a jest. It applies a threshold value: Most frequent It returns "Unknown sign" if the sign ratio is below the threshold value. A list of names... It returns a list (or "Unknown marker"). This class interacts with the user, providing real-world functionality. It is the main interface that performs real-time gesture recognition and comparison. those detected gestures, 3D avatar or 2D animation or video or audio or text transcription of the real person making the gestures, the presentation of that converted data by a visual and / or audio unit, Marking spoken language data received in audio and / or written and / or visual form to translate into gestures in the language, the entry of data into the system in audio and / or written and / or visual form, The data received by that system is processed into immutable feature vectors. transformation, those converted invariant feature vectors are found in the database Soft DTW (Dynamic Time Distortion) with reference gesture sequence data. The algorithm allows for the comparison of data by processing them in groups. Thanks to the PyTorch application, matching audio and / or text and / or visual identification of spoken language data, those detected gestures, 3D avatar or 2D animation or video or audio or text transcription of the real person making the gestures, the presentation of that converted data by a visual and / or audio unit. The invention includes a mobile application (iOS / Android) with offline mode, and integration with third-party services. A cloud platform with APIs that can be used by public service centers, hospitals, banks, It can be used with interactive terminals at airports and a web platform for online learning. It can work both online and offline and can be used for teaching gesture language. The general working principle of the invented method can be summarized as follows: Data collection: Video recordings. Preprocessing: Using MediaPipe (utils / landmark_utils.py, utils / mediapipe_utils.py) Key point extraction. Feature extraction: Transform-resistant embedding of key points conversion (models / hand_model.py, models / sign_model.py). Previously It increases processing speed by using calculated embedded data. 12 Data management: Loading, caching, and filtering reference transactions (utils / dataset_utils.py). Comparison: Calculation of Soft-DTW distances between gesture sequences (utils / dtw.py, utils / soft_dtw_batch.py). Real-time recognition: Frame buffering, "last known good" logic Implementation, comparison, and prediction (sign_recorder.py). PyTorch and GPU Usage: The code determines whether calculations are performed on the CPU or the GPU. To determine which functions to run, actively enable PyTorch tensors and device properties. It uses this, especially in batch processing and handling large datasets, for performance. This is of critical importance. Optimized Soft-DTW: Using Numba (@jit(nopython=True)) for the DTW computing core, It significantly speeds up numerical operations that would be slow in Python. To implement Soft-DTW as torch.autograd.Function(_SoftDTW_Batch), Numba-optimized code integrated into PyTorch's computational graph. This allows for the use of reverse transition; however, in this case, the reverse transition is not used. o Batch processing (MINI_BATCH_SIZE): Each recorded transaction is uniquely represented with each reference. Instead of comparing individual transactions, reference transactions are processed collectively. This facilitates the processes. It efficiently utilizes GPU capabilities by parallelizing processes. Batch processing And processing the hands individually speeds up the system and improves recognition accuracy. `pad_sequence`: To fill sequences of equal length within a batch job in PyTorch. It is a standard and efficient practice for handling variable-length arrays. High accuracy with DTW / Soft-DTW algorithms and PyTorch-based models. It offers gesture recognition and speech-to-gesture conversion. Improved hand feature extraction (HandModelTorch): Between link vectors The calculation of cosine similarity takes into account the position, orientation, and size of the hand. It is a robust method for obtaining immutable features. This allows the algorithm to work differently. It significantly improves the ability to compare gestures made in various ways. Robustness and fault management: o self_heal_cleanup: Automatically cleans corrupted or low-quality video data. It deletes. o Hand presence check: SignModel and dtw_distances are actively set to the left and / or right It checks for the presence of hands and adapts the comparison logic accordingly. Intelligent. High recognition accuracy is achieved through array comparison. 35 o Visibility filtering: MIN_HAND_VISIBILITY_RATIO in load_dataset, by helping to filter low-quality recordings where hands are not clearly visible 13 It improves the quality of the reference dataset. Filtering that checks for hand matching, processing references with the wrong hand combination (left / right) obstacles. That "last known good" logic: in compute_distances, the last valid hand for missing frames. The location is reused, making the system more resistant to temporary monitoring errors. It makes it durable. Caching and placement: Expensive feature extraction only once. Since this was done, the loading of the reference movements is largely reduced. It speeds things up. Modular structure: The code is well organized into modules, each responsible for a specific task. It is configured (MediaPipe, Landmark Utils, Models, DTW, Dataset Utils, SignRecorder). Additionally, the integrated training mode allows the user to practice the gestures and determine whether they execute them correctly. It enables control and is interactive for users who want to learn gesture language. A learning environment is created. In these respects, it differs from existing techniques only in that it is not based on translation. It not only does that, but it also facilitates the learning and visualization of gesture language and seamless communication. It makes integration possible. 14
Claims
1. To present gestures in sign language in an audible and / or written and / or visual form. spoken language, the spoken language it receives in an oral and / or written and / or visual form. It is a bidirectional translation system that translates data into sign language, and its working method is... feature; Presenting gestures in sign language in an audio and / or written and / or visual form for, With the help of that camera, real-time video of the hands and the person's movements. recorded as, By processing frames from the video through the Mediapipe Holistic application. the predefined key on the right and left hands Identifying the points, those identified key point movements are tracked through the module by Pytorch conversion to tensors, converting the obtained tensors into invariant feature vectors, those converted invariant feature vectors are found in the database Soft DTW (Dynamic Time Distortion) with reference gesture sequence data. Thanks to its algorithm, they are processed in groups, resulting in different movements and speeds. Even if they possess the same qualities, a separate comparison should be made for both the right and left hands. Thanks to the PyTorch application, matching gestures can be detected, those detected gestures, 3D avatar or 2D animation or video or audio or text transcription of the real person making the gestures, the presentation of that converted data by a visual and / or audio unit, Marking spoken language data received in audio and / or written and / or visual form to translate into gestures in the language, the entry of data into the system in audio and / or written and / or visual form, The data received by that system is processed into immutable feature vectors. transformation, those converted invariant feature vectors are found in the database Soft DTW (Dynamic Time Distortion) with reference gesture sequence data. The algorithm allows for the comparison of data by processing them in groups. Thanks to the PyTorch application, matching audio and / or text and / or visual identification of spoken language data, 35 of the detected gestures were either 3D avatars or 2D animations or video or audio or text transcription of the real person making the gestures, the presentation of that converted data by a visual and / or audio unit, It includes the steps of the process.
2. This is a working method that complies with Claim 1, and its characteristic feature is to speed up data processing. The detected key point movements are converted to Pytorch tensors via the module. After conversion, the pose shows the tensors created separately for the right hand and the left hand. the process step of combining them to obtain a single large tensor for each category It includes.
3. A working method that is suitable for any of the above requirements and whose characteristic is; To equalize the data sizes to be processed, the resulting tensors have invariant attributes. After being converted into vectors, the invariant feature vectors contain In case of missing elements in the arrays, the empty elements of the vectors are replaced with zero data. It includes the step of filling out the form.
4. A working method that is suitable for any of the above requirements and whose characteristic is; obtaining After the resulting tensors are converted into invariant feature vectors, using the cosine similarity method between invariant feature vectors calculating similarity, Combining all cosine similarities into a single vector to separate the hands Determining its existence, location, orientation, and size. 3D modeling by calculating the angles between invariant feature vectors Determining the coordinates, It includes the steps of the process.
5. A working method that is suitable for any of the above requirements and whose characteristic is; defective or deleting video data below a predetermined image quality It includes the process step.
6. A working method suitable for any of the above requirements, characterized by the use of hands. and / or inability to identify key points, resulting in low-quality data. This process involves improving image quality through filtering.
7. A working method suitable for any of the above requirements, characterized by the use of hands. and / or in the event that key points cannot be identified, the last identified data for the relevant hand. The process involves processing 35 location data points. 16 8. A working method that is suitable for any of the above requirements and whose characteristic is; data To speed up processing, the reference gesture sequences in the database are pre-processed. This includes the step of storing the information in memory.
9. A working method that is suitable for any of the above requirements and whose characteristic is; prior The process of filtering and improving the quality of reference gesture sequences stored in memory. It includes the step.
10. A working method that is suitable for any of the above requirements and whose characteristic is; Thanks to the PyTorch application, data is obtained before matching gestures are detected. Which hardware unit will be used for comparison based on size: CPU or GPU? This includes the step of determining what will happen using PyTorch. 17
