Vector Phase Analysis and Multichannel Digital Character Synchronization System in Sign Language Translation

Abstract

This invention is an edge-based sign language translation and visualization method developed to address the high processing costs and internet dependency issues associated with raw video footage. The invention is based on the following unique technical steps: Vector Data Reduction (Feature Extraction): Instead of processing the image as a whole, the system converts landmarks into digital rotation and phase data using unique algorithms such as get_finger_curl and jaw_scale. This optimizes data size by 95% compared to standard image processing methods. UDP-Based Real-Time Bridge: This analyzed digital data is transmitted to a digital character engine (Unity / Maya) via a low-latency communication protocol (UDP). Hybrid Synchronization (Lip-Sync & Gesture): This invention instantly matches a character's lip movements (viseme) and facial expressions (smile_power, blink_power) with AI-generated audio using linear interpolation (LERP). Benefit of the Innovation: While existing solutions are cloud server dependent, this system runs at the edge (on the user device), eliminating the need for the internet, protecting privacy, and providing a natural communication experience with near-zero latency.

Description

1 TARIFF Vector Phase Analysis and Multichannel Digital in Sign Language Interpretation CHARACTER SYNCHRONIZATION SYSTEM 1. Technical Field to Which the Invention Relates: The invention relates to artificial intelligence-based image processing, human-machine interaction. It relates to interaction and sign language translation systems. In particular, low camera data. Photorealistic digital characters are created in real-time by converting them into delayed vector data. This includes the method of transmitting and synchronizing this data with voice speech. 2. State of the Art (Prior Technology): Existing sign language translation systems use raw video 10 It transmits and processes data to cloud servers using high bandwidth. This situation leads to internet addiction, high transaction costs, and communication breakdowns. This leads to delays (latency). Additionally, the hand and mouth movements of digital characters... Synchronization between them is generally limited to pre-recorded animations. 3. Technical Problems that the Invention Aims to Solve: The invention aims to solve the technical problems mentioned above. To eliminate the disadvantages; it operates independently of the internet (on-device / edge). By offering capabilities to increase data privacy and speed, and by using vector data reduction methods for data processing. Minimizing data transfer overhead, connecting different programming languages ​​and platforms over UDP. It aims to build a bridge with near-zero latency. 20 4. Explanation of the Figures: Figure 1: Block diagram showing the overall system architecture of the invention. Figure 2: This is a diagram showing the vector landmark analysis and finger bending detail. Figure 3: This graph shows the chronology of the synchronization between the sound signal and lip movements. 25 Reference Numbers in the Figures: 10: Image Capture Unit (Camera) 20: Data Processing and Analysis Module (Edge Device) 21: Landmark Separation Unit 30 22: Vector Data Converter (Finger Bending and Phase Analysis Unit) 30: Data Communication Layer (UDP Socket / Local Bridge) 40: Visualization and Graphics Engine (Maya / Unity) 41: Linear Interpolation (LERP) and Smoothing Module 42: Skeletal Structure (Rig) and Muscular System (Blendshapes) Matching Unit 35 2 50: Voice and Lip Sync (Viseme) Controller 60: Digital Character (3D Avatar) 5. Description of the Invention The working architecture of the invention consists of the following stages: Vector Landmark Analysis: Data received via image capture unit (10), analysis 5 processed by the separation unit (21) in module (20) and delivered to the hand and face joint points The angles of finger bending are separated. By means of a vector data converter (22), the angles of finger bending are numerical. It is converted to a phase vector (Figure 2). UDP Data Packaging: All obtained numerical coordinates are packaged in JSON format, It is transmitted to the visualization engine (40) via a low-latency data communication layer (30). This 10 This allows only digital data to be transmitted instead of raw images, thus saving bandwidth. Interpolation and Synchronization: Data coming to the graphics engine, smoothing module (41) It is applied to the character skeleton by linear interpolation method (42). Simultaneously The sound controller (50) synchronizes the lip movements of the digital character (60) with the generated sound signal. It synchronizes within it (Figure 3). 15 6. Applicability of the Invention The invention can be used on mobile devices and web-based platforms. Especially because it eliminates the need for high-performance hardware, it is suitable for low-spec devices. It enables real-time sign language translation even on devices.

Claims

3 REQUESTS Vector Phase Analysis and Multichannel Digital in Sign Language Interpretation CHARACTER SYNCHRONIZATION SYSTEM 1. A method for sign language interpretation, characterized by: A video capture unit (camera). The raw video data received from the device is processed in real-time on a local processing unit (edge ​​device). decomposition of landmark coordinates, and the decomposition of these coordinates into their respective coordinates. vectorized data consisting of the bending and rotation angles of the joints (landmarks) conversion, the resulting digital data over a local area protocol (UDP) The data is transmitted to a digital character engine in asynchronous packets, and these transmitted data are then digitally processed. Real-time 10 by applying linear interpolation (LERP) method to the character skeleton It includes the steps for visualization.

2. A method that conforms to Claim 1, and its characteristic is that the vector data transformation step in question is done manually. a unique finger bending algorithm that calculates the degree of finger bending It contains (get_finger_curl).

3. A method that complies with Claim 1, characterized by the lip movements (viseme) of the digital character, 15 linearly with the phase and frequency values ​​of the AI ​​voice generated simultaneously by the system. It is about synchronization.

4. A method that complies with Claim 1, and its feature is that it extracts facial expressions from the user's facial features. data (smile_power, blink_power, jaw_scale) of the digital character's facial muscles in the character engine It is the real-time mapping to the system (blendshapes). 20 5. This is a method that complies with Claim 1 and is characterized by its serverless data transmission. This is done via a local socket, without relying on the internet. It is the ability to work.

Images