Method and system for generating a unified script code (USC) representation for multilingual text processing

Abstract

The present invention provides a computer-implemented method and system for generating a Unified Script Code (USC) representation of multilingual text to enable consistent, script-neutral, and phonemically accurate encoding across diverse languages and writing systems The system converts Unicode-encoded text from one or more Indic scripts into a Devanagari-based intermediate form using predefined or bitwise mapping. It then normalizes the text by inserting inherent vowels, converting dependent vowel signs (matras) into independent vowels, and removing halant characters. The resulting USC representation explicitly encodes consonant-vowel sequences, reduces script specific variation, and preserves phonetic integrity. This approach improves tokenization efficiency, enhances performance in natural language processing and machine learning tasks, and supports reversible conversion to original scripts.

Description

METHOD AND SYSTEM FOR GENERATING A UNIFIED SCRIPT CODE (USC) REPRESENTATION FOR MULTILINGUAL TEXT PROCESSING BACKGROUND

[0001] The present invention generally relates to text processing and natural language processing (NLP) systems. The present invention more particularly relates to a method and system for generating a unified script code (USC) representation of multilingual text to enable consistent, script-neutral, and phonemically accurate encoding across diverse languages and writing systems.

[0002] Textual data processed by computer programs is generally represented using standardized character encoding schemes, such as the Unicode standard, which provide a universal numerical representation of characters across a wide range of languages and scripts. In many applications, particularly those involving large-scale text analysis, natural language processing (NLP), or machine learning, such standard encodings are further transformed into custom encodings in order to improve computational performance or processing efficiency. Custom encodings may be optimized for specific tasks or generalized to enhance performance across a variety of use cases. A widely adopted approach to such encoding is tokenization, wherein textual content is segmented into discrete tokens that serve as computational units for downstream processing tasks.

[0003] However, conventional tokenization techniques exhibit inherent limitations when applied to multilingual text represented in multiple scripts. Because such approaches are typically data-driven, the resulting token vocabularies are often biased toward languages that are more heavily represented in the training data. Consequently, tokenization efficiency for less-represented languages can degrade significantly, leading to poor linguistic representation and reduced model performance for such languages. This imbalance is further intensified by the global dominance of English text, causing tokenization algorithms to disproportionately optimize for English while generating suboptimal token boundaries for languages written in other scripts, including but not limited to Indic scripts.

[0004] As a result, languages with smaller textual footprints suffer a dual disadvantage: limited data availability and inefficient tokenization, both of which negatively impact the accuracy and effectiveness of machine learning models trained on multilingual data. Furthermore, existing solutions fail to adequately address the structural and phonetic differences inherent in non-Latin, phonetic writing systems, thereby impeding equitable tokenization performance across languages.

[0005] Thus, there exists a need for a system and method that enable consistent, script- neutral, and phonemically accurate text representation across diverse languages and scripts, thereby improving tokenization efficiency and enhancing the performance of multilingual computational models.

[0006] The above-mentioned shortcomings, disadvantages and problems are addressed herein, and will be understood by reading and studying the following specification. SUMMARY

[0007] This summary is provided to introduce a selection of concepts in a simplified form that are further disclosed in the detailed description. This summary is not intended to determine the scope of the claimed subject matter.

[0008] The embodiments herein address the above-recited needs for a method and system for generating a unified script code (USC) representation for multilingual text processing.

[0009] The present invention provides a computer-implemented method and system for generating a Unified Script Code (USC) representation of text, enabling consistent, phonemically accurate, and script-neutral encoding across multiple languages and writing systems. The invention addresses the limitations of conventional tokenization techniques in multilingual environments by standardizing text at the phonetic level, thereby improving tokenization efficiency, representation quality, and downstream processing performance in natural language processing (NLP) and machine learning applications.

[0010] In accordance with one embodiment, the system receives text input encoded in Unicode from one or more Indic scripts and converts each character into a corresponding Devanagari character based on phonetic equivalence. The conversion may be performed using a predefined mapping table or a bitwise mapping approach that leverages systematic Unicode alignment. Following conversion, the system processes the text to generate a normalized USC representation by (i) inserting explicit inherent vowels where required, (ii) converting dependent vowel signs (matras) into their corresponding independent vowel characters, and (iii) identifying and removing halant (virama) characters to dissociate conjunct consonants.

[0011] The resulting USC sequence represents every consonant-vowel combination explicitly, with all phonetic components preserved and expressed in a script-neutral form. This representation significantly reduces token diversity across languages, improves multilingual tokenization performance, and enables reversible mapping back to original scripts. In certainembodiments, the system further supports concurrent processing of text from multiple scripts, integration with phonetic dictionaries for non-Indic inputs, and real-time USC generation for large-scale language models and NLP pipelines.

[0012] It is to be understood that the aspects and embodiments of the disclosure described above may be used in any combination with each other. Several of the aspects and embodiments may be combined to form a further embodiment of the disclosure.

[0013] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0014] These and other objects and advantages will become more apparent when reference is made to the following description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The other objects, features and advantages will occur to those skilled in the art from the following description of the preferred embodiment and the accompanying drawings in which:

[0016] FIG.1 depicts a flow diagram of the overall USC conversion system, in accordance with an embodiment of the present invention.

[0017] FIG. 2 depicts a flow diagram representing Devanagari Base Mapping, in accordance with an embodiment of the present invention.

[0018] FIG. 3 depicts a flow diagram to generate a USC representation of the converted Devanagari string, in accordance with an embodiment of the present invention.

[0019] FIG.4 depicts a system diagram to generate a USC representation, in accordance with an embodiment of the present invention.

[0020] Although the specific features of the embodiments herein are shown in some drawings and not in others. This is done for convenience only as each feature may be combined with any or all of the other features in accordance with the embodiments herein.DETAILED DESCRIPTION

[0021] The detailed description of various exemplary embodiments of the disclosure is described herein with reference to the accompanying drawings. It should be noted that the embodiments are described herein in such details as to clearly communicate the disclosure. However, the amount of details provided herein is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0022] It is also to be understood that various arrangements may be devised that, although not explicitly described or shown herein, embody the principles of the present disclosure. Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples, are intended to encompass equivalents thereof.

[0023] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0024] The detailed description of various exemplary embodiments of the disclosure is described herein with reference to the accompanying drawings. It should be noted that the embodiments are described herein in such details as to clearly communicate the disclosure. However, the details provided herein is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0025] It is also to be understood that various arrangements may be devised that, although not explicitly described or shown herein, embody the principles of the present disclosure. Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples, are intended to encompass equivalents thereof.

[0026] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood however, it is not intended to limit the disclosureto the forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0027] The various embodiments of the present technology provide a method and system for Unified Script Code (USC) generation, which has been initially implemented for Indian languages but can be extended to virtually all languages to achieve significantly improved text tokenization.

[0028] Key Definitions 1. Matra (Vowel Diacritic): A dependent vowel sign that attaches to a consonant and modifies its inherent vowel. Example: Devanagari(ka + ā). 2. Independent Vowel: A standalone vowel symbol that represents a full vowel sound. Example: 'अ' (U+0905), 'आ' (U+0906). 3. Halant (Virama): A sign (U+094D) that suppresses the inherent vowel of a consonant, producing a half-consonant. Example:(क + ◌्) represents only 'k'. 4. Unified Script Code (USC): A normalized sequence where every consonant is explicitly followed by a vowel, and all matras are converted into their corresponding independent vowels.

[0029] FIG.1 depicts a flow diagram of the overall USC conversion system, in accordance with an embodiment of the present invention. Generally, ten major scripts are used to write around 22 official languages, and many additional languages both within India and globally employ similar script structures. These ten scripts share several common characteristics because they are phonetic scripts: the alphabet and writing system represent sounds (phonemes) rather than abstract letters. Consequently, while the visual shapes of letters differ across scripts, the pronunciation of the corresponding letters is largely similar. For example, the consonant sounds “ka,” “cha,” “ta,” “tha,” “pa,” , exist across all Indian scripts as follows:

[0030] The present invention, referred to as Unified Script Code (USC), assigns a single encoded value to all the letters from different scripts that represent the same sound. For instance, the consonant “ka” in Bengali, Gujarati, Telugu, Odia, Kannada, and other scripts may all be mapped to the same USC value.

[0031] In addition, vowel sounds, which in many Indic scripts appear in two forms such as independent vowels (e.g., ई, pronounced “i”) and dependent vowels i.e., matras (e.g., ◌ी, pronounced “i”) are also unified. In the USC system, both forms are assigned the same re-encoded value across all scripts.

[0032] By re-encoding text in this manner, the total number of distinct vowel representations in any script is effectively reduced by approximately half, since the independent and dependent forms of vowels are unified. Furthermore, assigning the same USC values tophonetically equivalent letters across multiple scripts reduces the total number of letters to the equivalent of a single script.

[0033] This unified encoding ensures that any text, regardless of its original script, is represented consistently at the token level. As a result, text tokenization becomes more efficient, accurate, and scalable, improving the performance of natural language processing tasks across multiple languages.

[0034] FIG. 2 depicts a flow diagram representing Devanagari Base Mapping, in accordance with an embodiment of the present invention.

[0035] The system and method of the invention first involve receiving an input string comprising characters encoded in Unicode from an Indic script. The term input string refers to any sequence of characters, letters, or symbols forming text in a natural language, where each character is represented according to the Unicode standard. The Unicode standard provides a unique numerical codepoint for every character across multiple scripts, including Devanagari, Bengali, Gujarati, Kannada, Malayalam, Odia, Punjabi, Tamil, Telugu, Marathi, Assamese, and others.

[0036] Upon receiving the input string, the system obtains the Unicode value of each input character. The Unicode value is a numerical codepoint assigned to a character by the Unicode Consortium, which serves as a universal identifier for that character. For example, the Devanagari letter (pronounced “ka”) has a Unicode codepoint U+0915, while the Bengali letter(also pronounced “ka”) has the Unicode codepoint U+0995. By retrieving these Unicode values, the system can precisely identify each character and distinguish it from visually similar or phonetically identical characters in other scripts.

[0037] After obtaining the Unicode values of the input characters, the system converts the received input string from an Indic script into a corresponding Devanagari script representation. Devanagari has been selected as the base reference script for convenience and consistency of representation. This conversion is performed at the Unicode codepoint level, ensuring that each input character is accurately mapped to a Devanagari character representing the same phonetic sound. The conversion process can be implemented using one of the following two mapping approaches.

[0038] First approach is using a predefined mapping table, where each input character is replaced with a corresponding Devanagari character based on a predefined mapping table. The mapping table stores associations between the Unicode values of input script characters and their equivalent Devanagari Unicode values. The mapping ensures that characters with the samepronunciation, but differing in script representation, are converted to their phonetic equivalent in Devanagari. For example, the consonant “ka” in Bengali (ক, U+0995, pronounced “ka”), Gujarati (ક, U+0A95, pronounced “ka”), Telugu (క, U+0C15, pronounced “ka”), and other scripts are all mapped to the Devanagari letter क (U+0915, pronounced “ka”).

[0039] Second approach is using bitwise mapping, where a bitwise operation could be used for efficient conversion for consonants and vowels that have a systematic numeric alignment across Indic script Unicode blocks. The bitwise mapping involves bitwise AND operation for extracting the last seven bits and bitwise OR operation for combining the extracted last seven bits with the Devanagari base value.

[0040] The system performs a bitwise AND operation between the Unicode value of the input character and the constant 0x7F to extract the last seven bits. The constant 0x7F acts as a bitmask to retain only the seven least significant bits, clearing the most significant bit. Since Unicode has systematically assigned the same last seven bits to equivalent phonetic characters across Indic scripts, extracting these bits ensures character equivalence. This step isolates the common phonetic identity of the character across scripts. For example, the Gujarati character ક (Unicode U+0A95, pronounced “ka”), when bitwise-ANDed with 0x7F, yields the result 0x15, which is the same result obtained for its counterparts in Devanagari क (Unicode U+0915, pronounced “ka”) and Tamil க (Unicode U+0B95, pronounced “ka”).

[0041] The system performs a bitwise OR operation to combine the extracted seven bits with a predefined Devanagari base value (0x0900). The Devanagari base value represents the starting Unicode codepoint for the Devanagari script. The constant 0x0900 corresponds to the first 9 bits of the Devanagari Unicode block (U+0900 to U+097F). The resulting value corresponds to the Unicode codepoint of the equivalent Devanagari character.

[0042] According to exemplary embodiments for mapping of character from multiple Indic scripts into Devanagari considering Bitwise Mapping using Unicode Values, the working examples are illustrated as follows:

[0043] Example 1: Vowel ‘आ’ (pronounced “ā”) Unicode values: Devanagari आ (U+0906, pronounced “ā”), Gujarati આ (U+0A86, pronounced “ā”), Tamil(U+0B86, pronounced “ā”).Step 1: Bitwise AND with 0x7F → 0x06 for all three. Step 2: Bitwise OR with 0x0900 → 0x0906 (आ, Devanagari, pronounced “ā”). Thus, all three characters are normalized to Devanagari आ (pronounced “ā”).

[0044] Example 2: Consonant ‘क’ (pronounced “ka”) Unicode values: Devanagari क (U+0915, pronounced “ka”), Gujarati ક (U+0A95, pronounced “ka”), Tamil க (U+0B95, pronounced “ka”). Step 1: Bitwise AND with 0x7F → 0x15 for all three. Step 2: Bitwise OR with 0x0900 → 0x0915 (क, Devanagari, pronounced “ka”). Thus, all three characters are normalized to Devanagari क (pronounced “ka”).

[0045] Example 3: Consonant ‘म’ (pronounced “ma”) Unicode values: Devanagari म (U+092E, pronounced “ma”), Gujarati મ (U+0AAE, pronounced “ma”), Tamil ம (U+0BAE, pronounced “ma”). Step 1: Bitwise AND with 0x7F → 0x2E for all three. Step 2: Bitwise OR with 0x0900 → 0x092E (म, Devanagari, pronounced “ma”). Thus, all three characters are normalized to Devanagari म (pronounced “ma”).

[0046] Through the above mapping using either the predefined table or the bitwise conversion ensures that every input character from an Indic script is consistently converted to a Devanagari character representing the same phonetic sound. By operating at the Unicode codepoint level, the system preserves pronunciation information and enables downstream processing, including inherent vowel insertion, matra conversion, halant removal, and generation of a Unified Script Code (USC) representation.

[0047] FIG.3 depicts a flow diagram to generate a USC representation of the converted Devanagari string, in accordance with an embodiment of the present invention.

[0048] In one embodiment, after the characters of the input string have been mapped to the corresponding Devanagari characters, the system further processes the Devanagari representation by inserting an inherent vowel, converting matras (vowel signs) into their corresponding fullvowels and by reducing the use of halant (consonant joiner) characters. This process ensures that the representation of the string is normalized and suitable for uniform script conversion (USC).

[0049] As per step 1, in Indic scripts, consonants inherently carry a vowel sound, typically the short vowel "अ" (pronounced as a in "cut"). However, when writing, this inherent vowel is not explicitly represented unless required by script rules. To preserve phonetic accuracy during conversion to the USC representation, the system inserts the inherent vowel character into the converted Devanagari string in specific contexts.

[0050] The insertion occurs in the following cases. When, a) A consonant is directly followed by another consonant, b) A consonant is followed by an independent vowel, c) A consonant is followed by a non-Indic character, or d) The consonant appears as the last character in the string. By inserting the inherent vowel "अ" (pronounced as a in "cut") in such cases, the system ensures that consonants without explicit dependent vowel signs are properly expressed with their default phonetic value. This maintains phonetic completeness and consistency across multiple Indic scripts in accordance with Unicode and traditional linguistic rules.

[0051] For example, in the word "कप" (kap), the consonantis automatically assigned the inherent vowel "अ," making its pronunciation "ka," while "प" is pronounced "pa" by the same rule. Without this step, the phonetic integrity of the converted text would be compromised.

[0052] As per step 2, in Indic scripts, vowels are expressed in two ways: as independent characters when standing alone, and as dependent vowel signs (commonly known as matras) when attached to a consonant. For example, in Devanagari, the vowel(pronounced as “i”) is written independently as(U+0907) but appears as the matra "ि◌" (U+093F) when attached to a consonantpronounced as “ki”).

[0053] To ensure accurate phonetic reconstruction and script neutrality in the converted Devanagari string, the system identifies and processes matras as follows. For each consonant in the converted Devanagari string, the system checks if a dependent vowel sign (matra) is attached. The matra is identified by comparing the Unicode value of the attached sign with a predefined set of Unicode values corresponding to matras defined in Devanagari. Once identified, each matra is mapped to its equivalent independent vowel character. This conversion is performed by replacingthe Unicode value of the matra with the Unicode value of the corresponding independent vowel. For example, the matra "ि◌" (U+093F) is replaced by the independent vowel(U+0907).

[0054] The mapping between matra Unicode values and their corresponding independent vowel Unicode values is maintained in the system. This may be implemented as a static table or retrieved from a configuration database that defines all valid associations between dependent and independent vowel forms.

[0055] Through this process, the system transforms consonant–matra combinations into explicit consonant-plus-independent vowel sequences. This ensures that vowels are represented uniformly and unambiguously across multiple Indic scripts, thereby preserving phonetic accuracy and enabling consistent processing in the unified script representation.

[0056] For example, the Devanagari word "िकत" (pronounced as “kith”) contains the consonant (U+0915) with the matra "ि◌" (U+093F). The system identifies "ि◌" as a matra, maps it to the independent vowel(U+0907), and rewrites the sequence as(ka) + "इ" (i)+ "त" (tha) making the phonetic representation explicit. This mapping process is applied systematically to every matra in the string, ensuring that all vowels are represented in their independent form. Each consonant now explicitly carries the vowel as a separate character, eliminating ambiguity caused by inherent vowels.

[0057] The matra mapping table ensures that all matras across Indic scripts are consistently expanded into independent vowels during the conversion process. An exemplary mapping table for Devanagari is provided below:Table 1: The Matra mapping table

[0058] As per step 3, in Indic scripts, consonants by default carry an inherent vowel sound (commonly the short “अ” pronounced “a”). To suppress this inherent vowel and allow consonants to form conjunct clusters, a special diacritic known as the halant (or virama) is used. In Devanagari, the halant is represented by the Unicode value U+094D. For example, the sequence(ka + halant) indicates that the consonant(ka) should not be pronounced with its inherent vowel.

[0059] To normalize characters in the converted Devanagari string and dissociate conjunct consonants into standalone forms, the system performs the following. Halant is identified by the system by scanning each character in the converted Devanagari string and checks whether the character corresponds to the halant (U+094D). Since this codepoint uniquely identifies the halantin Devanagari, detection is based on checking if the character matches the Unicode value of the halant. When a halant character is detected, it is removed from the string. This removal breaks the conjunct formation, effectively restoring the preceding consonant to its standalone form.

[0060] By removing the halant, conjunct consonants are dissociated, and each consonant is represented individually. This ensures that the string no longer encodes conjunct forms, but rather a phonetic sequence of standalone consonants, making the representation uniform and script-neutral.

[0061] For example, the Devanagari conjunct(kta, formed by(ka) + “◌्” (halant) + "त" (tha)) is represented internally with the halant (U+094D) between the consonants. When the halant is identified and removed, the sequence becomes(ka) + "त" (tha), treating both consonants as independent characters. This preserves phonetic clarity and enables consistent handling of Indic script text across different scripts.

[0062] In Conclusion, the final output is the USC converted sequence which represents the USC form of the input text. In the final USC converted sequence, all characters are expressed within the Devanagari Unicode block. Step 1, ensures that each consonant is explicitly followed by a vowel, thereby removing any ambiguity associated with inherent vowels. Step 2, ensures that all matras, which are dependent vowel signs, are replaced by their corresponding independent vowel characters. Similarly step 3, ensures that all halant (virama) characters, which suppress inherent vowels and create conjunct consonants, are identified and removed, ensuring that each consonant remains independent. The result of this process is referred to as the USC sequence, thereby achieving the central objective of USC.

[0063] By way of illustration, a USC representation of the Gujarati text is generated as follows: Gujarati Input String: “પ્રકાશ” (Pronouned as “Prakash”) Unicode sequence: પ + ◌્ + ર + ક + ◌ા + શ Intermediate converted Devanagari string:प्रकाश (Pronouned as “Prakash”) "પ" (pa) is mapped to "प" (pa)"ર" (ra) is mapped to "र" (ra) "ક" (ka) is mapped"શ" (sha) is mapped to "श" (sha) Each character of Gujarati Input String to Intermediate converted Devanagari string is mapped as follows:Next, generate a USC representation of the converted Devanagari string in 3 steps: Step 1: Insert an inherent vowel character (Add “अ” after consonants where needed)Step 2: Convert the identified matras into independent vowels using the Matra mapping table (Table 1)Step 3: Halant is identified and removed from the converted Devanagari string to dissociate conjunct consonants into standalone forms

[0064] The final USC representation of the Gujarati input string “પ્રકાશ” (pronounced as Prakash) is generated as “परअकआशअ”. Gujarati Unicode sequence: પ (pa) + ◌્ (Halant) + ર (ra) + ક (ka) + ◌ા (Matra) + શ (sha) is mapped to Intermediate converted Devanagari string: प (pa) + ◌् (Halant) + र (ra) + क (ka) + ◌ा (Matra) + श (sha) Step (Halant)(Matra)Step 2: प + ◌् (Halant)StepFinal USC representation: परअकआशअThus, the phonetic value of the original Gujarati word પ્રકાશ (Prakash) is preserved exactly, and expressed in USC as a sequence of explicit consonant and vowel characters: “परअकआशअ.”

[0065] Further illustrative examples of USC Conversion demonstrate the operation of the Unified Script Code (USC) system, the following examples show how words written in multiple Indic scripts are converted into their standardized phonemic representation, while maintaining consistent pronunciation across languages: Word 1 – “Prakash” (pronounced Pra-kaash) Hindi: प्रकाश Gujarati: પ્રકાશUSC Representation:Word 2 – “Kamal” (pronounced Ka-mal) Hindi: कमल Gujarati: કમલ Bengali:USC Representation:Word 3 – “Kiran” (pronounced Ki-ran) Hindi: Gujarati:Bengali:USC Representation:These examples illustrate that, regardless of the original script, the phonemic structure of the word is consistently captured and represented in a unified encoding. This ensures that equivalent pronunciations across different scripts result in identical USC sequences, enabling script-independent processing.

[0066] In one embodiment, the predefined mapping table is stored in a configuration database that can be updated independently of the USC conversion software.

[0067] In another embodiment, the bitwise mapping approach is applied only to consonants and vowels whose Unicode values have a systematic numeric alignment with Devanagari Unicode values, and all other characters are converted using the predefined mapping table.

[0068] In another embodiment, the generated USC representation preserves the pronunciation of the original input text and enables conversion back to the original script.

[0069] In another embodiment, the insertion of the inherent vowel character is applied only to consonants that do not already have an explicit dependent vowel sign and not to the consonant which is followed by a matra or is part of a conjunct consonant representation.

[0070] In another embodiment, the matra identification further comprises checking for dependent vowel signs attached to conjunct consonants in the converted Devanagari string.

[0071] In another embodiment, the input string comprises text in a non-indic script, and the system performs phonetic transliteration of the input string into devanagari characters based on pronunciation rules.

[0072] FIG.4 depicts a system diagram to generate a USC representation, in accordance with an embodiment of the present invention. According to another embodiment of the present invention, the processor-implemented system for generating a unified script code (USC) representation of text comprises a set of specialized modules working together to normalize input from Indic scripts. An input module receives Unicode-encoded text from an Indic script. A Unicode extraction module retrieves the numerical codepoints of each character. A script conversion module maps these characters into equivalent Devanagari characters using either a predefined mapping table or bitwise alignment rules, ensuring phonetic equivalence. An inherent vowel insertion module adds explicit vowel sounds where required to preserve pronunciation. A matra identification and conversion module detects dependent vowel signs (matras) and transforms them into independent vowel characters based on stored mappings. A halant removal module eliminates halant (◌्, U+094D) characters, thereby dissociating conjunct consonants. Finally, a USC generation module produces a Devanagari string in USC format, consisting solely of explicit consonant and vowel characters without halants or matras, thereby preserving the original phonetic value of the input text.

[0073] The system further extends functionality by enabling multi-script concurrent processing, wherein the input module can simultaneously receive text strings encoded in different Indic scripts, and the USC generation module operates in real-time to generate corresponding USC representations for each input string without latency. Additionally, the system may incorporate a phonetic dictionary module configured to provide phonetic mappings for non-Indic script inputs (e.g., Roman script), thereby ensuring accurate phonetic equivalence and enabling their consistent conversion into unified script code (USC) representations.

[0074] The present invention enables representation of multiple languages and scripts within a single unified script framework. By converting diverse text into a phonemically consistent Unified Script Code (USC), it simplifies multilingual text processing and removes the need for language-specific handling.

[0075] The invention also improves processing efficiency for low-resource languages by reducing tokenization bias and enhancing token quality, resulting in better model performance and more balanced machine learning outcomes.

[0076] The system is applicable across various AI and NLP domains, including neural machine translation (NMT), automatic speech recognition (ASR), text-to-speech (TTS), large language models (LLMs), and text summarization, where consistent sub-word representation improves accuracy and efficiency.

[0077] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and / or adapt for various applications such as specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modifications. However, all such modifications are deemed to be within the scope of the claims. The scope of the embodiments will be ascertained by the claims to be submitted at the time of filing a complete specification.

Claims

CLAIMS What is claimed is:

1. A method for generating a unified script code (USC) representation of text, the method comprising: receiving an input string comprising characters encoded in Unicode from an Indic script; obtaining the Unicode values of each input character, the Unicode value being a numerical codepoint assigned to the character by a Unicode Consortium; converting the received input string from an Indic script into a corresponding Devanagari script representation by mapping the Unicode value of each input character to a new Unicode value corresponding to an equivalent Devanagari character, wherein the mapping comprises: replacing each input character with a corresponding Devanagari character as defined in a predefined mapping table, wherein the predefined mapping table stores associations between Unicode values of input script characters and their equivalent Devanagari Unicode values, and wherein the mapping ensures that each input character is converted to its phonetic equivalent in Devanagari script; inserting an inherent vowel character in the converted Devanagari string when a consonant is followed by another consonant, a vowel, a non-Indic character, or when the consonant is the last character in the string, wherein the insertion of the inherent vowel ensures that consonants without explicit dependent vowel signs are represented with their inherent vowel sound, thereby preserving phonetic completeness in accordance with Indic script rules; identifying matras associated with each consonant in the converted Devanagari string, wherein a matra is a dependent vowel sign attached to a consonant, and wherein the matra is identified by checking if the dependent vowel sign attached to the consonant matches the Unicode value of a predefined matra in the script;converting each identified matra in the converted Devanagari string into a corresponding independent vowel character by replacing the Unicode value of the identified matra with the Unicode value of its equivalent independent vowel character according to a predefined mapping, wherein the predefined mapping is stored in the system or provided by a configuration database that defines associations between matra Unicode values and corresponding independent vowel Unicode values; identifying and removing halant characters in the converted Devanagari string, wherein a halant is uniquely identified by the Unicode value U+094D, which suppresses the inherent vowel of a consonant and forms conjunct consonants, and wherein removing the halant from the string dissociates conjunct consonants into standalone consonants; and generating a converted Devanagari string representing the same pronunciation as the original input text, encoded in the USC format as a sequence of explicit consonant and vowel characters without any halants or matras.

2. A method for generating a unified script code (USC) representation of text, the method comprising: receiving an input string comprising characters encoded in Unicode from an Indic script; obtaining the Unicode values of each input character, the Unicode value being a numerical codepoint assigned to the character by a Unicode Consortium; converting the received input string from an Indic script into a corresponding Devanagari script representation by mapping the Unicode value of each input character to a new Unicode value corresponding to an equivalent Devanagari character, wherein the mapping comprises: (i) extracting the last seven bits of the Unicode value of the input character by performing a bitwise-AND operation between the Unicode value and the constant 0x7F, wherein 0x7F acts as a bitmask to retain only the seven least significant bits and clear the most significant bit, the seven least significant bits representing equivalent phonetic characters across Indic scripts as defined in Unicode; (ii) combining the extracted seven bits with a predefined Devanagari base value by performing a bitwise OR operation between the extracted last seven bits and the predefined devanagari base value 0x0900, thus generating the new Unicode valuecorresponding to the equivalent Devanagari character, wherein the devanagari base value is the starting codepoint of the Unicode which indicates the devanagari script, and the extracted seven bits specify the particular character; inserting an inherent vowel character in the converted Devanagari string when a consonant is followed by another consonant, a vowel, a non-Indic character, or when the consonant is the last character in the string, wherein the insertion of the inherent vowel ensures that consonants without explicit dependent vowel signs are represented with their inherent vowel sound, thereby preserving phonetic completeness in accordance with Indic script rules; identifying matras associated with each consonant in the converted Devanagari string, wherein a matra is a dependent vowel sign attached to a consonant, and wherein the matra is identified by checking if the dependent vowel sign attached to the consonant matches the Unicode value of a predefined matra in the script; converting each identified matra in the converted Devanagari string into a corresponding independent vowel character by replacing the Unicode value of the identified matra with the Unicode value of its equivalent independent vowel character according to a predefined mapping, wherein the predefined mapping is stored in the system or provided by a configuration database that defines associations between matra Unicode values and corresponding independent vowel Unicode values; identifying and removing halant characters in the converted Devanagari string, wherein a halant is uniquely identified by the Unicode value U+094D, which suppresses the inherent vowel of a consonant and forms conjunct consonants, and wherein removing the halant from the string dissociates conjunct consonants into standalone consonants; and generating a converted Devanagari string representing the same pronunciation as the original input text, encoded in the USC format as a sequence of explicit consonant and vowel characters without any halants or matras.

3. The method of claim 1, wherein the predefined mapping table is stored in a configuration database that can be updated independently of the USC conversion software.

4. The method of claim 2, wherein the bitwise mapping approach is applied to convert characters of other Indic scripts whose Unicode values have a systematic numeric alignment with Devanagari Unicode values, and all other characters are converted using the predefined mapping table.

5. The method of claim 1 or 2, wherein the generated USC representation preserves the pronunciation of the original input text and enables conversion back to the original script.

6. The method of claim 1 or 2, wherein the insertion of the inherent vowel character is applied only to consonants that do not already have an explicit dependent vowel sign and not to the consonant which is followed by a matra or is part of a conjunct consonant representation.

7. The method of claim 1 or 2, wherein the matra identification further comprises checking for dependent vowel signs attached to conjunct consonants in the converted Devanagari string.

8. The method of claim 1 or 2, wherein the input string comprises text in a non-indic script, and further comprising: performing phonetic transliteration of the input string into devanagari characters based on pronunciation rules.

9. A processor-implemented system for generating a unified script code (USC) representation of text, the system comprising: an input module configured to receive an input string comprising characters encoded in Unicode from an Indic script;a Unicode extraction module configured to obtain the Unicode values of each input character, the Unicode value being a numerical codepoint assigned to the character by a Unicode Consortium; a script conversion module configured to convert the received input string from the Indic script into a corresponding Devanagari script representation by mapping the Unicode value of each input character to a new Unicode value corresponding to an equivalent Devanagari character, wherein the mapping comprises: replacing each input character with a corresponding Devanagari character as defined in a predefined mapping table, the predefined mapping table storing associations between Unicode values of input script characters and their equivalent Devanagari Unicode values, and the mapping ensuring that each input character is converted to its phonetic equivalent in Devanagari script; an inherent vowel insertion module configured to insert an inherent vowel character in the converted Devanagari string when a consonant is followed by another consonant, a vowel, a non-Indic character, or when the consonant is the last character in the string, the insertion of the inherent vowel ensuring that consonants without explicit dependent vowel signs are represented with their inherent vowel sound, thereby preserving phonetic completeness in accordance with Indic script rules; a matra identification module configured to identify matras associated with each consonant in the converted Devanagari string, the matra being a dependent vowel sign attached to a consonant, wherein the matra is identified by checking if the dependent vowel sign attached to the consonant matches the Unicode value of a predefined matra in the script; a matra conversion module configured to convert each identified matra in the converted Devanagari string into a corresponding independent vowel character by replacing the Unicode value of the identified matra with the Unicode value of its equivalent independent vowel character according to a predefined mapping, the predefined mapping stored in the system or provided by a configuration database defining associations between matra Unicode values and corresponding independent vowel Unicode values; a halant removal module configured to identify and remove halant characters in the converted Devanagari string, the halant uniquely identified by the Unicode value U+094D, which suppresses the inherent vowel of a consonant and forms conjunct consonants, removal of the halant dissociating conjunct consonants into standalone consonants; anda USC generation module configured to generate a converted Devanagari string representing the same pronunciation as the original input text, encoded in the USC format as a sequence of explicit consonant and vowel characters without any halants or matras.

10. The system of claim 9, wherein the input module is configured to receive input strings in multiple Indic scripts concurrently, and wherein the USC generation module is configured to generate corresponding USC representations for each input string in real-time.

11. The system of claim 9, further comprising a phonetic dictionary module configured to provide phonetic mappings for input strings in non-Indic scripts to enable accurate conversion into corresponding unified script code (USC) representations.

Images