Glyph id-based control character filtering method and device in accordance with mongolian code standard, readable storage medium and equipment

Abstract

The application provides a control symbol filtering method, device, readable storage medium and equipment based on glyph ID and in line with the Mongolian coding standard, the method comprising: S1: inputting a Mongolian word; S2: counting the number of Mongolian word control symbols saved in the MWord; S3: setting the counter N=1; S4: judging whether the count of N exceeds the number of control symbols, if not, executing the loop body to acquire the glyph ID string corresponding to the Mongolian word before and after the Nth control symbol; S5: judging whether the glyph ID string corresponding to the Mongolian word before and after the Nth control symbol is equal, if equal, deleting the Nth control symbol of the Mongolian word saved in the MWord, if not equal, not deleting the Nth control symbol of the Mongolian word saved in the MWord. The method can filter out the "redundant" control symbols in the Mongolian text resource, save the system resource, keep the Mongolian text resource simple, and ensure the correctness of the Mongolian glyph.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer word processing, in particular, to a control symbol filtering method and device based on glyph ID and in line with the Mongolian coding standard, a readable storage medium and equipment. BACKGROUND

[0002] The international standard for Mongolian coding was approved by ISO / IEC JTC 1 / SC2 in February 2000, ending the era of no unified coding for Mongolian. In 2010, the standard "GB / T 25914-2010 Information Technology Traditional Mongolian Nominal Characters, Transformation Display Characters and Control Characters Usage Rules" was established, which is the first Mongolian conversion rule standard.

[0003] Since 2010, the development and production of Mongolian OpenType font must comply with the Unicode standard and national standard. In particular, GB / T 25914-2010 and GB / T 25914-2023, which is revised on the basis of the former, were revised in 2016. After the seven major Mongolian font development companies upgraded their own fonts based on this revised standard, the rules of the major Mongolian OpenType fonts used on the market have basically been unified. GB / T 25914-2023, the revised standard of GB / T 25914-2010, will be implemented from June 1, 2024, and GB / T 25914-2010 will be abolished. Future Mongolian OpenType fonts must comply with GB / T 25914-2023.

[0004] During the input process of Mongolian text, the developers of input methods and Mongolian input personnel do not consider the "contextual relationship" between Mongolian characters to input the correct Mongolian glyph, but use control symbols to forcibly output the correct Mongolian glyph. Although this can ensure the correctness of the Mongolian glyph, there are many "redundant" control symbols in Mongolian words that should not be used. As shown in Table 1, Mongolian coding string 1 and 2 display the same Mongolian glyph after being processed by the text engine. This shows that the control symbol 180C in Mongolian coding string 1 is redundant. Table 2 shows that Mongolian coding string 1 and 2 display different Mongolian glyphs for character 182D after being processed by the text engine. This shows that the control symbol 180B in Mongolian coding string 1 is necessary.

[0005] Table 1 Control symbols that can be filtered

[0006]

[0007] Table 2 Control symbols that cannot be filtered

[0008]

[0009] These redundant control symbols will seriously affect the use of Mongolian text resources. In the preprocessing of Mongolian text resources, one of the important tasks is to filter out the "redundant" control symbols, so as to ensure the quality of Mongolian text resources. Therefore, the filtering of control symbols in Mongolian resources is a very urgent problem in Mongolian information processing.

[0010] So far there is no control symbol filtering method. Reason: Before the release of GB / T 25914-2023, the font library developed by the Mongolian font library manufacturer needs to add a lot of "own conversion rules" on the basis of conforming to GB / T 25914-2010. GB / T 25914-2010 standard has many conversion rules that have not been covered, which means that there is no uniform standard for these rules. For these rules, each font library manufacturer adds them to the font library according to their own understanding (i.e. "own conversion rules"). This has led to the non-uniformity of conversion rules between fonts of various font library manufacturers.

[0011] After the release of GB / T 25914-2023 standard, the conversion rules of Mongolian are completely covered. Therefore, no matter which manufacturer's Mongolian font library is used, it is completely possible to filter out the "redundant" control symbols in the Mongolian resource. SUMMARY

[0012] Therefore, the present application provides a control symbol filtering method and device based on glyph ID in accordance with the Mongolian coding standard, to filter out the redundant control symbols in the Mongolian coding string.

[0013] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0014] On the one hand, the present application provides a control symbol filtering method based on glyph ID in accordance with the Mongolian coding standard, comprising:

[0015] S1: input a Mongolian word, and save the input Mongolian word in variable MWord; the coding of the Mongolian word conforms to the GB / T 25914-2023 standard;

[0016] S2: count the number of control symbols of the Mongolian word saved in MWord, and save the counting result in variable M;

[0017] S3: let the counter N = 1, and the counter N is a loop variable;

[0018] S4: judging whether the count of N exceeds the number of control symbols, if yes, outputting the Mongolian word saved in variable MWord and ending the process; otherwise, executing the loop body, including: saving the Mongolian word formed after pre-deleting the Nth control symbol in MWord in intermediate result variable MidWord, and obtaining the glyph ID string corresponding to the Mongolian word saved in MidWord through the text rendering engine; at the same time, obtaining the glyph ID string corresponding to the Mongolian word saved in MWord through the text rendering engine;

[0019] S5: comparing whether the glyph ID string corresponding to the Mongolian word saved in MidWord is equal to the glyph ID string corresponding to the Mongolian word saved in MWord, if yes, indicating that the Nth control symbol is a redundant control symbol, deleting the Nth control symbol of the Mongolian word saved in MWord, if not, indicating that the Nth control symbol is necessary, and not deleting the Nth control symbol of the Mongolian word saved in MWord;

[0020] S6: N = N + 1;

[0021] S7: repeating steps S4-S6 until N > M, processing all control symbols, and ending the loop;

[0022] S8: finally outputting the Mongolian word saved in variable MWord after filtering the redundant control symbols.

[0023] Further, the text rendering engine includes Harfbuzz and Directwrite.

[0024] In addition, the application provides a glyph ID-based control symbol filtering device conforming to the Mongolian coding standard, including:

[0025] An input unit is configured to input a Mongolian word and save the input Mongolian word in variable MWord; the coding of the Mongolian word conforms to the GB / T 25914-2023 standard;

[0026] A statistical unit is configured to count the number of control symbols of the Mongolian word saved in MWord and save the counting result in variable M;

[0027] A counting unit is configured to set a counter N = 1, and the counter N is a loop variable;

[0028] A judging execution unit is configured to judge whether the count of N exceeds the number of control symbols, and if yes, output the Mongolian word stored in the variable MWord and end the process; otherwise, execute a loop body, including: storing the Mongolian word formed after the pre-deletion of the Nth control symbol in the variable MWord in an intermediate result variable MidWord, and obtaining the glyph ID string corresponding to the Mongolian word stored in MidWord through a text rendering engine, and simultaneously obtaining the glyph ID string corresponding to the Mongolian word stored in MWord through the text rendering engine;

[0029] A comparison deletion unit is configured to compare whether the glyph ID string corresponding to the Mongolian word stored in MidWord is equal to the glyph ID string corresponding to the Mongolian word stored in MWord, and if yes, it indicates that the Nth control symbol is a redundant control symbol, and the Nth control symbol of the Mongolian word stored in MWord is deleted, and if not, it indicates that the Nth control symbol is necessary, and the Nth control symbol of the Mongolian word stored in MWord is not deleted.

[0030] An addition unit is configured to let N = N + 1.

[0031] A return unit is configured to repeatedly execute steps S4-S6 until N > M, and all control symbols are processed, and the loop ends.

[0032] An output unit is configured to finally output the Mongolian word stored in the variable MWord after the redundant control symbols are filtered.

[0033] Further, the text rendering engine includes Harfbuzz and Directwrite.

[0034] According to another aspect of the present application, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the program is executed by a processor to implement the steps in the glyph ID-based control symbol filtering method according to the Mongolian coding standard.

[0035] According to still another aspect of the present application, a computer device is provided, and includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the steps in the glyph ID-based control symbol filtering method according to the Mongolian coding standard.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] 0、The control symbol filtering method according to the present application can filter out the "redundant" control symbols in the Mongolian text resource, save system resources, keep the Mongolian text resource simple, and ensure the correctness of the Mongolian glyphs.

[0038] 1. The method is simple and easy to implement, without complex operation process. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0040] Figure 1 The flow chart of the control symbol filtering method based on glyph ID according to the Mongolian coding standard of the present application.

[0041] Figure 2 The structure block diagram of the control symbol filtering device based on glyph ID according to the Mongolian coding standard of the present application.

[0042] Figure 3 Glyph ID instance diagram in the Mongolian character library.

[0043] Figure 4 Code screenshot for counting the number of control symbols in the Mongolian word in the specific embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments.

[0045] As shown in Figure 1 The control symbol filtering method based on glyph ID according to the Mongolian coding standard includes:

[0046] S1: input a Mongolian word, save the input Mongolian word in variable MWord; the coding of the Mongolian word conforms to the GB / T 25914-2023 standard;

[0047] The Mongolian conversion rule is in units of "words", so the filtering of Mongolian control symbols is also in units of words. The input Mongolian word is saved in variable MWord for use in the subsequent filtering process.

[0048] In this embodiment, it is assumed that the coding string of the input Mongolian word is: / U+182A / U+1820 / U+1832 / U+1824 / U+180C / U+1830 / U+1826 / U+180B / U+182C / U+1821.

[0049] S2: Count the number of Mongolian word control symbols in the MWord and save the result in variable M;

[0050] By comparing the encoding of each character, count the number of character encodings equal to U+180B, U+180C, U+180D, and U+180F, and save the result in variable M.

[0051] For specific code implementation process of counting the number of control symbols, please refer to Figure 4 .

[0052] S3: Let the counter N = 1, and the counter N is a loop variable;

[0053] The counter is incremented by 1 at the end of each loop.

[0054] S4: Determine whether the count of N exceeds the number of control symbols. If so, output the Mongolian word saved in variable MWord and end the process; otherwise, execute the loop body, which includes: save the Mongolian word formed by pre-deleting the Nth control symbol in MWord in the intermediate result variable MidWord, and then obtain the glyph ID string corresponding to the Mongolian word saved in MidWord through the text rendering engine; at the same time, obtain the glyph ID string corresponding to the Mongolian word saved in MWord through the text rendering engine;

[0055] Specifically, the text rendering engine includes Harfbuzz and Directwrite.

[0056] Each glyph in the Mongolian font library has its own unique ID number. The reason why the encoding of Mongolian characters can finally be displayed as a Mongolian glyph is because of the ID of these glyphs used in the intermediate process. When each Mongolian word is displayed on the screen, the 'Mongolian encoding string' needs to be operated through the text rendering engine of the operating system to interpret the conversion rule of 'Mongolian encoding to Mongolian glyph' in the text rendering engine, then obtain the 'Mongolian glyph ID string', and finally draw the glyph ID corresponding glyph graphics to get the display result. For example Figure 3 The Mongolian font library glyph ID is displayed, and "#xxx" above each glyph is the ID of the glyph.

[0057] Suppose the encoding string of the above Mongolian word is on a certain font library. After deleting the first control symbol U+180C in the first loop, the glyph ID obtained through the rendering engine is: 369, 674, 300, 663, 317, 513.

[0058] Assuming the encoding string of the above Mongolian word on a certain font library, the glyph ID obtained by the rendering engine after deleting the second control symbol U+180B in the second loop is: 369, 674, 300, 663, 318, 513.

[0059] Assuming the encoding string of the above Mongolian word on a certain font library, the glyph ID obtained by the rendering engine without deleting the control symbol is: 369, 674, 300, 663, 317, 513.

[0060] S5: Compare whether the glyph ID string corresponding to the Mongolian word saved in MidWord and the glyph ID string corresponding to the Mongolian word saved in MWord are equal. If they are equal, it indicates that the Nth control symbol is a redundant control symbol, and the Nth control symbol saved in MWord for the Mongolian word is deleted. If they are not equal, it indicates that the Nth control symbol is necessary, and the Nth control symbol saved in MWord for the Mongolian word is not deleted.

[0061] The encoding string of the above Mongolian word on a certain font library, the first control symbol U+180C is deleted before the glyph ID (369, 674, 300, 663, 317, 513) and after the glyph ID (369, 674, 300, 663, 317, 513) are completely the same, so the first control symbol U+180C can be deleted, which is a redundant control symbol.

[0062] The encoding string of the above Mongolian word on a certain font library, the second control symbol U+180B is deleted before the glyph ID (369, 674, 300, 663, 317, 513) and after the glyph ID (369, 674, 300, 663, 318, 513), 317 and 318 are not the same, so the second control symbol U+180B cannot be deleted, which is a valid control symbol.

[0063] S6: Let N=N+1;

[0064] S7: Repeat steps S4-S6 until N>M, process all control symbols, and the loop ends.

[0065] S8: Finally output the Mongolian word saved in the variable MWord after filtering the redundant control symbols.

[0066] The encoding string of the input Mongolian word is: / U+182A / U+1820 / U+1832 / U+1824 / U+180C / U+1830 / U+1826 / U+180B / U+182C / U+1821. After two cycles, the final filtered result is: / U+182A / U+1820 / U+1832 / U+1824 / U+1830 / U+1826 / U+180B / U+182C / U+1821.

[0067] The present application obtains two "Mongolian character ID strings" before and after deleting a certain control symbol, and judges whether the control symbol is "redundant" by comparing the ID values in the two "Mongolian character ID strings". As shown in Table 3, the character ID string after filtering out the control symbol "180B" in the first encoding string is equal to the character ID string of the second encoding string without "180B", so the control symbol "180B" is redundant and can be filtered out. Note: The character of the control symbol is not displayed on the screen, that is, "124" in the character string has no figure.

[0068] Table 3: Control symbol filtering example

[0069]

[0070] As shown in Table 4, the character ID string after filtering out the control symbol "180B" in the first encoding string is not equal to the character ID string of the second encoding string without "180B", that is, the character ID of "182D" changes from "311" to "312" after filtering out the control symbol "180B". Therefore, the control symbol "180B" cannot be filtered out. Note: The character of the control symbol is not displayed on the screen, that is, "124" in the character string has no figure.

[0071] Table 4: Control symbol filtering example

[0072]

[0073] The present application can be in the form of a DLL (dynamic link library). When using the present application, the interface can be directly called by a computer high-level language (C++, C#, etc.) supporting DLL, the Mongolian text with control symbols to be filtered is sent in the parameters, and the Mongolian text with filtered control symbols conforming to the GB / T 25914-2023 standard is returned.

[0074] As shown in Figure 2 , the control symbol filtering device based on character ID and conforming to the Mongolian coding standard comprises:

[0075] The input unit 210 is configured to input a Mongolian word and save the input Mongolian word in a variable MWord; the encoding of the Mongolian word is consistent with the GB / T 25914-2023 standard;

[0076] The statistical unit 220 is configured to count the number of control symbols of the Mongolian word in the MWord and save the counting result in a variable M;

[0077] The counting unit 230 is configured to set a counter N = 1, where the counter N is a loop variable;

[0078] The judgment execution unit 240 is configured to judge whether the counting of the counter N exceeds the number of control symbols, if yes, output the Mongolian word saved in the variable MWord and end the process; otherwise, execute the loop body, including: save the Mongolian word formed after the pre-deletion of the Nth control symbol in the MWord in an intermediate result variable MidWord, and then obtain the glyph ID string corresponding to the Mongolian word saved in the MidWord through a text rendering engine, and simultaneously obtain the glyph ID string corresponding to the Mongolian word saved in the MWord through the text rendering engine;

[0079] The comparison deletion unit 250 is configured to compare whether the glyph ID string corresponding to the Mongolian word saved in the MidWord is equal to the glyph ID string corresponding to the Mongolian word saved in the MWord, if yes, it indicates that the Nth control symbol is a redundant control symbol, and the Nth control symbol of the Mongolian word saved in the MWord is deleted, if not, it indicates that the Nth control symbol is necessary, and the Nth control symbol of the Mongolian word saved in the MWord is not deleted;

[0080] The addition unit 260 is configured to set N = N + 1;

[0081] The return unit 270 is configured to repeatedly execute steps S4-S6 until N > M, process all control symbols, and end the loop;

[0082] The output unit 280 is configured to finally output the Mongolian word saved in the variable MWord after filtering the redundant control symbols.

[0083] Further, the text rendering engine includes Harfbuzz and Directwrite.

[0084] For the device embodiment, since it basically corresponds to the method embodiment, it is described more simply, and the related parts can be referred to the part of the method embodiment. The device embodiment described above is only schematic, and the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units.

[0085] The components of the system can or can not be physical units, i.e., can be located in one place or can be distributed over multiple network units, such as servers and clients. Some or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative effort.

[0086] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the steps of the control symbol filtering method based on glyph ID and in accordance with the Mongolian coding standard.

[0087] According to still another aspect of the present application, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the steps of the control symbol filtering method based on glyph ID and in accordance with the Mongolian coding standard.

[0088] The control symbol filtering method of the present application can filter out the "redundant" control symbols in the Mongolian text resources, and also ensure the correctness of the Mongolian glyphs and the cleanliness of the text resources.

[0089] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A control character filtering method based on glyph ID conforming to the Mongolian encoding standard, characterized in that, include: S1: Input a Mongolian word and save the input Mongolian word in the variable MWord; The encoding of the Mongolian words conforms to the GB / T 25914-2023 standard; S2: Count the number of Mongolian word control characters stored in the MWord and store the results in variable M; S3: Set counter N=1, where counter N is the loop variable; S4: Determine if the count of N exceeds the number of control characters. If so, output the Mongolian word stored in variable MWord and end the process. Otherwise, the loop body is executed, which includes: storing the word formed after pre-deleting the Nth control character from the Mongolian word in MWord in the intermediate result variable MidWord, and then obtaining the glyph ID string corresponding to the Mongolian word stored in MidWord through the text rendering engine; at the same time, obtaining the glyph ID string corresponding to the Mongolian word stored in MWord through the text rendering engine. S5: Compare the glyph ID strings corresponding to the Mongolian words stored in MidWord and MWord to see if they are equal. If they are equal, it means that the Nth control character is a redundant control character, and the Nth control character of the Mongolian words stored in MWord is deleted. If they are not equal, it means that the Nth control character is necessary, and the Nth control character of the Mongolian words stored in MWord is not deleted. S6: Let N = N + 1; S7: Repeat steps S4-S6 until N>M, process all control characters, and the loop ends; S8: The final output is the Mongolian word stored in the variable MWord after filtering out redundant control characters.

2. The control character filtering method based on glyph ID conforming to the Mongolian encoding standard as described in claim 1, characterized in that, The text rendering engines include Harfbuzz and Directwrite.

3. A control character filtering device based on glyph ID conforming to the Mongolian encoding standard, characterized in that, include: The input unit is used to input a Mongolian word and store the input Mongolian word in the variable MWord; The encoding of the Mongolian words conforms to the GB / T 25914-2023 standard; The statistical unit is used to count the number of Mongolian word control characters stored in the MWord and store the statistical results in the variable M; The counting unit is used to set counter N=1, where counter N is a loop variable; The judgment execution unit is used to determine whether the count of N exceeds the number of control characters. If so, the Mongolian word stored in the variable MWord is output and the process ends. Otherwise, the loop body is executed, which includes: saving the word formed after pre-deleting the Nth control character of the Mongolian word in MWord into the intermediate result variable MidWord, and then obtaining the glyph ID string corresponding to the Mongolian word stored in MidWord through the text rendering engine, and at the same time obtaining the glyph ID string corresponding to the Mongolian word stored in MWord through the text rendering engine. The comparison deletion unit is used to compare whether the glyph ID string corresponding to the Mongolian word stored in MidWord is equal to the glyph ID string corresponding to the Mongolian word stored in MWord. If they are equal, it indicates that the Nth control character is a redundant control character, and the Nth control character of the Mongolian word stored in MWord is deleted. If they are not equal, it indicates that the Nth control character is necessary, and the Nth control character of the Mongolian word stored in MWord is not deleted. Addend unit, used to set N = N + 1; The return unit is used to repeat steps S4-S6 until N>M, all control characters are processed, and the loop ends; The output unit is used to finally output the Mongolian words stored in the variable MWord after filtering out redundant control characters.

4. The control character filtering device based on glyph ID conforming to the Mongolian encoding standard as described in claim 3, characterized in that, The text rendering engines include Harfbuzz and Directwrite.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the control character filtering method based on glyph ID conforming to the Mongolian encoding standard as described in claim 1 or 2.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the control character filtering method based on glyph ID conforming to the Mongolian encoding standard as described in claim 1 or 2.

Images