Lexical analysis method, system and response method supporting binary data literal values

Abstract

The present application relates to the field of information technology, in particular to a lexical analysis method and system supporting binary data literal value and a response method, the lexical analysis method comprising the following steps: receiving a user-provided agreement character set; receiving a user-sent request statement encoded by the agreement character set; using a preset single-byte encoding character set to deserialize the request statement; performing lexical analysis on the deserialized request statement to obtain word segmentation; identifying the literal value type of the word segmentation; if the literal value type is binary data literal value, recording the word segmentation after binary escape as binary data, otherwise, decoding the word segmentation using the agreement character set to obtain a restored string; concatenating the obtained binary data and the restored string to obtain a complete request statement. The present application has the beneficial technical effects of being able to restore the key words in the original request statement, while correctly restoring the binary data to ensure the correct execution of the request statement.

Description

Technical Field

[0001] This invention relates to the field of information technology, and more specifically to a lexical analysis method, system, and response method that supports binary data literals. Background Technology

[0002] Lexical analysis: In computer science, this is the process of converting a sequence of characters into a sequence of tokens. Literal values: The value of a constant expression needs to be calculated at compile time, therefore, the type used when declaring a constant must be restricted. Because these types are generally simple and their values ​​are obvious and easy to obtain, they are called "literal types." Binary data literals: Simply put, these are the values ​​of binary data displayed as characters. They generally appear as a jumble of characters. An encoding set may not define every encoding. For example, Chinese may have more than 8000 characters; all data other than the binary data corresponding to these 8000 characters are undefined characters. Therefore, converting binary data into visible characters may not be supported by the character set. For example, converting a byte array [27,29,0,31] to a certain character set, such as GBK or UTF-8, and then printing it will result in a jumble of characters. Hexadecimal string literals refer to converting binary data into hexadecimal and then displaying it as a string. For example, the byte array [27,29,0,31] is converted to 0x1B1D001F. This format is more readable, but obviously takes up more space.

[0003] Binary data literals are commonly used in databases where binary data is directly written via SQL. If data collection, analysis, or rewriting is required before users write data to the database, lexical analysis is necessary to accurately locate the corresponding data or binary literals for correct processing. Existing lexical analysis tools do not support lexical analysis of binary literals, or may result in data loss after processing. Therefore, it is necessary to research technologies that support lexical analysis of binary literals. Summary of the Invention

[0004] The technical problem this invention aims to solve is the current lack of technology supporting binary data literal analysis. This invention proposes a lexical analysis method, system, and response method that supports binary data literals, enabling the analysis of binary data literals.

[0005] To solve the above technical problems, the present invention adopts the following technical solution: a lexical analysis method supporting binary data literals, comprising the following steps:

[0006] Receive the agreed character set provided by the user;

[0007] Receive request statements sent by users that are encoded using the agreed character set;

[0008] The request statement is deserialized using a preset single-byte encoded character set;

[0009] Lexical analysis is performed on the deserialized request statement to obtain word segmentation;

[0010] Identify the literal value type of the segmented words;

[0011] If the literal type is binary data literal, then the word segmentation is escaped in reverse binary and recorded as binary data; otherwise, the literal type must be string literal, and the word segmentation is decoded using the agreed character set to obtain the restored string.

[0012] The obtained binary data and the restored string are concatenated to obtain the complete request statement.

[0013] Preferably, the method for encoding request statements using the agreed character set includes:

[0014] Extract the binary data contained in the request statement;

[0015] Escape the preset characters contained in the binary data;

[0016] Add a preset identifier before the binary data.

[0017] Preferred methods for identifying the literal value type of word segmentation include:

[0018] Check if there is a preset identifier before word segmentation. If there is a preset identifier, the word segmentation is determined to be a binary data literal value; otherwise, the word segmentation is determined to be a string literal value.

[0019] Preferred methods for lexical analysis include:

[0020] Establish a word segmentation library;

[0021] Concatenate each single character with the following single characters one by one until it can match the word segmentation in the word segmentation library;

[0022] All matched single characters constitute a word.

[0023] Preferred methods for building a word segmentation library include:

[0024] List the keywords that the request statement can contain;

[0025] The keywords are encoded using multiple character encodings. The encoded keywords are used as word segments and added to a word segmentation library. The word segmentation library is then associated with the corresponding character encoding.

[0026] A method for transcoding SQL requests to responses, used to implement database responses to MySQL requests, includes the following steps:

[0027] Receive the agreed character set provided by the user;

[0028] Receive MySQL request statements sent by users, encoded using the agreed character set;

[0029] The MySQL request statement is deserialized using a preset single-byte encoded character set;

[0030] Lexical analysis is performed on the deserialized MySQL request statement to obtain word segments;

[0031] Identify the literal value type of the word segment. If the literal value type is binary data literal value, then the word segment is escaped in reverse binary and recorded as binary data. Otherwise, the literal value type must be string literal value. Use the agreed character set to decode the word segment to obtain the restored string.

[0032] The obtained binary data and the restored string are concatenated to obtain the complete MySQL request statement;

[0033] Construct the Abstract Syntax Tree (AST) of the MySQL request statement;

[0034] Transform the MySQL syntax features on the Abstract Syntax Tree (AST) into SQL syntax features;

[0035] Perform namespace mapping corrections on the Abstract Syntax Tree (AST);

[0036] Transform the Abstract Syntax Tree (AST) into SQL request statements using SQL syntax;

[0037] Submit the SQL request statement to the database and obtain the database response;

[0038] The database response is encoded into a MySQL response and sent back to the user.

[0039] Preferred methods for identifying the literal value type of word segmentation include:

[0040] Check if the string "_binary" exists before word segmentation. If the string "_binary" exists, the word segmentation is determined to be a binary data literal value; otherwise, the word segmentation is determined to be a string literal value.

[0041] Preferred methods for lexical analysis include:

[0042] Establish a word segmentation library;

[0043] Concatenate each single character with the following single characters one by one until it can match the word segmentation in the word segmentation library;

[0044] All matched single characters constitute a word.

[0045] Preferred methods for building a word segmentation library include:

[0046] List all the keywords in the MySQL request statement;

[0047] The keywords are encoded using multiple character encodings. The encoded keywords are used as word segments and added to a word segmentation library. The word segmentation library is then associated with the corresponding character encoding.

[0048] A lexical analysis system supporting binary data literals is used to execute the aforementioned lexical analysis method supporting binary data literals. It includes a deserialization module, a word segmentation module, and a recognition and conversion module. The deserialization module receives a user-provided agreed-upon character set and a request statement encoded by the user using the agreed-upon character set. It deserializes the request statement using a preset single-byte encoded character set. The word segmentation module performs lexical analysis on the deserialized request statement to obtain word segments. The recognition and conversion module identifies the literal type of the word segments. If the literal type is binary data literal, the word segment is escaped and recorded as binary data; otherwise, the literal type must be a string literal. The agreed-upon character set is used to decode the word segments to obtain a restored string. The obtained binary data and the restored string are concatenated to obtain a complete request statement.

[0049] Preferably, when the recognition and conversion module identifies the literal value type of the segmented words, it performs the following steps:

[0050] Check if there is a preset identifier before word segmentation. If there is a preset identifier, the word segmentation is determined to be a binary data literal value; otherwise, the word segmentation is determined to be a string literal value.

[0051] Preferably, the word segmentation module stores a word segmentation library. During lexical analysis, single characters are concatenated with subsequent single characters one by one until they can match the word segments in the word segmentation library. All the matched single characters constitute a word segment.

[0052] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the aforementioned lexical analysis method supporting binary data literals.

[0053] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned lexical analysis method supporting binary data literals.

[0054] The beneficial technical effects of this invention include: by first deserializing the request statement using a single-byte encoded character set to obtain single-character content, and then using lexical analysis for word segmentation, the keywords in the original request statement can be recovered, and the binary data can also be correctly recovered. Classifying the recovered data allows for the acquisition of a complete request statement, achieving lexical analysis that supports binary data literals and ensuring the correct execution of the request statement; by establishing a dedicated word segmentation library, accurate recognition of the recovered single characters is achieved; and by transcoding the SQL request before responding, the MySQL database can respond to the SQL request.

[0055] Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0056] The invention will be further described below with reference to the accompanying drawings:

[0057] Figure 1 This is a schematic diagram of the lexical analysis method according to an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of the method flow for encoding request statements in an embodiment of the present invention.

[0059] Figure 3 This is a schematic diagram of the lexical analysis method according to an embodiment of the present invention.

[0060] Figure 4 This is a schematic diagram of the composition of the lexical analysis system according to an embodiment of the present invention.

[0061] Figure 5 This is a schematic diagram of the SQL request transcoding response method according to an embodiment of the present invention.

[0062] Among them: 10, deserialization module; 20, word segmentation module; 30, recognition and conversion module. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.

[0064] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to indicate orientation or positional relationship for the convenience of describing the embodiments and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0065] Before introducing the technical solution of this embodiment, the application scenario of this embodiment will be introduced.

[0066] In computer programs, the value of constant expressions needs to be calculated at compile time, therefore, the types used when declaring parameters must be restricted. Because these types are generally simple and their values ​​are obvious and easily obtained, they are called "literal types." Arithmetic types, references, and pointers are all literal types. Some classes are also literal types; they may contain parameter function members. Although pointers and references can be defined as parameters, their initial values ​​are strictly restricted. The initial value of a parameter pointer must be null, 0, or an object stored at a fixed address. Variables defined within a function body are generally not stored at a fixed address, so parameter pointers cannot point to such variables. Objects defined outside a function body have fixed addresses and can be used to initialize parameter pointers. Functions are allowed to define a class of variables whose scope extends beyond the function itself, such as local static variables. These variables, like variables defined outside the function body, have fixed addresses, so parameter references can be bound to such variables, and parameter pointers can also point to such variables.

[0067] Although literals in computer program source code have a relatively simple function—representing fixed, unchanging values—in information processing, to reduce the number of data encoding and decoding operations, constant values ​​are often stored or transmitted directly using their literal values ​​under a specific encoding. For example, binary data literals are commonly used in databases to directly write binary data via SQL. If data collection, analysis, or rewriting is required before the user writes data to the database, lexical analysis is necessary to accurately locate the corresponding data or binary data before correct processing. Some existing lexical analysis tools do not support lexical analysis of binary literals, or may result in data loss after processing. Therefore, this embodiment provides a lexical analysis method that supports binary data literals. Please refer to the appendix. Figure 1 This includes the following steps:

[0068] Step A01) Receive the agreed character set provided by the user;

[0069] Step A02) Receive the request statement sent by the user, which is encoded using the agreed character set;

[0070] Step A03) Deserializing the request statement using a preset one-byte encoding character set;

[0071] Step A04) Performing lexical analysis on the deserialized request statement to obtain tokens;

[0072] Step A05) Identifying the literal type of the token;

[0073] Step A06) If the literal type is binary data literal, record it as binary data after performing unbinary escaping on the token; otherwise, the literal type must be a string literal, and decode the token using an agreed character set to obtain a restored string;

[0074] Step A07) Concatenating the obtained binary data and the restored string to obtain a complete request statement.

[0075] In the present solution, the encoded request statement is firstly decoded according to a single character, and a lexical analysis step is added after the single character decoding to tokenize single characters, and all keywords and parameters of the original request statement are obtained through tokenization, so that the collection, analysis or rewriting of the request statement can be supported. In the process, the literal of binary data is processed separately, and only unbinary escaping is required according to the binary data. For example, after parameterizing the SQL request statement: select * from table1 where name = 'Wang Ke', it becomes: select * from table1 where name = ?, parameter 1: _binary 'Wang Ke'. Moreover, the backslash "\", single quote "'" and terminator "character code 0" in Chinese data will all be escaped.

[0076] The request statement is encoded using UTF-8. If the value of name uses another encoding, such as Chinese characters encoded in GBK, and the value of name exceeds the coverage of UTF-8 to GBK encoding, information loss will occur, causing an error in the request statement. And this error situation is not limited to text encoding. When other binary data is written as literals, once the UTF-8 encoding cannot completely cover it, information loss will also occur, resulting in an error in the execution of the request statement.

[0077] In this embodiment, the literal is firstly decoded into single characters according to a preset character set, for example, single character decoding is performed using the single character set ISO-8859-1, which will obtain:

[0078] s, e, l, e, c, t, , *, , f, r, o, m, , t, a, b, l, e, 1, , w, h, e, r, e, , n, a, m, e, =, _, b, i, n, a, r, y, ’, ě, í, ° Then, lexical analysis is used for word segmentation, and the obtained word segmentation positions are as follows:

[0079] s、e、l、e、c、t||*||f、r、o、m||t、a、b、l、e、1||w、h、e、r、e||n、a、m、e|=|_、b、i、n、a、r、y|’|ě,í、°、 wherein the symbol "|" indicates a word segmentation position, single characters belonging to the same word segmentation are spliced into a character string, and ě,í、° represent garbled characters. The obtained word segments are as follows:

[0080]

[0081] Corresponding processing is performed according to the literal type of the word segmentation. If the literal type is a binary data literal, the word segmentation is recorded as binary data after reverse binary escaping. That is, _binary’ after performing reverse binary escaping and then UTF-8 decoding, the characters obtained are: 'Wang Ke'. Otherwise, it is processed as a string literal. Thus, the processed statement can be obtained: select||*||from||table1||where||name|=|’Wang Ke’|. The restored SQL request statement can be obtained by deleting the symbol "|".

[0082] See the appendix Figure 2 , the method for using a request statement with a conventional character set encoding includes:

[0083] Step B01) intercepting binary data contained in the request statement;

[0084] Step B02) performing escape processing on preset characters contained in the binary data;

[0085] Step B03) adding a preset identifier before the binary data.

[0086] The method for identifying the literal type of a word segmentation includes: checking whether a preset identifier exists before the word segmentation, if the preset identifier exists, determining that the word segmentation is a binary data literal, otherwise, determining that the word segmentation is a string literal.

[0087] On the other hand, the embodiment provides a specific method for performing lexical analysis, please refer to the appendix Figure 3 , comprising:

[0088] Step C01) establishing a word segmentation library;

[0089] Step C02) splicing a single character with subsequent single characters one by one until it can be matched with a word segmentation in the word segmentation library;

[0090] Step C03) Matches all single characters to form a word segment.

[0091] On the other hand, this embodiment provides a specific method for building a word segmentation library, including:

[0092] List the keywords that the request statement can contain;

[0093] Keywords are encoded using multiple character encodings. The encoded keywords are used as word segments and added to the word segmentation library. The word segmentation library is then associated with the corresponding character encoding.

[0094] The beneficial technical effects of this embodiment include: by first deserializing the request statement using a single-byte encoded character set to obtain single-character content, and then using lexical analysis for word segmentation, the keywords in the original request statement can be recovered, and the binary data can also be correctly recovered. By classifying and processing the recovered data, the complete request statement can be obtained, realizing lexical analysis that supports binary data literals and ensuring the correct execution of the request statement; and by establishing a dedicated word segmentation library, the accurate recognition of the recovered single characters can be achieved.

[0095] A lexical analysis system that supports binary data literals is used to perform the lexical analysis method that supports binary data literals as described above. Please refer to the appendix. Figure 4 The system includes a deserialization module 10, a word segmentation module 20, and a recognition and conversion module 30. The deserialization module 10 receives a user-provided character set and a request statement encoded by the user using the character set. It then deserializes the request statement using a preset single-byte character set. The word segmentation module 20 performs lexical analysis on the deserialized request statement to obtain word segments. The recognition and conversion module 30 identifies the literal value type of the word segments. If the literal value type is a binary data literal value, the word segment is escaped and recorded as binary data. Otherwise, the literal value type must be a string literal value. The system decodes the word segments using the character set to obtain the restored string. The obtained binary data and the restored string are concatenated to obtain the complete request statement.

[0096] When the recognition and conversion module 30 identifies the literal value type of the segmented words, it performs the following steps:

[0097] Check if there is a preset identifier before word segmentation. If there is a preset identifier, the word segmentation is determined to be a binary data literal value; otherwise, the word segmentation is determined to be a string literal value.

[0098] The word segmentation module 20 stores a word segmentation library. During lexical analysis, it concatenates single characters with subsequent single characters one by one until it can match the word segmentation in the word segmentation library. All the matched single characters constitute a word segmentation.

[0099] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the lexical analysis method supporting binary data literals as described above.

[0100] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned lexical analysis method supporting binary data literals.

[0101] A method for transcoding SQL requests and responses, used to implement database responses to MySQL requests; please refer to the appendix. Figure 5 This includes the following steps:

[0102] Step D01) Receive the agreed character set provided by the user;

[0103] Step D02) Receive the MySQL request statement sent by the user, which is encoded using the agreed character set;

[0104] Step D03) Deserialize the MySQL request statement using a preset single-byte encoded character set;

[0105] Step D04) Perform lexical analysis on the deserialized MySQL request statement to obtain word segmentation;

[0106] Step D05) Identify the literal value type of the word segment. If the literal value type is binary data literal value, then the word segment is escaped in reverse binary and recorded as binary data. Otherwise, the literal value type must be string literal value. Use the agreed character set to decode the word segment to obtain the restored string.

[0107] Step D06) Concatenate the obtained binary data and the restored string to obtain the complete MySQL request statement;

[0108] Step D07) Construct the Abstract Syntax Tree (AST) of the MySQL request statement;

[0109] Step D08) convert the MySQL syntax features on the Abstract Syntax Tree (AST) into SQL syntax features;

[0110] Step D09) Correct the namespace mapping on the Abstract Syntax Tree (AST);

[0111] Step D10) Transform the Abstract Syntax Tree (AST) into an SQL request statement in SQL syntax;

[0112] Step D11) Submit the SQL request statement to the database and obtain the database response;

[0113] Step D12) encoding the database response into a MySQL response and feeding it back to the user.

[0114] Wherein, the method for identifying the literal type of a token includes: checking whether the string "_binary" exists before the token, if the string "_binary" exists, determining that the token is a binary data literal, otherwise determining that the token is a string literal.

[0115] When SQL request statements using different character sets are encoded, that is, SQL statements containing setBinaryStream, if the prior art is directly used to transcode the response, there will also be problems.

[0116] For example:

[0117] SQL request statement:

[0118] PreparedStatement st = conn.prepareStatement("update test_data set b = ? where a = ? "); ByteArrayInputStream bais = new ByteArrayInputStream("Wang Ke".getBytes("GBK"));

[0119] st.setBinaryStream(1,bais);

[0120] st.setInt(2,1);

[0121] st.executeUpdate();

[0122] st.close();

[0123] Wherein, parameter 1 is the binary data of several characters of "Chinese data" encoded by GBK, which is written by the setBinaryStream method.

[0124] After encoding the SQL request statement in UTF8, the corresponding binary data is: 117,112,100.97, 116.101, 32, 116,101,115,116,95,100,97.116.97.32, 115,101.116, 32, 98,61,95,98.105.110.97,114,121.39'-17.-65,-67,-17,-65.-67.-17,-65,-67,-17,-65,-67,-17,-65.-67,-17,-65,-67,-17,-65,-67,-17,-65,-67,39, 32, 119,104,101,114.101.32, 97.61.4.

[0125] Direct deserialization by UTF-8 gives the following SQL request statement:

[0126] update test_data set b=_binary'□□□□□'where a=1.

[0127] It can be seen that information loss has occurred, because UTF8 encoding cannot be compatible with GBK encoding. At this time, the SQL request statement cannot be executed correctly, resulting in a program error.

[0128] Deserialize with ISO-8859-1 first, and the obtained SQL request statement is:

[0129] update test_data set b=_binary'?D??êy?Y'where a=1.

[0130] Process _binary'?D??êy?Y' as binary data, perform de-binary escaping, and then decode with GBK encoding to obtain the correct Chinese parameter, which is "Wang Ke".

[0131] Wherein, the method for lexical analysis in this embodiment includes: establishing a word segmentation library; splicing single characters with subsequent single characters one by one until the word can be matched with the word segmentation in the word segmentation library; all matched single characters constitute a word segmentation.

[0132] The method for establishing a word segmentation library includes: listing all keywords of MySQL request statements; encoding the keywords according to multiple character encodings, adding the encoding of the keywords as word segments to the word segmentation library, and associating the word segmentation library with the corresponding character encodings. By trans-coding the SQL request and then responding to it, this solution enables the MySQL database to respond to the SQL request.

[0133] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A lexical analysis method supporting binary data literals, characterized in that, Includes the following steps: Receive the agreed character set provided by the user; Receive request statements sent by users that are encoded using the agreed character set; The request statement is deserialized using a preset single-byte encoded character set; Lexical analysis is performed on the deserialized request statement to obtain word segmentation; Identify the literal value type of the segmented words; If the literal type is binary data literal, then the word segmentation is escaped in reverse binary and recorded as binary data; otherwise, the literal type must be string literal, and the word segmentation is decoded using the agreed character set to obtain the restored string. The obtained binary data and the restored string are concatenated to obtain the complete request statement; The method for using the agreed character set encoding for request statements includes: Extract the binary data contained in the request statement; Escape the preset characters contained in the binary data; Add a preset identifier before the binary data; Methods for performing lexical analysis include: Establish a word segmentation library; Concatenate each single character with the following single characters one by one until it can match the word segmentation in the word segmentation library; All matched single characters constitute a word.

2. The lexical analysis method supporting binary data literals according to claim 1, characterized in that, Methods for identifying the literal value type of word segmentation include: Check if there is a preset identifier before word segmentation. If there is a preset identifier, the word segmentation is determined to be a binary data literal value; otherwise, the word segmentation is determined to be a string literal value.

3. The lexical analysis method supporting binary data literals according to claim 1, characterized in that, Methods for building a word segmentation library include: List the keywords that the request statement can contain; The keywords are encoded using multiple character encodings. The encoded keywords are used as word segments and added to a word segmentation library. The word segmentation library is then associated with the corresponding character encoding.

4. A method for transcoding SQL requests to responses, used to implement database responses to MySQL requests, characterized in that, Includes the following steps: Receive the agreed character set provided by the user; Receive MySQL request statements sent by users, encoded using the agreed character set; The MySQL request statement is deserialized using a preset single-byte encoded character set; Lexical analysis is performed on the deserialized MySQL request statement to obtain word segments; Identify the literal value type of the word segment. If the literal value type is binary data literal value, then the word segment is escaped in reverse binary and recorded as binary data. Otherwise, the literal value type must be string literal value. Use the agreed character set to decode the word segment to obtain the restored string. The obtained binary data and the restored string are concatenated to obtain the complete MySQL request statement; Construct the Abstract Syntax Tree (AST) of the MySQL request statement; Transform the MySQL syntax features on the Abstract Syntax Tree (AST) into SQL syntax features; Perform namespace mapping corrections on the Abstract Syntax Tree (AST); Transform the Abstract Syntax Tree (AST) into SQL request statements using SQL syntax; Submit the SQL request statement to the database and obtain the database response; The database response is encoded into a MySQL response and sent back to the user. Methods for performing lexical analysis include: Establish a word segmentation library; Concatenate each single character with the following single characters one by one until it can match the word segmentation in the word segmentation library; All matched single characters constitute a word.

5. The SQL request transcoding response method according to claim 4, characterized in that, Methods for identifying the literal value type of word segmentation include: Check if the string "_binary" exists before word segmentation. If the string "_binary" exists, the word segmentation is determined to be a binary data literal value; otherwise, the word segmentation is determined to be a string literal value.

6. The SQL request transcoding response method according to claim 4, characterized in that, Methods for building a word segmentation library include: List all the keywords in the MySQL request statement; The keywords are encoded using multiple character encodings. The encoded keywords are used as word segments and added to a word segmentation library. The word segmentation library is then associated with the corresponding character encoding.

7. A lexical analysis system supporting binary data literals, used to execute the lexical analysis method supporting binary data literals as described in any one of claims 1 to 3, characterized in that, The system includes a deserialization module, a word segmentation module, and a recognition and conversion module. The deserialization module receives a user-provided agreed-upon character set and a request statement encoded by the user using the agreed-upon character set. It then deserializes the request statement using a preset single-byte encoded character set. The word segmentation module performs lexical analysis on the deserialized request statement to obtain word segments. The recognition and conversion module identifies the literal value type of the word segments. If the literal value type is binary data literal, the word segment is escaped and recorded as binary data; otherwise, the literal value type must be a string literal. The agreed-upon character set is used to decode the word segments to obtain a restored string. The obtained binary data and the restored string are concatenated to obtain a complete request statement.

8. The lexical analysis system supporting binary data literals according to claim 7, characterized in that, When the recognition and conversion module identifies the literal value type of the segmented words, it performs the following steps: Check if there is a preset identifier before word segmentation. If there is a preset identifier, the word segmentation is determined to be a binary data literal value; otherwise, the word segmentation is determined to be a string literal value.

9. The lexical analysis system supporting binary data literals according to claim 7 or 8, characterized in that, The word segmentation module stores a word segmentation library. During lexical analysis, single characters are concatenated with subsequent single characters one by one until they can match the word segments in the word segmentation library. All the matched single characters constitute a word segment.

10. A computer system, characterized by The computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the lexical analysis method supporting binary data literals as described in any one of claims 1 to 3.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the lexical analysis method supporting binary data literals as described in any one of claims 1 to 3.

Images