Data processing system based on Dlang language and storage medium

Abstract

The invention relates to a data processing system based on a Dlang language and a storage medium, and the system comprises a lexical analysis module which preprocesses a Dlang statement input by a user and converts the Dlang statement into a corresponding character string; the character string is segmented to obtain a plurality of mark sequences of the Dlang statement, and the mark sequences are recorded as two-tuple lt; marking types and marking values gt; the mark type comprises a keyword, an identifier, an operator, a character value and a punctuation mark; the syntactic analysis module generates a corresponding abstract syntax tree according to the mark sequence of the Dlang statement; the semantic analysis module verifies the abstract syntax tree and generates an intermediary code of the Dlang statement, the intermediary code is used for compiling to obtain an executable statement so as to process data in a preset database, and by means of the data processing method and device, the problem of how to improve the data processing flexibility in the database is solved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing system and a storage medium based on the Dlang language. Background Art

[0002] The SQL language is a structured and declarative query language based on the relational model. Its main function is to model data in the form of tables and provides a series of syntaxes that allow users to obtain data through direct declarations, with less concern for the specific steps or operation details required to obtain this data. Some limitations in the design and functionality of the SQL language make it difficult to meet some emerging data analysis requirements.

[0003] First, the operation of the SQL language depends on a structured data model. Data is generally organized into tables in a database and divided into several columns with clear data types. Second, it is difficult for the SQL language to directly meet the related requirements of stream computing. If users use the SQL language to develop stream computing tasks, the corresponding database software needs to first cache the stream data and convert it into a table form before using the SQL language for operations such as insert, delete, update, and query. In addition, it is difficult for the SQL language to directly meet the query and calculation requirements for data types such as tuples, large arrays, matrices, dictionaries, and tensors.

[0004] Currently, for the problem of how to improve the flexibility of data processing in a database in the related art, no effective solution has been proposed. Summary of the Invention

[0005] Embodiments of this application provide a data processing system and a storage medium based on the Dlang language to at least solve the problem of how to improve the flexibility of data processing in a database in the related art.

[0006] In a first aspect, embodiments of this application provide a data processing system based on the Dlang language. The system includes a lexical analysis module, a syntactic analysis module, and a semantic analysis module; The lexical analysis module is used to preprocess the Dlang statement input by the user and convert the Dlang statement into a corresponding string; The lexical analysis module is further used to split the string to obtain several token sequences of the Dlang statement, where the token sequence is denoted as a binary tuple <token type, token value>, and the token type includes keywords, identifiers, operators, literals, and punctuation marks; The syntactic analysis module is used to generate a corresponding abstract syntax tree according to the token sequence of the Dlang statement; The semantic analysis module is used to verify the abstract syntax tree and generate intermediate code for the Dlang statement, where the intermediate code is used to compile into executable statements to process data in a preset database.

[0007] In some embodiments, when the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value>, and the keyword is a keyword of functional programming: The syntax analysis module is used to parse the token sequence of the functional programming to obtain the abstract syntax tree of the functional programming; The semantic analysis module is used to verify the abstract syntax tree of the functional programming and generate intermediate code for the Dlang statement.

[0008] In some embodiments, when the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value>, and the keyword is a keyword of metaprogramming: The syntax analysis module is used to parse the token sequence of the metaprogramming to obtain the abstract syntax tree of the metaprogramming; The semantic analysis module is used to verify the abstract syntax tree of the metaprogramming and generate intermediate code for the Dlang statement.

[0009] In some embodiments, when the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value>, and the keyword is a keyword of object-oriented programming: The syntax analysis module is used to parse the token sequence of the object-oriented programming to obtain the abstract syntax tree of the object-oriented programming; The semantic analysis module is used to verify the abstract syntax tree of the object-oriented programming and generate intermediate code for the Dlang statement.

[0010] In some embodiments, when the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value>, and the keyword is a keyword of conditional branch statements or loop bodies: The syntax analysis module is used to parse the token sequence of the conditional branch statements or loop bodies to obtain the corresponding abstract syntax tree; the semantic analysis module is used to verify the abstract syntax tree and generate intermediate code for the Dlang statement, where the conditional branch statement is an if-else conditional branch statement, and the loop body includes a for loop body and a do-while loop body.

[0011] In some of these embodiments, the system further includes an optimization module; The optimization module is configured to perform constant folding optimization on the intermediate code of the Dlang statement to obtain optimized intermediate code.

[0012] In some of these embodiments, the system further includes a database execution module; The database execution module is configured to run the executable statement to process data in a preset database and return a data processing result to the user, where the preset database is a DolphinDB database.

[0013] In some of these embodiments, the database execution module includes a first database execution module and a second database execution module; The first database execution module runs a first executable statement to create a first streaming data table, and the second database execution module runs a second executable statement to create a second streaming data table; The first database execution module or the second database execution module runs a third executable statement to perform real-time streaming data calculation among the preset database, the first streaming data table, and the second streaming data table.

[0014] In some of these embodiments, the compiled executable statement supports processing data of multiple data types under twelve categories, where the twelve categories include the VOID category, the logical category LOGICAL, the temporal category TEMPORAL, the floating-point category FLOATING, the literal category LITERAL, the binary category BINARY, the system category SYSTEM, the mixed category MIXED, the other category OTHER, the decimal category DECIMAL, and the array category ARRAY.

[0015] In a second aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the system described in the first aspect above is implemented.

[0016] Compared with the related art, an embodiment of the present application provides a data processing system and a storage medium based on the Dlang language. The system includes a lexical analysis module, a syntactic analysis module, and a semantic analysis module. The lexical analysis module preprocesses the Dlang statements input by the user, converts the Dlang statements into corresponding strings, and then splits the strings to obtain several token sequences of the Dlang statements. The token sequences are denoted as tuples <token type, token value>, and the token types include keywords, identifiers, operators, literals, and punctuation marks. The syntactic analysis module generates a corresponding abstract syntax tree according to the token sequences of the Dlang statements. The semantic analysis module verifies the abstract syntax tree and generates an intermediate code of the Dlang statements. The intermediate code is used to compile into executable statements to process the data in a preset database. Through this system, the generation of token sequences of Dlang statements based on token types such as keywords, identifiers, operators, literals, and punctuation marks is realized, which enriches the types of subsequent generated executable statements, improves the flexibility of data processing in the database, and solves the problem of how to improve the flexibility of data processing in the database. Brief Description of the Drawings

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 is a schematic structural diagram of a data processing system based on the Dlang language according to an embodiment of the present application; Figure 2 is a schematic diagram of the generation of an abstract syntax tree according to an embodiment of the present application; Figure 3 is a schematic diagram of the generation of intermediate code according to an embodiment of the present application; Figure 4 is a schematic diagram of real-time calculation of streaming data according to an embodiment of the present application; Figure 5 is a schematic internal structure diagram of an electronic device according to an embodiment of the present application. Detailed Embodiments

[0018] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be described and explained below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0019] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, without creative efforts, the present application can also be applied to other similar scenarios based on these drawings. In addition, it can also be understood that although the efforts made in such a development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacturing, or production changes based on the technical content disclosed in the present application are only conventional technical means and should not be understood as the content disclosed in the present application being insufficient.

[0020] Reference to "embodiment" in the present application means that a specific feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical terms or scientific terms involved in the present application should have the ordinary meaning understood by those of ordinary skill in the technical field to which the present application belongs. The words such as "a", "an", "one", "the", etc. involved in the present application do not represent a quantity limitation and can represent a singular or plural number. The terms "including", "comprising", "having" and any variations thereof involved in the present application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may further include unlisted steps or units, or may further include other steps or units inherent to these processes, methods, products, or devices. The terms "connected", "coupled", etc. involved in the present application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" involved in the present application refers to two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. The terms "first", "second", "third", etc. involved in the present application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0022] An embodiment of the present application provides a data processing system based on the Dlang language. Figure 1 It is a schematic structural diagram of the data processing system based on the Dlang language according to the embodiment of the present application, as Figure 1As shown in the figure, the system includes a lexical analysis module, a syntactic analysis module, and a semantic analysis module; The lexical analysis module is used to preprocess the Dlang statements input by the user and convert the Dlang statements into corresponding strings; Specifically, the lexical analysis module preprocesses the Dlang statements input by the user, deleting newline characters, comments, spaces, form feed characters, tab characters, etc. that have no actual syntactic meaning. After preprocessing, the input Dlang script program (Dlang statements) will be converted into a single line of string.

[0023] The lexical analysis module is also used to split the string to obtain several token sequences of Dlang statements. Among them, the token sequence is recorded as a binary tuple <token type, token value>, and the token types include keywords, identifiers, operators, literals, and punctuation marks; Specifically, the lexical analysis module will compare all predefined token patterns, identify and determine the start and end points of each token in the string to split the string into several substrings; each substring obtained by splitting is recorded as a binary tuple <token type, token value> through pattern matching, such as <keyword, insert>, <literal, "hello">, etc., and the binary tuples are output in sequence in the form of a sequence, that is, the token sequence is obtained.

[0024] It should be noted that all possible token patterns in the Dlang language are predefined in the lexical analysis module. The token types of these tokens include keywords, identifiers, operators, literals, and punctuation marks. Keywords are words reserved in the Dlang language itself with special language functions, such as "insert", "select", etc. It should be noted that keywords cannot be used for other purposes such as identifiers. An identifier is a name, which can be the name of a variable, a function, an array, or other objects defined in the Dlang language; a literal is a constant, such as 100, "Hello", etc.; punctuation marks are the punctuation marks defined in the Dlang language, such as "(", ")", etc. For example, the string "if" can be recognized and converted into a token expressing "if". If an unrecognizable string is encountered, such as "o / / a2d%f&", the lexical analysis module will report a lexical error.

[0025] The syntactic analysis module is used to generate a corresponding abstract syntax tree according to the token sequence of the Dlang statement; It should be noted that the syntactic analysis module generates a corresponding abstract syntax tree according to the token sequence of the Dlang statement, Figure 2 is a schematic diagram of the generation of the abstract syntax tree according to the embodiments of the present application, such as Figure 2As shown, each node in an abstract syntax tree is a syntactic structure generated by converting a sequence of tokens. For example, a sequence of tokens such as "if", "(", "a", ">", "10", ")" will be recognized and converted into an if conditional statement, where the conditional expression is "a > 10". In other words, syntactic structure 1 can be "if(a>10)", syntactic structure 2 can be "return(a\10)", syntactic structure 3 can be "else if(a<=10&&a>1)", syntactic structure 4 can be "retum a", syntactic structure 5 can be "else", syntactic structure 6 can be "b =abs(a)*10", and syntactic structure 7 can be "return b". If an unconvertible token is encountered, the syntactic analysis module will report a syntax error.

[0026] A semantic analysis module is used to verify the abstract syntax tree and generate intermediate code for Dlang statements. The intermediate code is used to compile into executable statements to process data in a preset database.

[0027] Specifically, Figure 3 is a schematic diagram generated according to the intermediate code of the embodiment of the present application. As Figure 3As shown in the figure, first, the semantic analysis module will deduce the specific type of each grammatical structure appearing in the abstract syntax tree. The specific deduction methods include deduction based on the assignment relationship (for example, if the integer constant "10" is assigned to the variable a, then the type of the variable a is integer), deduction based on the database meta-information (for example, if a certain column c existing in the database is referenced in the Dlang statement, and the database meta-information records its type as single-precision floating point number, then the type of c is single-precision floating point number), deduction based on the operation rules (for example, the content of the variable c is obtained by assigning the result of the operation a + b, where a is an integer and b is a double-precision floating point number. According to the operation rules, the type of a + b is double-precision floating point number, so the type of c is double-precision floating point number), and so on; Second, the semantic analysis module will determine whether an abstract syntax tree conforms to the semantic rules of Dlang. For example, it checks whether the variable "a" and the constant "10" meet the semantic requirements of the ">" comparison operation. In other words, the specific type 1 can be selected as "if(a:int>10:int)", the specific type 2 can be selected as "return(a:int\10:int):int", the specific type 3 can be selected as "else if(a:int<=10:int&&a:int>1:int)", the specific type 4 can be selected as "retum(a:int):int", the specific type 5 can be selected as "else", the specific type 6 can be selected as "b:int=abs(a:int)*10:int", and the specific type 7 can be selected as "return(b:int):int". If a statement that does not meet the semantic requirements is encountered, the semantic analysis module will report a semantic error.

[0028] Through the lexical analysis module, syntactic analysis module and semantic analysis module in the embodiments of the present application, the generation of the token sequence of the Dlang statement based on token types such as keywords, identifiers, operators, literals, and punctuation marks is realized, which enriches the types of the subsequent generated executable statements, improves the flexibility of data processing in the database, and solves the problem of how to improve the flexibility of data processing in the database.

[0029] In some of these embodiments, when the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value>, and the keyword is a keyword of functional programming: The syntactic analysis module is used to parse the token sequence of functional programming to obtain the abstract syntax tree of functional programming; The semantic analysis module is used to verify the abstract syntax tree of functional programming and generate the intermediate code of the Dlang statement.

[0030] It should be noted that functional programming is a relatively new programming paradigm that defines a program as a process of obtaining an output value through multiple mappings of a set of input values, where each mapping is a function. Functional programming is different from imperative programming, which defines a program as a process of updating a state by executing a series of instructions on it. A function is the basic unit in functional programming. A function can have its own name and can also be anonymous. It can be passed as a parameter to another parameter and can also be the return value of another function. This embodiment defines and supports named functions (i.e., functions with names), anonymous functions, anonymous expressions, partial applications, closures, or higher-order functions, etc. This embodiment also supports user-defined various functions and designs and provides some built-in functions to implement specific functions.

[0031] In some of these embodiments, when the token sequence of a Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value> and the keyword is a keyword of metaprogramming: A syntactic analysis module for parsing the token sequence of metaprogramming to obtain an abstract syntax tree of metaprogramming; A semantic analysis module for verifying the abstract syntax tree of metaprogramming and generating an intermediate code of a Dlang statement.

[0032] It should be noted that metaprogramming is specifically divided into two types: function metaprogramming and SQL metaprogramming.

[0033] Specifically for function metaprogramming, it refers to a metaprogramming method of dynamically obtaining function definitions and parameters through means such as parameter passing. That is, the user does not have to explicitly write the code for calling a function, but stores the function name and parameters in variables and passes these variables as parameters to other functions. Other functions then assemble the function name and its parameters together at runtime and make a call. This method is usually applied to some complex data analysis scenarios, which allows users to write very complex conditional branches and function calls to achieve complex analysis requirements. The function definition is a data type supported in this application (function definition under the system category SYSTEM), that is, FUNCTIONDEF. It can be dynamically obtained through the built-in function funcByName according to the input function name and parameters. In the following example, the variable name stores the name of the function sin, and the variable v stores a set of parameters. By passing name and v to the built-in function funcByName respectively, the program can dynamically obtain the definition of the sin function and its parameters at runtime. The lexical, syntactic, and semantic analysis modules of this embodiment can parse corresponding function definitions, assign function definitions to variables, and statements such as function calls and convert them into intermediate code.

[0034] Function metaprogramming supports dynamically obtaining an anonymous function, that is, defining a function in the form of a string, taking it as a parameter, and dynamically parsing the function through the built-in parseExpr function. The function of the parseExpr function is to convert the input string representing a function into intermediate code after lexical, syntactic, and semantic analysis, and finally return a data of the code type, which represents a piece of code that can be executed through the built-in eval function. Three methods can be used in this embodiment to dynamically call a function: ① Call the function definition through the higher-order function call, syntax: function name.call(parameter list); ② Call the function definition through the operator "()", syntax: function name(parameter list); ③ Call the function definition through the function at, syntax: at (function name, parameter).

[0035] Specifically for SQL metaprogramming, it refers to executing SQL statements by dynamically generating SQL statements during code execution. This method facilitates users to generate corresponding SQL statements based on information that can only be obtained during program runtime, thereby achieving flexible data analysis. This embodiment provides two ways of SQL metaprogramming: function-based SQL metaprogramming and macro-variable-based SQL metaprogramming. Function-based SQL metaprogramming refers to a method of generating SQL meta-code through combined calls of built-in metaprogramming functions; macro-variable-based SQL metaprogramming means that in metaprogramming statements, macro variables can be used to replace single columns or multiple columns, and the corresponding columns will be dynamically obtained during execution. Among them, macro variables are divided into single-column macro variables and multi-column macro variables.

[0036] Single-column macro variable: It refers to assigning the field name of a column in the table to a variable, and using the "_$" symbol to dynamically obtain the value of the variable. For example, if the variable is defined as name="sym", the meta-code can be written as <select _$name from t>, which will be replaced with SELECT sym FROM t when executed.

[0037] Multi-column macro variable: It refers to a variable that assigns the names of multiple fields in a table, and the "_$$" symbol is used to dynamically obtain multiple values in this variable. For example, if the variable is defined as names=["sym", "time"], the meta-code can be written as< / select> , which will be replaced with SELECT sym, time FROM t during execution.

[0038] In some of these embodiments, when the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value>, and the keyword is a keyword for object-oriented programming:[[]]END]] The syntactic analysis module is used to parse the token sequence of object-oriented programming to obtain the abstract syntax tree of object-oriented programming; The semantic analysis module is used to verify the abstract syntax tree of object-oriented programming and generate the intermediate code of the Dlang statement.

[0039] It should be noted that this embodiment supports object-oriented programming. First, this embodiment supports the concept of a class. A class defines the attributes of an object, the constructor method of the object, and the behavior of the object. Second, this embodiment supports class inheritance. If a class inherits from another class, the former is called a derived class and the latter is called a parent class. The derived class contains the member variables, constructors, and methods of the parent class. Third, this embodiment supports polymorphism. If a derived class defines a method with the same function signature as a method in the parent class, then this method will override the method in the parent class. The function signature includes the function name, parameter list, and return value type of a function.

[0040] In some of these embodiments, when the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a binary tuple <keyword, token value>, and the keyword is the keyword of a conditional branch statement or a loop body: A syntactic analysis module, which is used to parse the token sequence of the conditional branch statement or the loop body to obtain the corresponding abstract syntax tree; a semantic analysis module, which is used to verify the abstract syntax tree and generate the intermediate code of the Dlang statement. Among them, the conditional branch statement is an if-else conditional branch statement, and the loop body includes a for loop body and a do-while loop body.

[0041] It should be noted that this embodiment supports if-else conditional branch statements. The lexical analysis module can recognize and parse keywords such as if, else if, and else, as well as variable names, constant values, and arithmetic operators in the conditions. The syntactic analysis module can further parse statement blocks such as if, else if, and else, as well as conditional expressions. The semantic analysis module can further associate the recognized statement blocks and expressions to form a complete executable intermediate code. This embodiment supports two types of loop bodies: for and do-while. The lexical analysis module can recognize keywords such as for, do, and while. The syntactic analysis module can further parse the for and do-while loop bodies. The semantic analysis module can further perform semantic checks on the loop bodies and finally form a complete executable intermediate code.

[0042] In addition, this embodiment also supports directly reading table data from the DolphinDB database. For example, a table can be read from the DolphinDB database into memory by calling the loadTable function. Data can be queried in the DolphinDB database by executing SQL and its results can be obtained. There is theoretically no limit to the amount of data that the Dlang language itself can handle. For example, users can use Dlang to construct and use an array of any length. It supports many operators such as addition, subtraction, multiplication, division, logical AND, logical OR, logical NOT, greater than, less than, equal to, logical right shift, etc. That is, the lexical analysis module, syntactic analysis module, and semantic analysis module of this embodiment support parsing functions, statements, operators, etc. relied on by these functions.

[0043] In some of these embodiments, the system further includes an optimization module; The optimization module is used to perform constant folding optimization on the intermediate code of Dlang statements to obtain the optimized intermediate code.

[0044] It should be noted that the optimizer will perform optimizations such as constant folding on the above intermediate code. For example, for the arithmetic expression "a = 100 * 200 * 300", although two multiplication operation nodes can be generated to calculate the three operands and finally assign the result to a, since the processor can directly perform multiplication operations at the instruction level, the optimizer will directly calculate 100 * 200 * 300 to get 6000000, and convert the original arithmetic expression into an assignment expression "a = 6000000".

[0045] In some of these embodiments, the system further includes a database execution module; The database execution module is used to run executable statements to process data in a preset database and return the data processing results to the user. The preset database is the DolphinDB database.

[0046] In some of these embodiments, the database execution module includes a first database execution module and a second database execution module; The first database execution module runs a first executable statement to create a first streaming data table, and the second database execution module runs a second executable statement to create a second streaming data table; The first database execution module or the second database execution module runs a third executable statement to perform real-time streaming data calculation among the preset database, the first streaming data table, and the second streaming data table.

[0047] It should be noted that Figure 4 is a schematic diagram of real-time streaming data calculation according to the embodiments of the present application, such as Figure 4As shown, the stream data table is a memory table for storing and publishing stream data. Users can use Dlang statements in a DolphinDB database instance through the data processing system of this embodiment to create a stream data table and publish it. This database instance then becomes the publisher. Other users can subscribe to the published stream data table in other DolphinDB database instances, and the corresponding instances become subscribers. The input real-time stream data can be written into the stream data table of the publisher through the data processing system of this embodiment and be synchronously replicated to all subscribers in real time through the data synchronization link. Users do not need to concern themselves with the data synchronization process. Users can perform real-time streaming computing tasks at the publisher and subscriber ends. Support for performing SQL queries on the stream data table or using functional programming, metaprogramming, object-oriented programming, etc. for analysis. Whether at the publisher or subscriber end, the data in the stream data table includes newly written real-time data, that is, users can perform real-time data stream computing tasks.

[0048] In some embodiments, the compiled executable statements support processing data of multiple data types under twelve categories. Among them, the twelve categories include the VOID category, the logical category LOGICAL, the temporal category TEMPORAL, the floating-point category FLOATING, the literal category LITERAL, the binary category BINARY, the system category SYSTEM, the mixed category MIXED, the other category OTHER, the decimal category DECIMAL, and the array category ARRAY.

[0049] Specifically, the multiple data types under the twelve categories are specifically thirty-eight data types: 1. VOID category: Includes the VOID data type, used to represent null values, occupying 1 byte.

[0050] 2. Logical category LOGICAL: Includes the boolean (BOOL) type, used to identify true (True) or false (False), occupying 1 byte.

[0051] 3. Integral category INTEGRAL: Includes numeric data types without decimal points such as character (CHAR), short integer (SHORT), integer (INT), long integer (LONG), and compressed integer (COMPRESSED), occupying 1, 2, 4, 8, and 1 byte respectively.

[0052] 4. TEMPORAL category: It includes data types such as DATE, MONTH, TIME, MINUTE, SECOND, DATETIME, TIMESTAMP, NANOTIME, NANOTIMESTAMP, and DATEHOUR, which occupy 4 to 8 bytes. Among them, nanosecond and nanosecond timestamp are not defined in the SQL standard, but they are also very common data types in the fields of financial computing and the Internet of Things. Dlang defines and supports them.

[0053] 5. FLOATING category: It includes single-precision floating-point number (FLOAT) and double-precision floating-point number (DOUBLE), which occupy 4 and 8 bytes respectively.

[0054] 6. LITERAL category: It includes data types such as SYMBOL, STRING, and BLOB, and the number of occupied bytes is not fixed.

[0055] 7. BINARY category: It includes data types such as 128-bit integer (INT128), unique ID (UUID), IP address (IPADDR), and geographical coordinate value (POINT). They store the corresponding data in binary form, and these data types are commonly used in the fields of financial computing and the Internet of Things.

[0056] 8. SYSTEM category: It defines data types required for many specific functions, specifically including types such as FUNCTIONDEF, HANDLE, CODE, DATASOURCE, RESOURCE, and DURATION. The support for functional programming, metaprogramming, and object-oriented programming is achieved by relying on types such as function definition, handle, and code among them. At the same time, functions such as direct operation of data are also achieved by relying on these types.

[0057] 9. Mixed Category MIXED: It includes ANY type and ANYDICTIONARY type. The ANY type refers to a data type for which the user does not need to declare a specific type (such as INT). There is no corresponding definition in the SQL standard, and this embodiment defines and supports it. This is a type that is very widely used in Internet of Things scenarios. For example, multiple sensors (such as temperature sensors) may generate multiple time series data with different specific types (such as representing temperature using integer, single-precision floating-point number, double-precision floating-point number, string, etc. types simultaneously). The ANYDICTIONARY type is a variable container data type. It is a container that contains any number of key-value pairs, where the value can be of any data type (i.e., ANY). In a dictionary, given a key, a value can be uniquely determined.

[0058] 10. Other Category OTHER: It includes the COMPLEX type. A complex number can be represented in the form of a + bi, where a and b are real numbers and i is the imaginary unit. The complex type is a basic data type commonly used in fields such as scientific computing. There is no corresponding definition in the SQL standard, and this embodiment defines and supports it.

[0059] 11. Decimal Category DECIMAL: It includes decimal numbers with various precisions, such as 32-bit, 64-bit, and 128-bit decimal numbers (DECIMAL32, DECIMAL64, DECIMAL128).

[0060] 12. Array Category ARRAY: It includes the ARRAY type. Appending square brackets after the above 11 basic data types forms an array of the corresponding data type. For example, INT[] represents an array composed of integer numbers.

[0061] It can be seen that this embodiment supports a detailed definition and design of data forms commonly used in big data analysis, numerical calculation, and machine learning scenarios such as vectors, data pairs, matrices, sets, dictionaries, tables, and tensors, and defines many practical subtypes, while traditional SQL language lacks means to handle these data types.

[0062] Based on the above embodiments, the data processing system based on the Dlang language provided by the present application supports rich data types and a variety of different programming paradigms. Users only need to use the Dlang language to complete various tasks including SQL queries, data reading and writing, functional programming, object-oriented programming, etc. in the same script, and can achieve their data analysis tasks in one stop, without the need to use multiple different languages respectively, without the need to program in high-level languages such as C / C++ or Python, with low development, use and operation and maintenance difficulties and low learning costs. By mixing programming paradigms of functional programming and object-oriented programming, users can more conveniently implement very complex analysis tasks. At the same time, because the present application supports data types necessary for machine learning tasks including vectors, tensors, etc., and also supports functional programming and object-oriented programming, users can write script programs using the Dlang language to process data analysis and machine learning tasks simultaneously.

[0063] It should be noted that each module in the above embodiments can be a functional module or a program module, and can be implemented either by software or by hardware. For the modules implemented by hardware, the above-mentioned modules can be located in the same processor; or the above-mentioned modules can also be located in different processors in any combined form.

[0064] This embodiment also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0065] Optionally, the above electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0066] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be repeated here.

[0067] In addition, in combination with the data processing system based on the Dlang language in the above embodiments, the embodiments of the present application can be implemented by providing a storage medium. A computer program is stored on the storage medium; when the computer program is executed by a processor, any one of the data processing systems based on the Dlang language in the above embodiments is implemented.

[0068] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a data processing system based on the Dlang language. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0069] In one embodiment, Figure 5 is a schematic internal structure diagram of an electronic device according to an embodiment of the present application, as Figure 5 shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as Figure 5 shown. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected through an internal bus. Among them, the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities. The network interface is used to communicate with an external terminal through a network connection. The internal memory is used to provide an environment for the operation of the operating system and the computer program. When the computer program is executed by the processor, it implements a data processing system based on the Dlang language. The database is used to store data.

[0070] Those skilled in the art can understand that Figure 5 the structure shown in

[0071] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. This computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0072] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0073] The above embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A data processing system based on Dlang language, characterized in that: The system includes a lexical analysis module, a syntactic analysis module and a semantic analysis module; The lexical analysis module is used to pre-process the Dlang sentence input by the user and convert the Dlang sentence into a corresponding character string; The lexical analysis module is further used to segment the string to obtain a plurality of token sequences of the Dlang statement, wherein the token sequence is recorded as a tuple <token type, token value>, and the token type includes a keyword, an identifier, an operator, a literal value, and a punctuation mark; The syntax analysis module is used to generate a corresponding abstract syntax tree according to the tag sequence of the Dlang statement; The semantic analysis module is used to verify the abstract syntax tree and generate an intermediate code for the Dlang statement, wherein the intermediate code is used to compile an executable statement to process the data in the preset database.

2. The system according to claim 1, characterized in that In the case where the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a tuple <keyword, token value>, and the keyword is a keyword of functional programming: The syntax analysis module is used to parse the tag sequence of the functional programming to obtain an abstract syntax tree of the functional programming; The semantic analysis module is used to verify the abstract syntax tree of the functional programming and generate the intermediate code of the Dlang statement.

3. The system according to claim 1, characterized in that In the case where the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a tuple <keyword, token value>, and the keyword is a metaprogramming keyword: The syntax analysis module is used to parse the tag sequence of the metaprogramming to obtain the abstract syntax tree of the metaprogramming; The semantic analysis module is used to verify the abstract syntax tree of the metaprogramming and generate the intermediate code of the Dlang statement.

4. The system according to claim 1, characterized in that In the case where the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a tuple <keyword, token value>, and the keyword is a keyword of object-oriented programming: The syntax analysis module is used to parse the tag sequence of the object-oriented programming to obtain an abstract syntax tree of the object-oriented programming; The semantic analysis module is used to verify the abstract syntax tree of the object-oriented programming and generate the intermediate code of the Dlang statement.

5. The system according to claim 1, characterized in that In the case where the token sequence of the Dlang statement obtained by the lexical analysis module is recorded as a tuple <keyword, token value>, and the keyword is a keyword of a conditional branch statement or a loop body: The syntax analysis module is used to parse the tag sequence of the conditional branch statement or loop body to obtain a corresponding abstract syntax tree; The semantic analysis module is used to verify the abstract syntax tree and generate the intermediate code of the Dlang statement, wherein the conditional branch statement is an if-else conditional branch statement, and the loop body includes a for loop body and a do-while loop body.

6. The system according to claim 1, characterized in that The system also includes an optimization module; The optimization module is used to perform constant folding optimization on the intermediate code of the Dlang statement to obtain optimized intermediate code.

7. The system according to claim 1, characterized in that The system also includes a database execution module; The database execution module is used to run the executable statement to process the data in the preset database and return the data processing result to the user, wherein the preset database is the DolphinDB database.

8. The system according to claim 7, characterized in that The database execution module includes a first database execution module and a second database execution module; The first database execution module runs a first executable statement to create a first stream data table, and the second database execution module runs a second executable statement to create a second stream data table; The first database execution module or the second database execution module runs a third executable statement to perform real-time calculation of streaming data between the preset database, the first stream data table, and the second stream data table.

9. The system according to claim 1, characterized in that The compiled executable statements support processing of data of various data types under twelve categories, wherein the twelve categories include VOID category, logical category LOGICAL, time category TEMPORAL, floating point category FLOATING, literal category LITERAL, binary category BINARY, system category SYSTEM, mixed category MIXED, other category OTHER, decimal category DECIMAL, and array category ARRAY.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the system according to any one of claims 1 to 9 is implemented.

Images