Personalized language model with cross-computing capability

Abstract

The invention discloses a personalized language model with cross computing capability, and relates to the technical field of compilation computing of computer software systems. The problems that an existing mode is large in workload, low in accuracy and large in complexity are solved. Comprising a database, a cross computing module and a client, according to the method, the file data and the database data are calculated respectively, fusion calculation is carried out through a unified cross calculation engine, the obtained result is directly sent to the client, and the client can further carry out fusion calculation on the off-line data to finally obtain the desired data for rendering display.

Description

Technical Field

[0001] The present invention relates to the technical field of compiling and computing of computer software systems, and in particular to a personalized language model with cross computing capability. Background Art

[0002] With the continuous development of information technology, the scale of big data has also expanded rapidly. In addition to the data volume, the data form has also increased greatly, and the difficulty of data management has greatly increased. Traditional big data management is to uniformly process data in various data forms (such as semi-structured data such as xls, cvs, json, yaml, etc.) into a two-dimensional table form, import it into a relational database, and then perform unified calculations. This method has several disadvantages. First, it is necessary to map the original data with the database data, which increases the workload of mapping conversion, increases the risk of data inconsistency, and reduces the accuracy of the data. Secondly, for users, after the data is pre-processed and stored in the warehouse, the data form at the data consumption end is completely different from the original data form, which destroys the data lineage relationship and causes great difficulty in data traceability. In the actual work process, it greatly increases the complexity of data analysis and calculation.

[0003] In view of the above situation, a computing engine for native structured data is built, and computing engines of various data forms are integrated into a personalized language model with cross-computing capabilities to achieve flat data management, reduce data processing links, and thus ensure data accuracy. Summary of the invention

[0004] The purpose of the present invention is to solve the problems of large workload, low accuracy and high complexity in the existing methods, and to propose a personalized language model with cross-computing capability.

[0005] The object of the present invention can be achieved by the following technical scheme: a personalized language model with cross-computing capability, comprising a database, a cross-computing module and a client, wherein the cross-computing module is used to respectively calculate the file data and the database data, and then perform fusion calculation and output the corresponding results through a unified cross-computing engine, and directly send the results to the client for rendering and displaying the results; comprising a code editor, a code parser and a code executor;

[0006] The specific workflow of the code parser is as follows:

[0007] Word segmentation / lexical analysis: breaking a string of characters into meaningful code blocks;

[0008] Parsing / syntactic analysis: The stream of tokens is converted into an abstract syntax tree consisting of nested elements that represents the grammatical structure of the program;

[0009] Code generation: Converting an abstract syntax tree into executable code.

[0010] As a preferred embodiment of the present invention, the code editor is used to write business codes, highlight keywords, perform syntax color matching and syntax detection, and perform debugging operations.

[0011] As a preferred embodiment of the present invention, the specific workflow of the code executor is as follows: the code of the standard abstract syntax tree is calculated by the V8 engine, compiled into native machine code, and the performance is improved by inline caching.

[0012] As a preferred embodiment of the present invention, the database is also used to classify and save the file data that needs to be involved in the calculation, specifically: save according to business and file type, and establish a file directory, and the file path complies with the Linux directory specification.

[0013] As a preferred implementation of the present invention, the database is also used to manage file data according to business requirements, and the specific process is as follows:

[0014] Several files in the file data are numbered as i, the time when the files are cross-calculated is collected to obtain the cross-calculation time, and the number of cross-calculation times is counted to obtain the total number of cross-calculations; the cross-calculations are sorted in chronological order, the difference between two adjacent cross-calculations is calculated to obtain the calculation interval duration, all the calculation interval durations are averaged and the value is taken to obtain the average interval value; the calculation interval durations are sorted in order, and the two adjacent calculation interval durations are compared. If the calculation interval duration of the first sorted one is greater than the calculation interval duration of the second sorted one, the total number of previous intervals is increased by one; the total number of cross-calculations, the average interval value, and the total number of previous intervals are normalized to obtain the cross-calculation value JSi of the file, where es1, es2, and es3 are all preset weight factors;

[0015] Preset a number of business requirement keywords, each of which corresponds to a preset fixed requirement value. Match the corresponding content of the file with the business requirement keywords to obtain the corresponding business requirement keywords, and sum the fixed requirement values ​​corresponding to the matched business requirement keywords to obtain the total requirement value, which is marked as Xi.

[0016] Collect the total number of accesses and the storage duration corresponding to the file, extract the values ​​of the total number of accesses and the storage duration, and divide the value of the storage duration by the value of the total number of accesses to obtain the file access value Wi;

[0017] Normalize the settlement value, total demand value and document visit value and substitute the values ​​of the three into the preset file management model Output document management value Foi, where λ1, λ2 and λ3 are all preset weight factors;

[0018] The file path is set to include the first-level directory path, the second-level directory path, the third-level directory path, ...; each level of the directory path corresponds to a value range; the file's document management value is matched with several value ranges. If the document management value belongs to the corresponding value range, the directory path corresponding to the value range is marked as the storage path of the file, and the file is stored under the directory path.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. The present invention calculates the file data and database data separately, and performs fusion calculation through a unified cross-computing engine. The obtained results are directly sent to the client. The offline data can be further fused and calculated on the client to finally obtain the desired data for rendering and display.

[0021] 2. The code editor of the present invention is a tool for writing business codes, which is mainly responsible for highlighting keywords, syntax color matching and syntax detection, and comes with running debugging and other functions to assist users or developers in writing calculation codes; the code parser converts AST into executable code; it uses the V8 engine for calculation, and has good portability and cross-platform characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0023] Figure 1 It is a model principle block diagram of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Embodiment 1:

[0026] See also Figure 1 As shown, a personalized language model with cross-computing capability includes a database, a cross-computing module and a client. The database classifies and saves the file data that needs to be involved in the calculation, and can save them according to business and file types. The file path complies with the Linux directory specification, and Chinese characters are avoided as much as possible in the path to achieve the purpose of rapid positioning.

[0027] The cross-computing module calculates the file data and database data separately, and performs fusion calculations through a unified cross-computing engine. The results are sent directly to the client, where the offline data can be further fused and calculated to obtain the desired data for rendering and display.

[0028] The cross-calculation module uses JavaScript as the basic development language, and develops a custom DSL (domain-specific language) on this basis, integrating file data calculations into JavaScript (for example, the formula syntax in Excel files cannot be recognized in JavaScript, and Excel formulas need to be processed by DSL) for unified calculations. It is divided into three stages: code editing, code parsing, and code execution, which correspond to the three parts of the code editor, code parser, and code executor respectively.

[0029] The code editor is a tool used to write business codes. It is mainly responsible for keyword highlighting, syntax color matching and syntax detection, and comes with running debugging and other functions to assist users or developers in writing computing codes.

[0030] The code contains some Excel formula codes, such as:

[0031] '[ / excel / Declaration Form / Enterprise Income Tax Return Schedule 1.xlsx]A10000'! F2:J12;

[0032] This expression can be recognized as a running parameter and needs to be integrated with normal JavaScript syntax. Here you need to use the codemirror component for online editing, which supports syntax highlighting, code prompts, copy and paste, and other functions.

[0033] Code parser:

[0034] Word segmentation / lexical analysis: breaking a string of characters into meaningful code blocks;

[0035] Parsing / syntactic analysis: The stream of lexical units is converted into an AST (abstract syntax tree) consisting of nested elements that represents the grammatical structure of the program;

[0036] Code generation: convert AST into executable code;

[0037] The acorn component is used to output the consistency of the es specification, and the custom syntax structure is used to express the JavaScript AST. The generated syntax tree is as follows:

[0038] reportData('[ / excel / Declaration Form / Enterprise Income Tax Return Appendix 1.xlsx]A10000'!F2:J12).

[0039] Code executor: For codes that can generate standard AST, the V8 engine is used for calculation. The V8 engine is a JavaScript engine implementation. V8 is developed in C++. Before running JavaScript, compared with other JavaScript engines that convert it into bytecode or interpret it, V8 compiles it into native machine code (IA-32, x86-64, ARM, or MIPS CPUs), and uses methods such as inline caching to improve performance. With these features, JavaScript programs run at speeds comparable to binary programs under the V8 engine. V8 supports many operating systems, such as Windows, Linux, Android, etc., and also supports other hardware architectures, such as IA32, X64, ARM, etc., with good portability and cross-platform features.

[0040] Embodiment 2: Based on Embodiment 1, the database is also used to manage file data according to business needs. The specific process is as follows:

[0041] Several files in the file data are numbered as i, the time when the files are cross-calculated is collected to obtain the cross-calculation time, and the number of cross-calculation times is counted to obtain the total number of cross-calculations; the cross-calculations are sorted in chronological order, the difference between two adjacent cross-calculations is calculated to obtain the calculation interval duration, and the average value of all calculation interval durations is calculated and the value is taken to obtain the average interval value; the calculation interval durations are sorted in order, and the two adjacent calculation interval durations are compared. If the calculation interval duration of the first sort is greater than the calculation interval duration of the second sort, the total number of previous intervals is increased once; the total number of cross-calculations, the average interval value, and the total number of previous intervals are normalized and the values ​​of the three are taken, marked as JC1, JP2, and JZ3 respectively, and substituted into the formula JSi=(JC1×es1+JZ3×es3) / (JP2×es2+1) to obtain the cross-calculation value JSi of the file, where es1, es2, and es3 are preset weight factors, and their sizes are custom settings, such as 0.35, 0.24, and 0.41 respectively;

[0042] Preset a number of business requirement keywords, each of which corresponds to a preset fixed requirement value. Match the corresponding content of the file with the business requirement keywords to obtain the corresponding business requirement keywords, and sum the fixed requirement values ​​corresponding to the matched business requirement keywords to obtain the total requirement value, which is marked as Xi.

[0043] Collect the total number of accesses and the storage duration corresponding to the file, extract the values ​​of the total number of accesses and the storage duration, and divide the value of the storage duration by the value of the total number of accesses to obtain the file access value Wi;

[0044] Normalize the settlement value, total demand value and document visit value and substitute the values ​​of the three into the preset file management model Output document management value Foi, where λ1, λ2 and λ3 are preset weight factors, with values ​​of 1.6, 1.8 and 2.3 respectively;

[0045] The file path is set to include the first-level directory path, the second-level directory path, the third-level directory path, ...; each level of the directory path corresponds to a value range, and the first-level directory path corresponds to the largest value range, and the second-level directory path, the third-level directory path, ... The value ranges are reduced in turn; the document management value of the file is matched with several value ranges. If the document management value belongs to the corresponding value range, the directory path corresponding to the value range is marked as the storage path of the file, and the file is stored under the directory path; for example: if the document management value belongs to the value range corresponding to the first-level directory path, the file is stored under the first-level directory path. The document management value of the file is calculated by the settlement value, the total demand value and the document access value, and the file is stored and managed according to the document management value. It is convenient to query and store files according to business needs, and avoid unreasonable file storage paths, which cause long file query time.

[0046] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. Personalized language model with cross-computing capability, including database, cross-computing module and client, It is characterized in that The cross-calculation module is used to calculate the file data and the database data respectively, and then perform fusion calculations through a unified cross-calculation engine to output corresponding results, and directly send the results to the client for rendering and display; it includes a code editor, a code parser and a code executor; The specific workflow of the code parser is as follows: Word segmentation / lexical analysis: breaking a string of characters into meaningful code blocks; Parsing / syntactic analysis: The stream of tokens is converted into an abstract syntax tree consisting of nested elements that represents the grammatical structure of the program; Code generation: Converting an abstract syntax tree into executable code.

2. The personalized language model with cross-computing capability according to claim 1, It is characterized in that The code editor is used to write business codes, highlight keywords, perform syntax color matching and syntax detection, and also perform debugging operations.

3. The personalized language model with cross-computing capability according to claim 1, It is characterized in that The specific workflow of the code executor is as follows: the code of the standard abstract syntax tree is calculated by the V8 engine, compiled into native machine code, and the performance is improved through inline caching.

4. The personalized language model with cross-computing capability according to claim 1, It is characterized in that The database is also used to classify and save the file data that needs to be involved in the calculation, specifically: save according to business and file type, and establish a file directory, and the file path complies with the Linux directory specification.

Images