Data query method based on Go language and related device
By loading data into memory in Go data query method and converting pseudo-SQL statements using Go parser, the problem of traditional database query delay and complexity is solved, and efficient and concise data query is achieved, which is suitable for high-concurrency large-scale data processing.
Patent Information
- Application Number
- CN202411938854.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-04-11
AI Technical Summary
Traditional database queries can easily lead to data access delays, and complex query logic reduces the maintainability of the program.
Using Go-based data query method, data query is realized by loading the data from the data source to be queried into a pre-determined data structure in memory, and using a parser written in Go-based language to convert pseudo-SQL statements into executable SQL statements.
It effectively reduces the access delay to data sources during query, reduces the complexity of data query logic, simplifies code, improves the maintainability and scalability of the system, and is suitable for scenarios of high concurrency and large-scale data processing.
Smart Images
Figure CN119807238A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data query method and related devices based on Go language. Background Art
[0002] In modern applications, the efficiency of data query and processing has a significant impact on the overall performance. Currently, traditional database queries often rely on network communications, resulting in data access delays. At the same time, complex query logic often requires more code to implement, which reduces the maintainability of the program. Therefore, in order to increase query speed and reduce code complexity, a new method is urgently needed to optimize the data query and update process. Summary of the invention
[0003] In view of the technical problems existing in the prior art, the present invention provides a data query method and related devices based on the Go language to solve the technical problems that traditional database queries are prone to cause data access delays and complex query logic reduces the maintainability of the program.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is: The present invention provides a data query method based on Go language, comprising: Loading data from the data source to be queried into a predetermined data structure in memory; Create pseudo SQL statements for data query based on pseudo SQL syntax rules; Using a preset parser, convert the pseudo SQL statement for data query into an executable SQL statement; wherein the preset parser is a parser written based on the Go language; The executable SQL statement is executed, data is read from a predetermined data structure in the memory, and a data query result is obtained.
[0005] Furthermore, the data in the data source to be queried is loaded into a predetermined data structure in the memory, as follows: Use connection pool technology to obtain data from the data source to be queried, and load the obtained data into the cache structure in the memory.
[0006] Furthermore, the cache structure of the memory is a cache dictionary.
[0007] Furthermore, after the data in the data source to be queried is loaded into a predetermined data structure in the memory, a memory data updating step is also included; The memory data updating steps are as follows: Monitor changes in the data source to be queried; According to the monitored changes of the data source to be queried, the changes of the data source to be queried are updated to the predetermined data structure in the memory based on the event-driven mechanism.
[0008] Furthermore, the parser written based on the Go language is a text / scanner package in the Go language or a parsing tool customized using the Go language.
[0009] Furthermore, the executable SOL statement includes data query conditions and sorting requirement information.
[0010] The present invention also provides a data query system based on the Go language, characterized in that it includes: A data loading module, used to load the data in the data source to be queried into a predetermined data structure in the memory; A statement creation module is used to create pseudo SQL statements for data query based on pseudo SQL grammar rules; A statement conversion module, used to convert the pseudo SQL statement for data query into an executable SQL statement using a preset parser; wherein the preset parser is a parser written in Go language; The query execution module is used to execute the executable SQL statement, read data from a predetermined data structure in the memory, and obtain data query results.
[0011] The present invention also provides a data query device based on the Go language, comprising: a processor suitable for executing a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the Go language-based data query method is executed.
[0012] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the data query method based on the Go language is implemented.
[0013] The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the data query method based on the Go language is implemented.
[0014] Compared with the prior art, the present invention has the following beneficial effects: The Go language-based data query method provided by the present invention preloads the data in the data source to be queried into a predetermined data structure in the memory, effectively reducing the access delay to the data source during the query; a parser written in the Go language is used to convert pseudo SQL statements into executable SQL statements, reducing the complexity of the data query logic and making the code more concise; at the same time, it satisfies the developer's need to expand or modify the parser according to the query requirements to adapt to the ever-changing data query requirements or integrate new data query features, thereby improving the maintainability and scalability of the system; in the data novelty search method described in the present invention, the query process is performed in the memory, avoiding frequent disk I / O operations, thereby reducing the consumption of system resources; at the same time, it can improve the query speed and reduce the complexity of the code, and is particularly suitable for scenarios with high concurrency and large-scale data processing, and can effectively improve the response speed of the program and the user experience.
[0015] The Go language-based data query system, Go language-based data query device, computer-readable storage medium and computer program product provided by the present invention have all the advantages of the above-mentioned Go language-based data query method. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A flowchart of a data query method based on the Go language provided in Example 1; Figure 2 A structural block diagram of a data query system based on the Go language provided in Example 2; Figure 3 This is a structural block diagram of a data query device based on the Go language provided in Example 3. DETAILED DESCRIPTION
[0017] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail in the following specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0018] Example 1 As attached Figure 1 As shown, this embodiment 1 provides a data query method based on the Go language, comprising the following steps: Step 1: Load the data in the data source to be queried into a predetermined data structure in the memory.
[0019] Specifically, a connection pool technology is used to obtain data from a data source to be queried, and the obtained data is loaded into a cache structure of a memory; wherein the memory cache structure is a cache dictionary; by loading the data in the data source to be queried into a predetermined data structure of the memory, frequent IO operations on the disk can be avoided, and access delays to the data source during querying can be effectively reduced; secondly, the obtained data is loaded into the cache dictionary of the memory for fast query and update, thereby ensuring high efficiency of data operations.
[0020] Example description: Using the connection pool technology, obtain data from the data source to be queried, and load the obtained data into the cache structure in the memory. The implementation steps are as follows: Configure the connection and connection pool of the data source to be queried; among them, create a database through the create_engine function to provide a basis for connecting and operating with the data source to be queried; use the create_engine function to configure the size of the connection pool and the maximum number of overflow connections through the pool_size and max_overflow parameters; query data from the data source to be queried through the load_data_into_cache function and store it in the cache.
[0021] Step 2: Monitor the changes of the data source to be queried; based on the monitored changes of the data source to be queried, update the changes of the data source to be queried to the predetermined data structure in the memory based on the event-driven mechanism. Among them, through the event-driven mechanism, monitor the changes of the data source and update the data in the memory in real time; only update the changed data to avoid repeated loading of the full amount of data and improve performance.
[0022] Specifically, the changes of the data source to be queried are monitored; and according to the monitored changes of the data source to be queried, the changes of the data source to be queried are updated to the predetermined data structure in the memory based on the event-driven mechanism. The implementation process is as follows: Monitor change events of the data source to be queried; if the data source to be queried is a database, use the database's trigger, notification mechanism or polling measurement to monitor the database's insert, update or delete operations; if the data source to be queried is an API, monitor changes by subscribing to change notifications provided by the API or polling the API endpoint; if the data source to be queried is a file, use the file system's monitoring function to monitor changes.
[0023] When a change of the data source to be queried is monitored, the change of the data source to be queried is updated into a predetermined data structure in the memory according to the monitored change of the data source to be queried.
[0024] Step 3: Create a pseudo SQL statement for data query based on the pseudo SQL grammar rules, wherein the pseudo SQL statement for data query can support a predetermined query operation, and the predetermined query operation includes SELECT, WHERE or ORDER BY.
[0025] It should be noted that the pseudo-SQL syntax rule refers to a simplified and easy-to-understand SQL syntax representation designed for certain purposes, such as teaching, demonstration, and document writing; in pseudo-SQL statements, keywords such as SELECT, FROM, and WHERE usually use uppercase letters to improve readability; non-keywords such as table names, column names, and aliases use lowercase letters, or mixed case letters to conform to actual naming conventions; pseudo-SQL statements usually follow the standard SQL statement structure, including SELECT, FROM, WHERE, and ORDER BY clauses; the order and grammatical structure of clauses are usually similar to standard SQL, but some parts will be simplified or omitted as needed; in pseudo-SQL, it is usually not necessary to explicitly declare data types, but data type representations similar to standard SQL are used in scenarios where data types need to be explained; pseudo-SQL contains comments and explanatory text to explain the purpose of the statement or provide additional contextual information; in pseudo-SQL, placeholders are used to represent query parameters to demonstrate the concept of parameterized queries.
[0026] Step 4: Use a preset parser to convert the pseudo SQL statement for data query into an executable SQL statement; wherein the preset parser is a parser written in Go language. The parser written in Go language is a text / scanner package in Go language or a parsing tool customized in Go language; the executable SQL statement includes data query conditions and sorting requirement information.
[0027] It should be noted that the use of a parser written in Go language can convert pseudo-SQL statements into query operations of memory data structures and perform efficient memory queries; specifically, the built-in string processing function and regular expression library of Go language are used to parse pseudo-SQL statements.
[0028] It should also be noted that in the process of converting the pseudo SQL statement for data query into an executable SQL statement using a preset parser, a query execution plan is also generated; wherein the query execution plan is used to optimize the query speed of the data storage method and reduce unnecessary calculations.
[0029] Step 5: Execute the executable SQL statement, read data from a predetermined data structure in the memory, and obtain a data query result.
[0030] Specifically, a linear query operation is performed in a predetermined data structure in memory according to an executable SQL statement; for example, if the query contains a WHERE clause, data needs to be filtered according to the conditions in the WHERE clause; then, the query result is constructed into a preset data structure to be returned to the user; preferably, the query result is a slice or a map, which contains the values of the fields of the elements that meet the query conditions; when returning the query result to the user or caller, it is achieved by returning a value, a pointer or a reference to pass the result.
[0031] The Go language-based data query described in this embodiment 1 uses a parser written in the Go language to convert pseudo-SQL statements into executable SQL statements, which reduces the complexity of the data query logic and makes the code more concise; at the same time, it satisfies the developer's need to expand or modify the parser according to the query requirements to adapt to the ever-changing data query requirements or integrate new data query features, thereby improving the maintainability and scalability of the system.
[0032] Example 2 As attached Figure 2 As shown, this embodiment 2 provides a data query system based on the Go language, including a data loading module, a data updating module, a statement creation module, a statement conversion module and a query execution module.
[0033] A data loading module is used to load data from a data source to be queried into a predetermined data structure in memory; a data updating module is used to monitor changes in the data source to be queried; based on the monitored changes in the data source to be queried, the changes in the data source to be queried are updated to the predetermined data structure in memory based on an event-driven mechanism; a statement creation module is used to create a pseudo-SQL statement for data query based on pseudo-SQL grammar rules; a statement conversion module is used to use a preset parser to convert the pseudo-SQL statement for data query into an executable SQL statement; wherein the preset parser adopts a parser written in Go language; a query execution module is used to execute the executable SQL statement, read data from the predetermined data structure in memory, and obtain data query results.
[0034] Example 3 As attached Figure 3 As shown, this embodiment 3 provides a data query device based on the Go language, including: a memory for storing a computer program; a processor for implementing the steps of a data query method based on the Go language when executing the computer program.
[0035] When the processor executes the computer program, the steps of the above-mentioned data query method based on the Go language are implemented, for example: The invention relates to a method for loading data in a to-be-queried data source into a predetermined data structure in a memory; monitoring changes in the to-be-queried data source; updating the changes in the to-be-queried data source to the predetermined data structure in the memory based on an event-driven mechanism according to the monitored changes in the to-be-queried data source; creating a pseudo-SQL statement for data query based on pseudo-SQL grammar rules; using a preset parser, converting the pseudo-SQL statement for data query into an executable SQL statement; wherein the preset parser adopts a parser written in Go language; executing the executable SQL statement, reading data from the predetermined data structure in the memory, and obtaining a data query result.
[0036] Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned Go language-based data query system are implemented, for example: A data loading module is used to load data from a data source to be queried into a predetermined data structure in memory; a data updating module is used to monitor changes in the data source to be queried; based on the monitored changes in the data source to be queried, the changes in the data source to be queried are updated to the predetermined data structure in memory based on an event-driven mechanism; a statement creation module is used to create a pseudo-SQL statement for data query based on pseudo-SQL grammar rules; a statement conversion module is used to use a preset parser to convert the pseudo-SQL statement for data query into an executable SQL statement; wherein the preset parser adopts a parser written in Go language; a query execution module is used to execute the executable SQL statement, read data from the predetermined data structure in memory, and obtain data query results.
[0037] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing preset functions, and the instruction segments are used to describe the execution process of the computer program in the Go language-based data query device.
[0038] For example, the computer program can be divided into a data loading module, a data updating module, a statement creating module, a statement conversion module and a query execution module, and the specific functions of each module are as follows: a data loading module, used to load the data in the data source to be queried into a predetermined data structure in the memory; a data updating module, used to monitor the changes of the data source to be queried; based on the monitored changes of the data source to be queried, the changes of the data source to be queried are updated to the predetermined data structure in the memory based on the event-driven mechanism; a statement creating module, used to create a pseudo-SQL statement for data query based on pseudo-SQL grammar rules; a statement conversion module, used to use a preset parser to convert the pseudo-SQL statement for data query into an executable SQL statement; wherein the preset parser adopts a parser written based on the Go language; a query execution module, used to execute the executable SQL statement, read data from the predetermined data structure in the memory, and obtain a data query result.
[0039] The data query device based on the Go language can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The data query device based on the Go language may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above is an example of a data query device based on the Go language, and does not constitute a limitation on the data query device based on the Go language, and may include more components than the above, or a combination of certain components, or different components, for example, the data query device based on the Go language may also include input and output devices, network access devices, buses, etc.
[0040] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the Go language-based data query device, and uses various interfaces and lines to connect the various parts of the entire Go language-based data query device.
[0041] The memory can be used to store the computer program and / or module, and the processor implements various functions of the Go language-based data query device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.
[0042] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0043] Example 4 This embodiment 4 further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data query method based on the Go language are implemented, for example: The invention relates to a method for loading data in a to-be-queried data source into a predetermined data structure in a memory; monitoring changes in the to-be-queried data source; updating the changes in the to-be-queried data source to the predetermined data structure in the memory based on an event-driven mechanism according to the monitored changes in the to-be-queried data source; creating a pseudo-SQL statement for data query based on pseudo-SQL grammar rules; using a preset parser, converting the pseudo-SQL statement for data query into an executable SQL statement; wherein the preset parser adopts a parser written in Go language; executing the executable SQL statement, reading data from the predetermined data structure in the memory, and obtaining a data query result.
[0044] If the module / unit integrated by the Go language-based data query system is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0045] Based on such understanding, the present invention implements all or part of the processes in the above-mentioned data query method based on Go language, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned data query method based on Go language when executed by a processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.
[0046] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.
[0047] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electrical carrier signals and telecommunication signals.
[0048] Example 5 This embodiment 5 provides a computer product, which computer program product includes a computer program, which is stored in a computer-readable storage medium; the processor of the Go language-based data query device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the Go language-based data query device can execute the Go language-based data query method described in Embodiment 1, which will not be repeated here.
[0049] It should be noted that a person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods.
[0050] The present invention achieves extremely low query latency by loading all data into memory; at the same time, the introduction of a pseudo-SQL query parser reduces the complexity of data query logic, makes the code more concise, and improves the maintainability and scalability of the system; the method is particularly suitable for scenarios with high concurrency and large-scale data processing, and can effectively improve the program's response speed and user experience.
[0051] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention.
Claims
1. A data query method based on Go language, characterized in that: include: Loading data from the data source to be queried into a predetermined data structure in memory; Create pseudo SQL statements for data query based on pseudo SQL syntax rules; Using a preset parser, convert the pseudo SQL statement for data query into an executable SQL statement; wherein the preset parser is a parser written based on the Go language; The executable SQL statement is executed, data is read from a predetermined data structure in the memory, and a data query result is obtained.
2. According to a data query method based on Go language according to claim 1, it is characterized in that: Load the data from the data source to be queried into a predetermined data structure in memory, as follows: Use connection pool technology to obtain data from the data source to be queried, and load the obtained data into the cache structure in the memory.
3. According to a data query method based on Go language according to claim 2, it is characterized in that: The cache structure of the memory is a cache dictionary.
4. According to a data query method based on Go language according to claim 1, it is characterized in that: After loading the data in the to-be-queried data source into a predetermined data structure in the memory, the step of updating the memory data is also included; The memory data updating steps are as follows: Monitor changes in the data source to be queried; According to the monitored changes of the data source to be queried, the changes of the data source to be queried are updated to the predetermined data structure in the memory based on the event-driven mechanism.
5. According to a data query method based on Go language according to claim 1, it is characterized in that: The parser written in Go language is the text / scanner package in Go language or a parsing tool customized in Go language.
6. A data query method based on Go language according to claim 1, characterized in that: The executable SOL statement includes data query conditions and sorting requirement information.
7. A data query system based on Go language, characterized in that: include: A data loading module, used to load the data in the data source to be queried into a predetermined data structure in the memory; A statement creation module is used to create pseudo SQL statements for data query based on pseudo SQL grammar rules; A statement conversion module, used to convert the pseudo SQL statement for data query into an executable SQL statement using a preset parser; wherein the preset parser is a parser written in Go language; The query execution module is used to execute the executable SQL statement, read data from a predetermined data structure in the memory, and obtain data query results.
8. A data query device based on Go language, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the Go language-based data query method as described in any one of claims 1 to 6 is executed.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the Go language-based data query method as described in any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the Go language-based data query method according to any one of claims 1 to 6 is implemented.