Structured data processing method and device based on cloud mode, medium and equipment

Abstract

The invention provides a structured data processing method and device based on a cloud mode, a medium and equipment. The method comprises the following steps: receiving structured data sent by equipment, and storing the structured data in a database; receiving a data query request sent by a user; analyzing the data query request to obtain query parameters; selecting an SQL statement template corresponding to the data query request from a template library; wherein SQL (Structured Query Language) statement templates in a plurality of business scenes are stored in the template library; the SQL statement template is filled with the query parameters, and SQL statements are obtained; and executing the SQL statement to query the required structured data from the database, and feeding back a query result to the user. According to the method, the distributed computing capability of cloud computing and the flexibility and the real-time performance of the end equipment are combined, the processing efficiency of the structured data can be improved, the cost is reduced, and more convenient and safer data processing services are provided for users.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cloud-based structured data processing method and device, medium, and equipment. Background Art

[0002] With the rapid development of information technology, the demand for processing structured data (e.g., table data in databases, log data, etc.) is growing. Traditional data processing methods mainly rely on local servers or single computing nodes, which have performance bottlenecks and high costs when processing large-scale data. Summary of the invention

[0003] In response to at least one of the above technical problems, an embodiment of the present invention provides a structured data processing method and apparatus, medium, and equipment based on a cloud mode.

[0004] According to the first aspect, the structured data processing method based on the cloud mode provided by the embodiment of the present invention is implemented by a distributed cloud computing system, and the method includes:

[0005] receiving structured data sent by a terminal device and storing the structured data in a database;

[0006] Receive data query requests sent by users;

[0007] Parsing the data query request to obtain query parameters;

[0008] Selecting an SQL statement template corresponding to the data query request from a template library; wherein the template library stores SQL statement templates for various business scenarios;

[0009] Fill the query parameters into the SQL statement template to obtain an SQL statement;

[0010] The SQL statement is executed to query the required structured data from the database, and the query result is fed back to the user.

[0011] In one embodiment, the terminal device is used to: collect structured data according to business needs, and preprocess the collected structured data; wherein the preprocessing includes filtering out invalid information and redundant information in the structured data through preset rules and identifying and correcting abnormal values ​​in the structured data.

[0012] In one embodiment, the method further comprises:

[0013] Analyze the execution plan of the SQL statement;

[0014] Generate an execution plan report based on the analysis results;

[0015] Determine bottleneck points in the execution plan report;

[0016] According to the bottleneck point, the SQL statement is structurally adjusted to obtain an optimized SQL statement, wherein the template of the optimized SQL statement is an optimized SQL statement template;

[0017] The optimized SQL statement template is added to the template library to replace the original SQL statement template.

[0018] In one embodiment, analyzing the execution plan of the SQL statement includes:

[0019] A query optimizer or an execution plan analysis tool is called to enable the query optimizer or the execution plan analysis tool to analyze the execution plan of the SQL statement.

[0020] In one embodiment, the execution plan report includes at least one of an estimated execution cost, index usage, and a data access path.

[0021] In one embodiment, the structural adjustment of the SQL statement includes:

[0022] The SQL statement is optimized for at least one of query order, adjustment of connection conditions, and selection of an optimal index according to data distribution characteristics.

[0023] In one embodiment, the method further comprises:

[0024] caching the query parameters and the structured data queried from the database;

[0025] Correspondingly, before selecting the SQL statement template corresponding to the data query request from the template library, the method further includes:

[0026] Determine whether the structured data corresponding to the query parameter in the current data query request exists in the cache;

[0027] If yes, returning the corresponding structured data in the cache to the user;

[0028] Otherwise, the SQL statement template corresponding to the data query request is selected from the template library and executed.

[0029] According to a second aspect, a structured data processing device based on a cloud mode provided by an embodiment of the present invention is deployed on a distributed cloud computing system, and the device includes:

[0030] A first receiving module, configured to receive structured data sent by a terminal device and store the structured data in a database;

[0031] The second receiving module is used to receive a data query request sent by a user;

[0032] A request parsing module, used to parse the data query request to obtain query parameters;

[0033] A template query module, used to select an SQL statement template corresponding to the data query request from a template library; wherein the template library stores SQL statement templates for various business scenarios;

[0034] A parameter filling module, used to fill the query parameters into the SQL statement template to obtain an SQL statement;

[0035] The statement execution module is used to execute the SQL statement to query the required structured data from the database and feed back the query result to the user.

[0036] In one embodiment, the terminal device is used to: collect structured data according to business needs, and preprocess the collected structured data; wherein the preprocessing includes filtering out invalid information and redundant information in the structured data through preset rules and identifying and correcting abnormal values ​​in the structured data.

[0037] In one embodiment, the device may further include:

[0038] An execution analysis module, used to analyze the execution plan of the SQL statement;

[0039] Report generation module, used to generate execution plan reports based on analysis results;

[0040] A bottleneck determination module, used to determine the bottleneck point in the execution plan report;

[0041] A template optimization module, used for adjusting the structure of the SQL statement according to the bottleneck point to obtain an optimized SQL statement, wherein the template of the optimized SQL statement is an optimized SQL statement template;

[0042] The template replacement module is used to add the optimized SQL statement template to the template library to replace the original SQL statement template.

[0043] In one embodiment, the execution analysis module is specifically used to: call a query optimizer or an execution plan analysis tool, so that the query optimizer or the execution plan analysis tool analyzes the execution plan of the SQL statement.

[0044] In one embodiment, the execution plan report includes at least one of an estimated execution cost, index usage, and a data access path.

[0045] In one embodiment, the template optimization module is specifically used to: optimize the query order of the SQL statement, adjust the connection conditions, and select at least one of the best indexes according to data distribution characteristics.

[0046] In one embodiment, the device may further include:

[0047] A data caching module, used for caching the query parameters and the structured data queried from the database;

[0048] Correspondingly, before selecting the SQL statement template corresponding to the data query request from the template library, the template query module is also used to: determine whether the structured data corresponding to the query parameters in this data query request exists in the cache; if so, return the corresponding structured data in the cache to the user; otherwise, execute the SQL statement template corresponding to the data query request selected from the template library.

[0049] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method provided in the first aspect.

[0050] According to a fourth aspect, an embodiment of the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.

[0051] The structured data processing method, device, medium, and equipment based on the cloud mode provided by the embodiment of the present invention, the terminal device collects structured data and sends the structured data to the distributed cloud computing system, so that the distributed cloud computing system will store the structured data collected by the terminal device in the database. When the distributed cloud computing system receives the data query request sent by the user, it parses the data query request to obtain the query parameters, selects the SQL statement template corresponding to the data query request from the template library, and then fills the query parameters into the SQL statement template to obtain the SQL statement, and then executes the SQL statement, thereby querying the required structured data in the database and feeding back the query results to the user. The terminal device has the characteristics of high flexibility and strong real-time performance, and can collect and preliminarily process data in real time. The distributed cloud computing system can perform distributed computing, and has powerful computing power, storage capacity, and data processing capacity. Therefore, the embodiment of the present invention combines cloud computing with terminal devices to achieve high efficiency, security, and real-time performance of data processing. It can be seen that the embodiments of the present invention combine the distributed computing capabilities of cloud computing with the flexibility and real-time performance of terminal devices. Compared with the existing technology that mainly relies on local servers or single nodes, the embodiments of the present invention can improve the processing efficiency of structured data, reduce costs, and provide users with more convenient and secure data processing services. It can not only significantly improve data query efficiency and save processing time, but also better support complex and changing business needs and promote business innovation and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of a structured data processing method based on a cloud mode in one embodiment of the present invention;

[0053] Figure 2 The following is a processing flow chart based on SpringBoot in one embodiment of the present invention.

[0054] Figure 3 It is a structural block diagram of a structured data processing device based on a cloud mode in one embodiment of the present invention. DETAILED DESCRIPTION

[0055] In a first aspect, an embodiment of the present invention provides a structured data processing method based on a cloud mode, wherein the method is implemented by a distributed cloud computing system, see Figure 1 The method comprises the following steps S110 to S160:

[0056] S110, receiving structured data sent by a terminal device, and storing the structured data in a database;

[0057] The terminal device can be used to collect structured data according to business needs and pre-process the collected structured data; wherein the pre-processing includes filtering out invalid information and redundant information in the structured data and identifying and correcting abnormal values ​​in the structured data through preset rules. After pre-processing the structured data, the terminal device sends the pre-processed structured data to the distributed cloud computing system, so that the structured data is stored in the database.

[0058] Among them, terminal devices include, for example, mobile phones, tablets, computers and other terminal devices.

[0059] S120, receiving a data query request sent by a user;

[0060] That is, when a user wants to query data, he sends a data query request to the distributed cloud computing system, so that the distributed cloud computing system receives the data query request.

[0061] S130, parsing the data query request to obtain query parameters;

[0062] For example, the data query request is parsed through parameter parsing technology to obtain the parameters in the data query request, which reflect the information such as what data the user wants to query, query conditions, query methods, etc. Among them, the query methods, such as single-table query, multi-table connection query, and aggregate query, are the business scenarios mentioned below.

[0063] S140, selecting an SQL statement template corresponding to the data query request from a template library; wherein the template library stores SQL statement templates for various business scenarios;

[0064] Among them, business scenarios include single-table query, multi-table join query, aggregate query, etc. There are different SQL statement templates in different scenarios.

[0065] S150, filling the query parameters into the SQL statement template to obtain an SQL statement;

[0066] For example, by using template engine technology, the parsed query parameters are added to the SQL statement template to obtain the SQL statement.

[0067] S160: Execute the SQL statement to query the required structured data from the database, and feed back the query result to the user.

[0068] That is, the SQL statement is executed to query the structured data corresponding to the query parameter in the database, and after the query result is obtained, the query result is fed back to the user, thereby completing the query task.

[0069] In one embodiment, the method may further include the following 5 steps:

[0070] 1. Analyze the execution plan of the SQL statement;

[0071] The analyzing the execution plan of the SQL statement may include: calling a query optimizer or an execution plan analysis tool, so that the query optimizer or the execution plan analysis tool analyzes the execution plan of the SQL statement.

[0072] 2. Generate an execution plan report based on the analysis results;

[0073] The execution plan report may include at least one of the estimated execution cost, index usage and data access path.

[0074] 3. Determine the bottleneck points in the execution plan report;

[0075] 4. According to the bottleneck point, the SQL statement is structurally adjusted to obtain an optimized SQL statement, wherein the template of the optimized SQL statement is an optimized SQL statement template;

[0076] The structural adjustment of the SQL statement may include: optimizing the query order of the SQL statement, adjusting the connection conditions, and selecting the best index according to the data distribution characteristics. By optimizing the SQL statement through at least one of the above adjustment methods, the cost of query execution can be significantly reduced and the query efficiency can be improved.

[0077] 5. Add the optimized SQL statement template to the template library to replace the original SQL statement template.

[0078] It can be seen that in order to evaluate the performance of the initial SQL statement, the execution plan of the SQL statement is analyzed, for example, the query optimizer built into the database or an external execution plan analysis tool is called, these tools use statistical information, cost estimation models and other technologies to conduct in-depth analysis of the execution plan of the SQL statement, and then generate an execution plan report based on the analysis results. The execution plan report may include information such as the estimated execution cost, index usage and data access path. Then, the bottleneck points in the execution plan report can be analyzed according to the optimization algorithm, and then the SQL statement can be optimized according to the bottleneck points, and the template of the optimized SQL statement can replace the original SQL statement template in the template library.

[0079] Among them, the optimization algorithm can be a rule-based optimization strategy, a cost-based optimization strategy, or an optimization strategy assisted by machine learning.

[0080] It is understandable that after optimizing an SQL query template, when subsequently using the SQL statement corresponding to the SQL query template to perform data query, the query efficiency can be improved.

[0081] In one embodiment, the method may further include: caching the query parameter and the structured data queried from the database;

[0082] Correspondingly, before selecting the SQL statement template corresponding to the data query request from the template library, the method also includes: determining whether the structured data corresponding to the query parameters in this data query request exists in the cache; if so, returning the corresponding structured data in the cache to the user; otherwise, executing the selection of the SQL statement template corresponding to the data query request from the template library.

[0083] That is, the query parameters and the structured data retrieved from the database are cached. In this way, when the query operation is performed next time, it is first determined whether there is corresponding structured data in the cache. If so, the corresponding structured data in the cache is directly returned to the user. If not, the SQL statement template corresponding to the data query request is selected from the template library, and then the subsequent steps are executed, so that the user obtains the query data.

[0084] It can be seen that the query caching technology can reduce the overhead of repeated queries.

[0085] The “cloud” in the embodiments of the present invention refers to a distributed cloud computing system, and the “end” refers to an end device.

[0086] Of course, in addition to query caching technology, you can also use database connection pool technology to reduce database connection overhead. At the same time, it also supports streaming or paging loading of query results to reduce memory pressure and improve user experience.

[0087] Of course, encryption technology, access control strategies and data desensitization technology can also be used to effectively protect the confidentiality and integrity of data and improve data security. At the same time, potential security threats can be discovered and handled in a timely manner through real-time monitoring and anomaly detection mechanisms.

[0088] In actual scenarios, the Spring Boot framework can be used to build a cloud server to achieve efficient and flexible data query services, that is, to improve data query efficiency.

[0089] See also Figure 2,The distributed cloud computing system first configures the database connection and the thread pool access port, and the SQL statement template in the template library. ,When receiving the URL request from the user, the control layer parses the query parameters from the request, and then passes the query parameters to the calling service layer. ,The service layer assembles the query parameters and the corresponding SQL statement template into a SQL statement by calling the mapping interface, and then executes the SQL statement and returns the queried data to the user.

[0090] In actual scenarios, a user-friendly data processing interface and interactive method can be provided, so that users can easily get started and quickly complete data query tasks. At the same time, real-time feedback and error handling mechanisms are provided to help users quickly solve problems and improve work efficiency, thereby enhancing user experience.

[0091] Among them, the terminal device has the characteristics of high flexibility and strong real-time performance, and can collect and preliminarily process data in real time. As a distributed computing technology, cloud computing provides computing resources, storage resources and information resources to users in an on-demand and easily scalable manner. Cloud computing has powerful computing power, storage capacity and data processing capabilities, and can perform in-depth analysis, mining and visualization of large-scale data. Therefore, combining cloud computing with terminal devices can achieve efficient, secure and real-time data processing.

[0092] It can be seen that the embodiments of the present invention combine the distributed computing power of cloud computing with the flexibility and real-time performance of terminal devices, which can improve the processing efficiency of structured data, reduce costs, and provide users with more convenient and secure data processing services. The structured data processing method based on the "cloud + terminal" model in the embodiments of the present invention can not only significantly improve data query efficiency and save processing time, but also better support complex and changing business needs and promote business innovation and development.

[0093] In a second aspect, an embodiment of the present invention provides a structured data processing device based on a cloud mode, wherein the device is deployed on a distributed cloud computing system, see Figure 3 , the device 100 comprises:

[0094] A first receiving module 110 is used to receive structured data sent by a terminal device and store the structured data in a database;

[0095] The second receiving module 120 is used to receive a data query request sent by a user;

[0096] A request parsing module 130, used to parse the data query request to obtain query parameters;

[0097] The template query module 140 is used to select the SQL statement template corresponding to the data query request from the template library; wherein the template library stores SQL statement templates for various business scenarios;

[0098] A parameter filling module 150 is used to fill the query parameters into the SQL statement template to obtain an SQL statement;

[0099] The statement execution module 160 is used to execute the SQL statement to query the required structured data from the database and feed back the query result to the user.

[0100] In one embodiment, the terminal device is used to: collect structured data according to business needs, and preprocess the collected structured data; wherein the preprocessing includes filtering out invalid information and redundant information in the structured data through preset rules and identifying and correcting abnormal values ​​in the structured data.

[0101] In one embodiment, the device may further include:

[0102] An execution analysis module, used to analyze the execution plan of the SQL statement;

[0103] Report generation module, used to generate execution plan reports based on analysis results;

[0104] A bottleneck determination module, used to determine the bottleneck point in the execution plan report;

[0105] A template optimization module, used for adjusting the structure of the SQL statement according to the bottleneck point to obtain an optimized SQL statement, wherein the template of the optimized SQL statement is an optimized SQL statement template;

[0106] The template replacement module is used to add the optimized SQL statement template to the template library to replace the original SQL statement template.

[0107] In one embodiment, the execution analysis module is specifically used to: call a query optimizer or an execution plan analysis tool, so that the query optimizer or the execution plan analysis tool analyzes the execution plan of the SQL statement.

[0108] In one embodiment, the execution plan report includes at least one of an estimated execution cost, index usage, and a data access path.

[0109] In one embodiment, the template optimization module is specifically used to: optimize the query order of the SQL statement, adjust the connection conditions, and select at least one of the best indexes according to data distribution characteristics.

[0110] In one embodiment, the device may further include:

[0111] A data caching module, used for caching the query parameters and the structured data queried from the database;

[0112] Correspondingly, before selecting the SQL statement template corresponding to the data query request from the template library, the template query module is also used to: determine whether the structured data corresponding to the query parameters in this data query request exists in the cache; if so, return the corresponding structured data in the cache to the user; otherwise, execute the SQL statement template corresponding to the data query request selected from the template library.

[0113] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0114] In a third aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0115] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0116] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0117] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0118] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0119] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0120] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0121] In a fourth aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any embodiment of the specification.

[0122] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0123] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0125] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A structured data processing method based on cloud mode, characterized in that: The method is implemented by a distributed cloud computing system, and the method includes: receiving structured data sent by a terminal device and storing the structured data in a database; Receive data query requests sent by users; Parsing the data query request to obtain query parameters; Selecting an SQL statement template corresponding to the data query request from a template library; wherein the template library stores SQL statement templates for various business scenarios; Fill the query parameters into the SQL statement template to obtain an SQL statement; The SQL statement is executed to query the required structured data from the database, and the query result is fed back to the user.

2. The method according to claim 1, characterized in that The terminal device is used to collect structured data according to business needs and pre-process the collected structured data; wherein the pre-processing includes filtering out invalid information and redundant information in the structured data through preset rules and identifying and correcting abnormal values ​​in the structured data.

3. The method according to claim 1, characterized in that Also includes: Analyze the execution plan of the SQL statement; Generate an execution plan report based on the analysis results; Determine bottleneck points in the execution plan report; According to the bottleneck point, the SQL statement is structurally adjusted to obtain an optimized SQL statement, wherein the template of the optimized SQL statement is an optimized SQL statement template; The optimized SQL statement template is added to the template library to replace the original SQL statement template.

4. The method according to claim 3, characterized in that The analyzing the execution plan of the SQL statement includes: A query optimizer or an execution plan analysis tool is called to enable the query optimizer or the execution plan analysis tool to analyze the execution plan of the SQL statement.

5. The method according to claim 3, characterized in that: The execution plan report includes at least one of the estimated execution cost, index usage and data access path.

6. The method according to claim 3, characterized in that The step of structurally adjusting the SQL statement includes: The SQL statement is optimized for at least one of query order, adjustment of connection conditions, and selection of an optimal index according to data distribution characteristics.

7. The method according to claim 1, characterized in that Also includes: caching the query parameters and the structured data queried from the database; Correspondingly, before selecting the SQL statement template corresponding to the data query request from the template library, the method further includes: Determine whether the structured data corresponding to the query parameter in the current data query request exists in the cache; If yes, returning the corresponding structured data in the cache to the user; Otherwise, the SQL statement template corresponding to the data query request is selected from the template library and executed.

8. A structured data processing device based on a cloud mode, characterized in that: The device is deployed on a distributed cloud computing system, and includes: A first receiving module, configured to receive structured data sent by a terminal device and store the structured data in a database; The second receiving module is used to receive a data query request sent by a user; A request parsing module, used to parse the data query request to obtain query parameters; A template query module, used to select an SQL statement template corresponding to the data query request from a template library; wherein the template library stores SQL statement templates for various business scenarios; A parameter filling module, used to fill the query parameters into the SQL statement template to obtain an SQL statement; The statement execution module is used to execute the SQL statement to query the required structured data from the database and feed back the query result to the user.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Images