Data transmission method, device and equipment based on myhouse database, and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a data transmission method and device based on a Myhouse database, equipment and a storage medium, the method comprises the following steps: based on a preset syntax analysis algorithm in a preset analyzer, performing syntax analysis on a target service request to generate target data corresponding to the target service request; preconfiguring a target service end and writing hot spot data in the target data into the target service end; writing data exceeding a preset time period in the target data into a large database and storing the data into the large database, and providing a read-write interface through the large database to realize data insertion. After syntax analysis of the target service request, the hot spot data is directly written into the target service end within the preset time period, and the target data outside the preset time period is written into a distributed large database, a read-write interface is provided through the large database to realize data insertion, and the ability to cope with high concurrency in the data transmission process is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data transmission method, apparatus, device and storage medium based on the Myhouse database. Background Technology

[0002] Database technology can be broadly categorized into two main types: online transaction processing (OLTP) and online analytical processing (OLAP). OLTP primarily handles data addition, deletion, and modification operations in the database. Transactions are short-lived and must meet ACID (Atomicity, Consistency, Isolation, Durability) properties, requiring complex concurrency control mechanisms to ensure correct execution. OLAP, exemplified by multidimensional analysis in data warehouses employing multidimensional storage models (star or snowflake schemas), involves grouping and aggregation calculations through joins between fact tables and multiple dimension tables, representing a read-only, long-running query. Currently, new databases designed for enterprise development primarily address high concurrency and real-time query and statistical functions for large datasets. While many companies choose MySQL, it lacks support for large-scale OLAP, particularly large-scale statistical analysis. Therefore, improving the ability to handle high concurrency during data transmission has become a critical technical challenge. Summary of the Invention

[0003] This application provides a data transmission method, apparatus, device, and storage medium based on the Myhouse database to improve the ability to handle high concurrency during data transmission.

[0004] Firstly, this application provides a data transmission method based on a Myhouse database, the data transmission method based on a Myhouse database comprising:

[0005] Based on the preset syntax analysis algorithm in the preset analyzer, the target service request is parsed to generate the target data corresponding to the target service request.

[0006] The target server is pre-configured, and hot data from the target data is written to the target server, wherein the hot data is the target data within a preset time period;

[0007] Data exceeding the preset time period in the target data is written and stored in a preset large database, and read / write interfaces are provided through the large database to realize data insertion.

[0008] Further, before performing syntax parsing on the target service request based on the preset syntax analysis algorithm in the preset analyzer to generate the target data corresponding to the target service request, the process includes:

[0009] Based on Java Database Connectivity (JDBC), a connection is established between the target client and a preset connector in the Serve layer.

[0010] Furthermore, the target server is pre-configured, and hot data from the target data is written to the target server, wherein the hot data refers to data preceding the target data within a preset time period, including:

[0011] Based on each historical service request, determine the historical hotspot retention period corresponding to the hotspot data of each historical service request;

[0012] Based on the historical hotspot retention periods, the average hotspot retention period is calculated, and the average hotspot retention period is determined as the preset time period.

[0013] Further, based on the historical hotspot retention time periods, the average hotspot retention time period is calculated, and after determining the average hotspot retention time period as the preset time period, the process includes:

[0014] Based on the average hotspot retention period, the data writing process corresponding to the target service request is detected to determine whether the average hotspot retention period is sufficient to retain the hotspot data corresponding to the target service request.

[0015] If the average hotspot retention period is insufficient to retain the hotspot data, the average hotspot retention period is extended based on the hotspot data.

[0016] Furthermore, data exceeding the preset time period from the target data is written to and stored in a preset large database, and read / write interfaces are provided through the large database to achieve data insertion, including:

[0017] Based on the data length corresponding to data exceeding the preset time period, and using a preset formula to calculate the number of preset columnar storage engines in the large database, the preset formula is as follows:

[0018]

[0019] Where L is the data length corresponding to data exceeding the preset time period, S is the capacity of a single preset columnar storage engine, N is the number of preset columnar storage engines, and n is the number of redundancies not less than 1.

[0020] Furthermore, before performing syntax parsing on the target service request based on the preset syntax analysis algorithm in the preset analyzer to generate the target data corresponding to the target service request, the process further includes:

[0021] The target service request is matched in the target server.

[0022] If there is a matching service request in the target server that matches the target service request, the matching service request is transmitted to the big database, and the big database provides a read / write interface to insert the data.

[0023] Secondly, this application also provides a data transmission device based on the Myhouse database, the device comprising:

[0024] The syntax analysis module is used to perform syntax parsing on the target service request based on the preset syntax analysis algorithm in the preset analyzer, and generate target data corresponding to the target service request.

[0025] The hot data writing module is used to pre-configure the target server and write the hot data in the target data to the target server, wherein the hot data is the target data within a preset time period;

[0026] A distributed storage module is used to write and store data from the target data that exceeds the preset time period into a preset large database, and to provide a read / write interface through the large database to realize data insertion.

[0027] Thirdly, this application also provides an apparatus comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the data transmission method based on the Myhouse database as described above.

[0028] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the data transmission method based on the Myhouse database as described above.

[0029] This application discloses a data transmission method, apparatus, device, and storage medium based on a Myhouse database. The data transmission method based on the Myhouse database includes: parsing a target service request using a preset syntax analysis algorithm in a preset analyzer to generate target data corresponding to the target service request; pre-configuring the target server and writing hot data from the target data into the target server, wherein the hot data is the target data within a preset time period; writing and storing data from the target data outside the preset time period into a preset large database, and providing read / write interfaces through the large database to achieve data insertion. This application improves the ability to handle high concurrency during data transmission by performing syntax analysis on the target service request, directly writing hot data (data within the preset time period) to the target server, writing target data outside the preset time period to a distributed large database, and providing read / write interfaces through the large database to achieve data insertion. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic flowchart illustrating a data transmission method based on the Myhouse database, provided for the first embodiment of this application;

[0032] Figure 2 A schematic flowchart illustrating a data transmission method based on the Myhouse database, provided for a second embodiment of this application;

[0033] Figure 3 A schematic diagram illustrating a data transmission method based on the Myhouse database provided in this application;

[0034] Figure 4 A schematic block diagram of a data transmission device based on a Myhouse database provided for embodiments of this application;

[0035] Figure 5 A schematic block diagram of the structure of a device provided for an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0038] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0039] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0040] This application provides a data transmission method, apparatus, device, and storage medium based on a Myhouse database. The Myhouse database-based data transmission method can be applied to a server. After parsing the target service request, hot data within a preset time period is directly written to the target server, while target data outside the preset time period is written to a distributed large database. The large database provides read / write interfaces to achieve data insertion, improving the ability to handle high concurrency during data transmission. The server can be a standalone server or a server cluster.

[0041] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0042] Please see Figure 1 , Figure 1This is a schematic flowchart illustrating a data transmission method based on a Myhouse database, provided in the first embodiment of this application. This data transmission method based on a Myhouse database can be applied to a server. It involves performing syntactic analysis on the target service request, directly writing hot data (data within a preset time period) to the target server, and writing target data outside the preset time period to a distributed large database. The large database provides read / write interfaces to achieve data insertion, thus improving the ability to handle high concurrency during data transmission.

[0043] like Figure 1 As shown, the data transmission method based on the Myhouse database specifically includes steps S10 to S30.

[0044] Step S10: Based on the preset syntax analysis algorithm in the preset analyzer, perform syntax parsing on the target service request to generate target data corresponding to the target service request;

[0045] Step S20: Pre-configure the target server and write the hot data in the target data into the target server, wherein the hot data is the target data within a preset time period;

[0046] Step S30: Write and store the data in the target data that exceeds the preset time period into a preset large database, and provide a read / write interface through the large database to realize data insertion.

[0047] In this specific embodiment, OLAP and OLTP are used to adapt to all service scenarios.

[0048] The current issues with MySQL are as follows:

[0049] 1. The `count(*)` function and its built-in functions consume a lot of CPU and have poor performance, often causing CPU overload.

[0050] 2. With large data volumes, scaling up increases hardware costs, typically doubling the cost of each expansion.

[0051] 3. High-concurrency read / write scenarios. Transfer efficiency is relatively low, especially for high-concurrency write performance.

[0052] 4. Adding fields to large datasets can easily impact business operations; even a simple modification to a field can potentially cause database crashes and business system failures.

[0053] 5. Unable to add fields or modify table structure in real time.

[0054] ClickHouse currently has the following issues:

[0055] 1. The database currently does not support transactions. This means there is a risk of data loss.

[0056] 2. This is caused by the columnar architecture used. Update and delete operations are particularly inefficient, consuming a large amount of CPU and memory.

[0057] 3. Inadequate index design. It's usually impossible to assess performance through indexes later.

[0058] 4. Unable to perform high-concurrency real-time updates and deletions.

[0059] 5. No complete transaction.

[0060] 6. Lacks the ability to frequently modify or delete existing data with low latency. It can only be used for batch deletion or modification of data.

[0061] 7. Sparse indexes make Clickhouse unsuitable for point queries that retrieve a single row by its key.

[0062] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the scenario of this application, where each module is shown below:

[0063] Myhouse-dashboard: A cluster management tool. It supports adding, deleting, and migrating data from myhouse-proxy when the cluster changes. Myhouse-dashboard maintains the consistency of myhouse-proxy's state, achieving high concurrency, high performance, and high availability.

[0064] Myhouse-proxy: Provides external database ports and services; one proxy is recognized as one service. Three services mean that even if two servers fail, myhouse-proxy can still function. Inserted database entries are automatically inserted into my-server based on the time.

[0065] My-server: Primarily used for high-concurrency databases. It only stores frequently accessed data and supports high concurrency. It uses the JDBC protocol and allows configuration of data retention time.

[0066] For example, if the data is retained for 6 months, then the database in myserver will report an error indicating that the last 6 months of data are not stored. Any additional data is stored in bigdata-house. myserver uses the same database storage algorithm and B-tree architecture as MySQL, thus supporting transactions and all JDBC protocols. Therefore, theoretically, it can be seamlessly integrated into the system. However, because the data volume is always the set amount (6 months), the data in the myserver layer is always limited.

[0067] Bigdata-house: A columnar storage engine. It retains data past its expiration date. It features built-in distributed storage and supports distributed scaling. It can handle hundreds of millions of data points with sub-second response times. Because the storage uses a columnar architecture, it is well-suited for data analysis and querying, making it highly efficient even for hundreds of millions of data points.

[0068] This embodiment discloses a data transmission method, apparatus, device, and storage medium based on a Myhouse database. The data transmission method based on the Myhouse database includes: parsing a target service request using a preset syntax analysis algorithm in a preset analyzer to generate target data corresponding to the target service request; pre-configuring the target server and writing hot data from the target data into the target server, wherein the hot data is the target data within a preset time period; writing and storing data from the target data outside the preset time period into a preset large database, and using the large database to provide read / write interfaces for data insertion. This application improves the ability to handle high concurrency during data transmission by performing syntax analysis on the target service request, directly writing hot data (data within the preset time period) to the target server, writing target data outside the preset time period to a distributed large database, and using the large database to provide read / write interfaces for data insertion.

[0069] based on Figure 1 In the illustrated embodiment, before step S10, the following steps are included:

[0070] Based on Java Database Connectivity (JDBC), a connection is established between the target client and a preset connector in the Serve layer.

[0071] Specifically, data compression is not used in some columnar database management systems (such as InfiniDB CE and MonetDB). However, data compression does play a crucial role in achieving superior performance.

[0072] In addition to efficient general-purpose compression codecs that make different trade-offs between disk space and CPU consumption, Myhouse also provides dedicated codecs for specific types of data, which allows Myhouse to compete with and surpass smaller databases such as time-series databases.

[0073] Many columnar databases (such as SAP HANA and Google PowerDrill) can only operate in memory, which results in a higher equipment budget than necessary. Myhouse is designed for systems that operate on traditional disks, offering lower storage costs per GB, but it also makes efficient use of SSDs and memory if available.

[0074] Myhouse utilizes all available resources on the server to process large queries in parallel in the most natural way through multi-core parallel processing.

[0075] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a data transmission method based on a Myhouse database, provided in the second embodiment of this application. This data transmission method based on a Myhouse database can be applied to a server. It involves performing syntactic analysis on the target service request, directly writing hot data (data within a preset time period) to the target server, and writing target data outside the preset time period to a distributed large database. The large database provides read / write interfaces to achieve data insertion, thus improving the ability to handle high concurrency during data transmission.

[0076] like Figure 2 As shown, steps S11 to S12 are included before step S20.

[0077] Step S11: Based on each historical service request, determine the historical hotspot retention time period corresponding to the hotspot data of each historical service request;

[0078] Step S12: Calculate the average hotspot retention period based on each of the historical hotspot retention periods, and determine the average hotspot retention period as the preset time period.

[0079] In specific embodiments, such as Figure 3 As shown, Figure 3 The following is a schematic diagram illustrating a data transmission method based on the Myhouse database provided in this application, wherein the modules are shown below:

[0080] Myhouse-dashboard: A cluster management tool. It supports adding, deleting, and migrating data from myhouse-proxy when the cluster changes. Myhouse-dashboard maintains the consistency of myhouse-proxy's state, achieving high concurrency, high performance, and high availability.

[0081] Myhouse-proxy: Provides external database ports and services; one proxy is recognized as one service. Three services mean that even if two servers fail, myhouse-proxy can still function. Inserted database entries are automatically inserted into my-server based on the time.

[0082] My-server: Primarily used for high-concurrency databases. It only stores frequently accessed data and supports high concurrency. It uses the JDBC protocol and allows configuration of data retention time.

[0083] For example, if the data is retained for 6 months, then the database in myserver will report an error indicating that the last 6 months of data are not stored. Any additional data is stored in bigdata-house. myserver uses the same database storage algorithm and B-tree architecture as MySQL, thus supporting transactions and all JDBC protocols. Therefore, theoretically, it can be seamlessly integrated into the system. However, because the data volume is always the set amount (6 months), the data in the myserver layer is always limited.

[0084] Bigdata-house: A columnar storage engine. It retains data past its expiration date. It features built-in distributed storage and supports distributed scaling. It can handle hundreds of millions of data points with sub-second response times. Because the storage uses a columnar architecture, it is well-suited for data analysis and querying, making it highly efficient even for hundreds of millions of data points.

[0085] This embodiment discloses a data transmission method, apparatus, device, and storage medium based on a Myhouse database. The data transmission method based on the Myhouse database includes determining historical hotspot retention time periods corresponding to hotspot data in each historical service request; calculating an average hotspot retention time period based on each historical hotspot retention time period; and determining the average hotspot retention time period as a preset time period. This application, by performing syntax analysis on the target service request, directly writes the hotspot data within the preset time period to the target server, and writes the target data outside the preset time period to a distributed large database. The large database provides read / write interfaces to achieve data insertion, improving the ability to handle high concurrency during data transmission.

[0086] based on Figure 2 In this embodiment, after step S12, the following is included:

[0087] Based on the average hotspot retention period, the data writing process corresponding to the target service request is detected to determine whether the average hotspot retention period is sufficient to retain the hotspot data corresponding to the target service request.

[0088] If the average hotspot retention period is insufficient to retain the hotspot data, the average hotspot retention period is extended based on the hotspot data.

[0089] Based on the above embodiments, in this embodiment, step S30 includes:

[0090] Based on the data length corresponding to data exceeding the preset time period, and using a preset formula to calculate the number of preset columnar storage engines in the large database, the preset formula is as follows:

[0091]

[0092] Where L is the data length corresponding to data exceeding the preset time period, S is the capacity of a single preset columnar storage engine, N is the number of preset columnar storage engines, and n is the number of redundancies not less than 1.

[0093] Specifically, to enable queries to quickly perform range searches within the primary key, data is always stored incrementally and ordered in the MergeTree. Therefore, data can be continuously and efficiently written to the table without any locking behavior during the write process. This is suitable for online queries, meaning queries are processed with extremely low latency and the results are loaded onto the user's page without any preprocessing of the data.

[0094] In Myhouse, data can be stored on different shards, and each shard consists of a set of replicas for fault tolerance. Queries can be processed in parallel across all shards. This is transparent to the user.

[0095] Myhouse supports a SQL-based declarative query language that is identical to the ANSI SQL standard in many cases.

[0096] Supported queries include GROUP BY, ORDER BY, FROM, JOIN, IN, and non-correlated subqueries.

[0097] Related (dependent) subqueries and window functions are not currently supported, but will be implemented in the future.

[0098] Based on all the above embodiments, in this embodiment, before step S10, the method further includes:

[0099] The target service request is matched in the target server.

[0100] If there is a matching service request in the target server that matches the target service request, the matching service request is transmitted to the big database, and the big database provides a read / write interface to insert the data.

[0101] Please see Figure 4 , Figure 4 This application provides a schematic block diagram of a data transmission apparatus based on a Myhouse database, which is used to execute the aforementioned data transmission method based on a Myhouse database. The Myhouse database-based data transmission apparatus can be configured on a server.

[0102] like Figure 4 As shown, the data transmission device 400 based on the Myhouse database includes:

[0103] Syntax analysis module 10 is used to perform syntax parsing on the target service request based on the preset syntax analysis algorithm in the preset analyzer, and generate target data corresponding to the target service request;

[0104] Hotspot data writing module 20 is used to pre-configure the target server and write hotspot data from the target data to the target server, wherein the hotspot data is the target data within a preset time period;

[0105] The distributed storage module 30 is used to write and store data from the target data that exceeds the preset time period into a preset large database, and to provide a read / write interface through the large database to realize data insertion.

[0106] Furthermore, the data transmission device based on the Myhouse database also includes a JDBC module, specifically comprising:

[0107] The JDBC unit is used to establish a connection between the target client and the preset connector in the Serve layer based on Java Database Connectivity (JDBC).

[0108] Furthermore, the data transmission device based on the Myhouse database also includes a pre-configuration module, specifically comprising:

[0109] The historical hotspot retention period unit is used to determine the historical hotspot retention period corresponding to the hotspot data of each historical service request based on each historical service request.

[0110] A preset time period unit is used to calculate the average hotspot retention time period based on each of the historical hotspot retention time periods, and to determine the average hotspot retention time period as the preset time period.

[0111] Furthermore, the pre-configuration module also includes:

[0112] The hotspot data detection unit is used to detect the data writing process corresponding to the target service request based on the average hotspot retention period, so as to determine whether the average hotspot retention period is sufficient to retain the hotspot data corresponding to the target service request.

[0113] A hotspot data storage unit is used to extend the average hotspot storage time period based on the hotspot data if the average hotspot storage time period is insufficient to store the hotspot data.

[0114] Furthermore, the distributed storage module 30 also includes:

[0115] The engine quantity calculation unit is used to calculate the number of preset columnar storage engines in the large database based on the data length corresponding to data exceeding the preset time period and using a preset formula, as follows:

[0116]

[0117] Where L is the data length corresponding to data exceeding the preset time period, S is the capacity of a single preset columnar storage engine, N is the number of preset columnar storage engines, and n is the number of redundancies not less than 1.

[0118] Furthermore, the data transmission device based on the Myhouse database also includes a service request matching module, specifically comprising:

[0119] A matching unit is used to match the target service request in the target server.

[0120] The direct transmission unit is used to transmit the unmatched service request to the large database if there is an unmatched service request in the target server that matches the target service request, and to provide a read / write interface through the large database to realize data insertion.

[0121] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0122] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the device shown.

[0123] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a device according to an embodiment of this application. The device may be a server.

[0124] See Figure 5 The device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0125] The non-volatile storage medium can store the operating system and computer program. The computer program includes program instructions that, when executed, cause the processor to perform any data transfer method based on the Myhouse database.

[0126] The processor provides computing and control capabilities to support the operation of the entire device.

[0127] Internal memory provides an environment for the execution of computer programs on non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any data transfer method based on the Myhouse database.

[0128] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the device to which the present application is applied. Specific devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0129] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0130] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0131] Based on the preset syntax analysis algorithm in the preset analyzer, the target service request is parsed to generate the target data corresponding to the target service request.

[0132] The target server is pre-configured, and hot data from the target data is written to the target server, wherein the hot data is the target data within a preset time period;

[0133] Data exceeding the preset time period in the target data is written and stored in a preset large database, and read / write interfaces are provided through the large database to realize data insertion.

[0134] In one embodiment, before parsing the target service request based on a preset syntax analysis algorithm in a preset analyzer and generating the target data corresponding to the target service request, the following is performed:

[0135] Based on Java Database Connectivity (JDBC), a connection is established between the target client and a preset connector in the Serve layer.

[0136] In one embodiment, the target server is pre-configured, and hot data from the target data is written to the target server. The hot data refers to data preceding the target data within a preset time period, used to achieve the following:

[0137] Based on each historical service request, determine the historical hotspot retention period corresponding to the hotspot data of each historical service request;

[0138] Based on the historical hotspot retention periods, the average hotspot retention period is calculated, and the average hotspot retention period is determined as the preset time period.

[0139] In one embodiment, after calculating the average hotspot retention period based on each of the historical hotspot retention periods and determining the average hotspot retention period as the preset time period, it is used to achieve:

[0140] Based on the average hotspot retention period, the data writing process corresponding to the target service request is detected to determine whether the average hotspot retention period is sufficient to retain the hotspot data corresponding to the target service request.

[0141] If the average hotspot retention period is insufficient to retain the hotspot data, the average hotspot retention period is extended based on the hotspot data.

[0142] In one embodiment, data from the target data that exceeds the preset time period is written and stored in a preset large database, and the large database provides a read / write interface to implement data insertion, thereby achieving:

[0143] Based on the data length corresponding to data exceeding the preset time period, and using a preset formula to calculate the number of preset columnar storage engines in the large database, the preset formula is as follows:

[0144]

[0145] Where L is the data length corresponding to data exceeding the preset time period, S is the capacity of a single preset columnar storage engine, N is the number of preset columnar storage engines, and n is the number of redundancies not less than 1.

[0146] In one embodiment, before parsing the target service request based on a preset syntax analysis algorithm in a preset analyzer to generate the target data corresponding to the target service request, the following is also implemented:

[0147] The target service request is matched in the target server.

[0148] If there is a matching service request in the target server that matches the target service request, the matching service request is transmitted to the big database, and the big database provides a read / write interface to insert the data.

[0149] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the data transmission methods based on the Myhouse database provided in the embodiments of this application.

[0150] The computer-readable storage medium can be an internal storage unit of the device described in the foregoing embodiments, such as the hard disk or memory of the device. The computer-readable storage medium can also be an external storage device of the device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data transmission method based on the Myhouse database, characterized in that, The data transmission method based on the Myhouse database includes: Based on the preset syntax analysis algorithm in the preset analyzer, the target service request is parsed to generate the target data corresponding to the target service request. The target server is pre-configured, and hot data from the target data is written to the target server. The hot data is the target data within a preset time period. The preset time period is the average hot data retention period calculated based on the hot data retention period of historical service requests. Data exceeding the preset time period in the target data is written and stored in a preset large database, and the large database provides a read / write interface to realize data insertion. Based on the data length corresponding to data exceeding the preset time period, and using a preset formula to calculate the number of preset columnar storage engines in the large database, the preset formula is as follows: Where L is the data length corresponding to data exceeding the preset time period, S is the capacity of a single preset columnar storage engine, N is the number of preset columnar storage engines, and n is the number of redundancies not less than 1. The step of pre-configuring the target server and writing hot data from the target data into the target server, wherein the hot data refers to data preceding the target data within a preset time period, includes: Based on each historical service request, determine the historical hotspot retention period corresponding to the hotspot data of each historical service request; Based on the historical hotspot retention time periods, the average hotspot retention time period is calculated, and the average hotspot retention time period is determined as the preset time period; Based on the average hotspot retention period, the data writing process corresponding to the target service request is detected. To determine whether the average hotspot retention period is sufficient to retain the hotspot data corresponding to the target service request; If the average hotspot retention period is insufficient to retain the hotspot data, the average hotspot retention period shall be extended based on the hotspot data.

2. The data transmission method based on the Myhouse database according to claim 1, characterized in that, Before the preset syntax analysis algorithm in the preset analyzer performs syntax parsing on the target service request and generates the target data corresponding to the target service request, the process includes: Based on Java Database Connectivity (JDBC), a connection is established between the target client and a preset connector in the Serve layer.

3. The data transmission method based on the Myhouse database according to any one of claims 1 to 2, characterized in that, Before the preset syntax analysis algorithm in the preset analyzer performs syntax parsing on the target service request and generates the target data corresponding to the target service request, the method further includes: The target service request is matched in the target server. If there is a matching service request in the target server that matches the target service request, the matching service request is transmitted to the big database, and the big database provides a read / write interface to insert the data.

4. A data transmission device based on a Myhouse database, characterized in that, The device uses the data transmission method based on the Myhouse database as described in claim 1, wherein the data transmission device based on the Myhouse database comprises: The syntax analysis module is used to perform syntax parsing on the target service request based on the preset syntax analysis algorithm in the preset analyzer, and generate target data corresponding to the target service request. The hot data writing module is used to pre-configure the target server and write the hot data in the target data to the target server. The hot data is the target data within a preset time period. The preset time period is the average hot data retention period calculated based on the hot data retention period of historical service requests. A distributed storage module is used to write and store data from the target data that exceeds the preset time period into a preset large database, and to provide a read / write interface through the large database to realize data insertion. Based on the data length corresponding to data exceeding the preset time period, and using a preset formula to calculate the number of preset columnar storage engines in the large database, the preset formula is as follows: Where L is the data length corresponding to data exceeding the preset time period, and S is a single preset columnar storage engine.

5. A device, characterized in that, The device includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the data transmission method based on the Myhouse database as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the data transmission method based on the Myhouse database as described in any one of claims 1 to 3.

Images