Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method which can be applied to the technical field of big data. The data processing method comprises the steps that in response to a received data processing request, to-be-processed data is inquired from a first database, the data in the first database is loaded from a second database, and a first key value of the to-be-processed data in the first database and a second key value of the to-be-processed data in the second database have a corresponding relation; the response to the data processing request comprises calculation operation, to-be-processed data in the second database is determined according to the corresponding relation, the calculation operation is executed in the second database, and the response time for executing the query operation on the data in the first database is shorter than that of the second database; the response time for executing the calculation operation on the data in the second database is shorter than that of the first database. The invention further provides a data processing device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the field of big data, and specifically to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With business systems generating massive amounts of data daily, there is a need for both complex calculations on historical data and rapid data retrieval. Current solutions typically rely on a single type of database, which usually has different advantages and disadvantages. For example, some databases have parallel computing capabilities but lack precise indexes, resulting in long query times and unresponsive pages, while others have low query latency but are unsuitable for large-scale data processing. Consequently, both data querying and data processing cannot achieve fast response times. Summary of the Invention

[0003] In view of the above problems, this disclosure provides a data processing method, apparatus, device, medium and program product to improve the response speed of data query and data processing.

[0004] According to a first aspect of this disclosure, a data processing method is provided, comprising: in response to receiving a data processing request, querying data to be processed from a first database, wherein the data in the first database is loaded from a second database, a first key value of the data to be processed in the first database corresponds to a second key value of the data to be processed in the second database, wherein the first key value is generated based on a pre-configured key value generation rule, and the storage location of the data in the first database is determined based on partition range information corresponding to the first key value and the data access frequency; and in response to the data processing request, including a calculation operation, determining the data to be processed in the second database according to the correspondence, and performing the calculation operation in the second database, wherein the response time for performing the query operation on the data in the first database is less than that for the second database, and the response time for performing the calculation operation on the data in the second database is less than that for the first database.

[0005] According to embodiments of this disclosure, the method further includes: for each sub-data in the data to be loaded in the second database, generating a first key value for the sub-data using a second key value in the second database according to a key value generation rule pre-configured in the first database; loading the data to be loaded into the first database according to the first key value; and determining the key-value correspondence between the first key value and the second key value based on the second key value of each sub-data in the second database.

[0006] According to an embodiment of this disclosure, loading data to be loaded into a first database based on a first key value includes: obtaining partition range information for the first key value in the first database; determining the partition to which each sub-data belongs and the target file corresponding to the partition based on the first key value and partition range information for each sub-data; storing each sub-data in the corresponding target file and loading the target file into the first database.

[0007] According to embodiments of this disclosure, storing each sub-data in a corresponding target file includes: determining the access frequency of each field in the sub-data based on historical access records in a first database and / or a second database; for each sub-data in the data to be loaded, in response to a field access frequency greater than a preset threshold, storing the field in a first region of the target file, wherein the fields in the first region are pre-loaded into memory for direct access when the access process starts; and in response to a field access frequency less than or equal to the preset threshold, compressing the field and storing it in a second region of the target file, wherein the fields in the second region remain unloaded when the access process starts.

[0008] According to embodiments of this disclosure, the method further includes: pre-allocating a first storage capacity to a first region, the first storage capacity being determined based on the storage capacity of a storage unit in a file storage system, the file storage system being a storage system corresponding to the second database; and pre-allocating a second storage capacity to a second region, the second storage capacity being greater than the first storage capacity.

[0009] According to an embodiment of this disclosure, loading data to be loaded into a first database includes: performing a conflict check between the first key value of each sub-data in the data to be loaded and the first key value in the first database; in response to a check result indicating no conflict, loading the target file into the corresponding partition in the first database; and in response to a check result indicating a conflict, recording the conflicting first key value and generating an alarm message.

[0010] According to embodiments of this disclosure, loading data to be loaded into a first database includes: performing integrity verification and format verification on the data to be loaded; in response to a successful verification result, loading the data to be loaded into a second database; in response to a failed verification result, loading the verified sub-data from the data to be loaded into the second database, and generating alarm information based on the failed sub-data from the data to be loaded; wherein, the data to be loaded is newly added data obtained from the second database; the integrity verification includes at least one of the following: verifying the data volume of the newly added data and the data volume of the data to be loaded, and verifying the hash values ​​of key fields in the newly added data and the data to be loaded; the format verification includes verifying the table structure to which the newly added data belongs and the number of fields in the data to be loaded.

[0011] A second aspect of this disclosure provides a data processing apparatus, comprising: a query module, configured to, in response to receiving a data processing request, query data to be processed from a first database, wherein the data in the first database is loaded from a second database, and a first key value of the data to be processed in the first database corresponds to a second key value of the data to be processed in the second database, wherein the first key value is generated based on a pre-configured key value generation rule, and the storage location of the data in the first database is determined based on partition range information corresponding to the first key value and the data access frequency; and a first processing module, configured to, in response to a data processing request including a calculation operation, determine the data to be processed in the second database according to the correspondence, and perform the calculation operation in the second database, wherein the response time for performing a query operation on the data in the first database is less than that for the second database, and the response time for performing a calculation operation on the data in the second database is less than that for the first database.

[0012] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0013] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0014] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0015] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0017] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 3 A flowchart illustrating a data processing method according to another embodiment of the present disclosure is shown schematically;

[0019] Figure 4 A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0020] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0021] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0026] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0027] Embodiments of this disclosure provide a data processing method, comprising: in response to receiving a data processing request, querying data to be processed from a first database, wherein the data in the first database is loaded from a second database, a first key value of the data to be processed in the first database corresponds to a second key value of the data to be processed in the second database, wherein the first key value is generated based on a pre-configured key value generation rule, and the storage location of the data in the first database is determined based on partition range information corresponding to the first key value and the data access frequency; and in response to the data processing request including a calculation operation, determining the data to be processed in the second database according to the correspondence, and performing the calculation operation in the second database, wherein the response time for performing a query operation on the data in the first database is less than that for the second database, and the response time for performing a calculation operation on the data in the second database is less than that for the first database.

[0028] Figure 1 The diagram illustrates an application scenario of data processing according to an embodiment of the present disclosure.

[0029] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0030] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0031] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0032] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0033] It should be noted that the data processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the data processing device provided in this embodiment can generally be located in server 105. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data processing device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0034] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0035] The following will be based on Figure 1 The described scene, through Figures 2-3 The data processing method of the disclosed embodiments will be described in detail.

[0036] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0037] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S220, and this data processing method can be executed, for example, on the server 105 described above.

[0038] In operation S210, in response to receiving a data processing request, the system queries the first database for data to be processed.

[0039] In this embodiment of the disclosure, the data in the first database is loaded from the second database. The first key value of the data to be processed in the first database has a corresponding relationship with the second key value of the data to be processed in the second database. The first key value is generated based on a pre-configured key value generation rule, and the storage location of the data in the first database is determined according to the partition range information corresponding to the first key value and the access frequency of the data.

[0040] A data processing request can refer to an instruction initiated by a business system or user, requesting the execution of operations such as querying, statistical analysis, or calculation on a batch of data. The first database can refer to a database with an indexing mechanism; the second database can refer to a database with parallel computing capabilities. It is understood that the second database is different from the first database.

[0041] The data to be processed can be data that needs to be calculated or queried in response to a data processing request; the first key value can be a unique identifier or index field of the data to be processed in the first database. The second key value can be a unique identifier used by the same data to be processed in the second database. The correspondence refers to a one-to-one correspondence or a regular mapping relationship between the first key value and the second key value, so that the same record in the second database can be accurately found based on the record located in the first database.

[0042] Key-value generation rules can refer to algorithms or strategies pre-configured in the first database to generate unique identifiers for a single piece of data. These rules can be pre-configured based on business query patterns. Specifically, since the second database can be a database with parallel computing capabilities, the key values ​​of data in the second database are usually not conducive to performing query operations on the data in the first database. Therefore, it is necessary to reset the key-value space through key-value generation rules.

[0043] Taking an auto-incrementing order ID as the second key value in the second database as an example, if the order ID is used as the first key value in the first database, this key value may not be suitable for the indexing mechanism of the first database, and may lead to newly added data being stored in a relatively concentrated location in the first database. Therefore, by using key value generation rules, the user ID, order date, and order ID can be combined and encoded as the first key value. Using this first key value to store newly added data in the first database allows the newly added data to fall into different storage locations. Distributing the storage of newly added data avoids the problem of excessive pressure caused by auto-incrementing IDs leading to newly added data being stored in the same partition, and also facilitates parallel loading of newly added data, improving loading efficiency.

[0044] Specifically, key-value generation rules can be designed based on specific business query patterns. Business query patterns refer to several frequently executed types of query statements. Specifically, query statements within a preset time period can be analyzed to obtain the business query patterns for the data to be loaded, and key-value generation rules can be designed based on these patterns. For example, if queries based on amount are the majority, then key-value generation rules can be designed based on order amount; if queries based on time and user ID are the majority, then key-value generation rules can be designed based on time and user ID.

[0045] Furthermore, pre-configuring key-value generation rules using business query patterns can improve query efficiency. For example, frequently queried statements might involve a user's orders within a certain period, or the total amount of a user's orders within a certain period. Assuming the user ID, order date, and order ID are combined as the first key-value pair, queries like "query a user's orders within the last 7 days" can be quickly located and retrieved with minimal scanning overhead.

[0046] Partition range information can refer to the predefined interval rules in the first database that divide the data distribution according to the first key value. It can be understood that in the first database, the data is stored according to the partition range of the first key value.

[0047] Determining the storage location based on the partition range information corresponding to the first key value and the data access frequency can mean first dividing the data into specific logical partitions or target files according to the partition range information corresponding to the first key value. This partition or file can correspond to a specific business dimension (such as time). Then, based on the access frequency, data with different access frequencies are stored in different areas of the partition. For example, high-frequency data is stored in areas that facilitate fast access or areas that are easier to preload, while low-frequency data is stored in areas that only need to be loaded when needed.

[0048] Understandably, relying solely on partition range to determine data storage location may lead to scanning large amounts of infrequently accessed data during data queries, which is detrimental to data query efficiency. Conversely, relying solely on access frequency to determine data storage location may result in data being stored in many random locations, leading to data fragmentation and potentially making it unsuitable for migrating data to the primary database, further hindering data query efficiency. Through the embodiments of this disclosure, by jointly determining data storage location using partition range information and data access frequency, query efficiency can be improved and unnecessary I / O overhead can be reduced.

[0049] In operation S220, in response to a data processing request including a calculation operation, the data to be processed in the second database is determined according to the correspondence, and the calculation operation is performed in the second database.

[0050] In this embodiment of the disclosure, the response time for performing a query operation on data in the first database is shorter than that for the second database, and the response time for performing a calculation operation on data in the second database is shorter than that for the first database.

[0051] Calculation operations can refer to actions that perform statistical, aggregation, calculation, or rule-based judgment on data, such as summation, average, maximum and minimum values, grouping and summarizing, or indicator calculation.

[0052] Specifically, when a data processing request containing computational operations is received, the faster query capability of the first database can be used to filter out records that meet the conditions based on the first key value. Then, the same batch of data to be processed in the second database can be located through the key value correspondence. Finally, the specific computational operation can be performed on the second database with better computational performance, thereby balancing query speed and computational efficiency.

[0053] For example, in a billing analysis scenario, if a request is received to calculate a customer's total bill amount for the past year, the first key-value pair of the relevant bills can be quickly retrieved from the first database based on conditions such as customer number and billing period. This is then mapped to the bill records in the second database, where the parallel computing capabilities of the second database are used to perform the summation operation. As another example, in a log statistics scenario, if a data processing request is received to calculate the daily call count of a certain interface, the first database can be filtered by interface number and time range to obtain the key-value pair of the target logs. Then, these log records are grouped and counted by date in the second database, and finally, the daily statistical results are returned.

[0054] It is understood that the embodiments of this disclosure achieve rapid location of the data to be processed by prioritizing access to a first database with faster response during the query phase, and then mapping the data to a second database with stronger computing power based on the key-value correspondence to complete complex calculation operations. This allows queries and calculations to be performed in more suitable databases, avoiding performance bottlenecks caused by a single database simultaneously handling high-concurrency queries and heavy calculations, thereby significantly shortening the overall response time and improving system resource utilization and processing throughput.

[0055] According to embodiments of this disclosure, the data processing method further includes: for each sub-data in the data to be loaded in the second database, generating a first key value for the sub-data using a second key value in the second database according to a key value generation rule pre-configured in the first database; loading the data to be loaded into the first database according to the first key value; and determining the key-value correspondence between the first key value and the second key value based on the second key value of each sub-data in the second database.

[0056] In this embodiment, the key-value generation rule can refer to an algorithm or strategy pre-configured in the first database to generate a unique identifier for each piece of data. For example, it could specify concatenating the business primary key with the date, performing hash calculations on several fields, or rearranging fields according to a fixed encoding format. This is used to uniformly generate the first key-value pair, ensuring that the identifiers of similar data in the first database are unique and reproducible. For another example, for time-series data (such as order dates), the key-value pair can be designed as "date + order ID" to ensure that the data is stored in chronological order, improving the efficiency of range queries.

[0057] For example, the key-value generation rule can be associated with the second key-value. If the second key-value is the user ID, then the key-value generation rule can be based on the user ID and the order ID to generate the key-value. That is, the first key-value of each sub-data in the data to be loaded is generated by using the second key-value in the second database corresponding to each sub-data in the data to be loaded.

[0058] For the data to be loaded from the second database into the first database, the sub-data records can be read one by one. According to the key-value generation rules pre-configured in the first database, one or more business fields of the sub-data can be combined, operated or encoded to automatically calculate the corresponding first key value. The first key value is then associated with the sub-data and used to store and look up the index identifier of the sub-data in the first database.

[0059] Furthermore, tools can be used to upload files and data from multiple servers to a file storage system in parallel. The data is then processed within the file storage system and loaded into a primary database. This avoids bottlenecks on a single node and solves the slow upload problem inherent in traditional loading methods. Additionally, storage paths and unloading paths can be configured to facilitate management. Before uploading, the system can check for historical files in the target path and automatically clean up redundant data from several days ago, freeing up storage space.

[0060] After the first key value is generated, it can be used as the primary key or index field. The data to be loaded is written to the first database one by one. When inserting or updating a record, the sub-data content is stored in the corresponding data table, and the generated first key value is written to the specified key value field. This forms a record structure that can be quickly located through the first key value within the first database, which is used for efficient querying and access.

[0061] Sub-data can have its original second key value in the second database. During the loading process, the second key value can be obtained at the same time, and the first key value and the second key value are stored in pairs in the mapping table or association structure. This clearly records the key value correspondence of each sub-data in the two databases, so that when processing a request, the second key value in the second database can be accurately retrieved or located based on the first key value in the first database, realizing accurate association and jump of cross-database data.

[0062] For example, if the key-value generation rule uses the second key-value to generate the first key-value, it is not necessary to record the key-value correspondence between the two databases for each piece of sub-data. Then, for the queried data to be processed, the second key-value of the data to be processed can be determined based on the first key-value of the queried data and the key-value generation rule, without needing to occupy additional resources for storing the correspondence.

[0063] It is understood that the embodiments of this disclosure automatically generate a first key value for each piece of sub-data based on a unified key value generation rule during the loading phase, and establish a correspondence between the first key value and the second key value in the second database while writing to the first database. This enables the standardization and automatic mapping of primary keys across databases, avoiding mismatches and omissions caused by manual configuration or temporary splicing. It can quickly and accurately locate the same record between two databases, reduce association overhead, and improve data access efficiency and overall processing stability.

[0064] According to an embodiment of this disclosure, loading data to be loaded into a first database based on a first key value includes: obtaining partition range information for the first key value in the first database; determining the partition to which each sub-data belongs and the target file corresponding to the partition based on the first key value and partition range information for each sub-data; storing each sub-data in the corresponding target file and loading the target file into the first database.

[0065] In this embodiment of the disclosure, the partition range information may refer to the interval rules predefined in the first database for dividing the data distribution according to the first key value. Specifically, before executing data loading, the currently effective partitioning strategy can be read from the metadata or configuration table of the first database. This strategy is the start and end range, hash bucket number, or routing rules of each partition defined for the first key value. Thus, when processing each sub-data, the partition and corresponding file where it should be stored can be accurately determined based on this range information.

[0066] For each piece of sub-data to be loaded, interval matching or hash calculation can be performed based on the first key value and partition range information to find the specific partition to which the key value falls. Then, based on the mapping relationship between the partition and the storage medium, the corresponding target file can be located. A partition can specifically be multiple independent regions or fragments logically or physically divided into data according to strategies such as key value ranges or hash results. The target file can specifically be a storage file or data block corresponding to a partition, used to hold the sub-data records under that partition. It should be noted that before locating the target file, each piece of sub-data can be pre-sorted based on its key value to reduce the sorting overhead during loading. Pre-sorting and block partitioning can solve the problem of garbled conversion in traditional loading methods.

[0067] Each sub-data item is written to the target file corresponding to its respective partition, forming a partitioned data file. Then, the batch loading or import mechanism provided by the first database can be used to load or mount these target files as a whole to the corresponding data table and partition.

[0068] It is understood that the embodiments of this disclosure utilize partition range information to automatically determine the target partition and corresponding file to which each sub-data should be written, thereby achieving precise mapping between data and partitions / files, avoiding manual specification, and improving the efficiency of loading data to be processed into the second database.

[0069] According to embodiments of this disclosure, storing each sub-data in a corresponding target file includes: determining the access frequency of each field in the sub-data based on historical access records in a first database and / or a second database; for each sub-data in the data to be loaded, in response to a field access frequency greater than a preset threshold, storing the field in a first region of the target file, wherein the fields in the first region are pre-loaded into memory for direct access when the access process starts; and in response to a field access frequency less than or equal to the preset threshold, compressing the field and storing it in a second region of the target file, wherein the fields in the second region remain unloaded when the access process starts.

[0070] In this embodiment, historical access records can refer to logs or statistical information of operations such as querying and reading on various fields within a certain period of time, used to reflect actual usage. Access frequency can refer to the number or proportion of times a certain field is accessed per unit time, as statistically determined from historical access records. The first area can refer to a storage segment in the target file specifically reserved for frequently accessed fields, whose data will be preferentially loaded into memory when the process starts. The access process can specifically be a program process or service instance responsible for performing data read, write, calculation, and other operations. Compression processing is the process of encoding and compressing data to reduce storage space and I / O overhead. The second area can refer to a storage segment in the target file used to store low-frequency accessed fields, whose data is not loaded into memory by default when the process starts.

[0071] Specifically, the query logs, API call records, or statistical tables of the first and second databases over a period of time can be analyzed to count the number of times each field is read or used in calculations, and the access frequency index can be calculated accordingly to classify the fields into different levels such as high-frequency access and low-frequency access.

[0072] When the access frequency of a field exceeds a preset threshold, the field can be considered a high-frequency access field. When writing to the target file, it can be placed in the first region. The first region is a storage area in the target file specifically reserved for high-frequency access fields. The data block corresponding to the first region will be loaded into memory as a whole during the access process startup or task initialization phase. Then, during the query, these fields can be read directly from memory, thereby reducing disk I / O and improving the access response speed of high-frequency fields.

[0073] When the access frequency of a field is less than or equal to a preset threshold, the field can be considered a low-frequency access field. When writing to the target file, it can be compressed and encoded first to reduce storage usage, and then placed in a special low-frequency area. The second area is not actively loaded into memory when the process starts, but is only decompressed and read as needed when there is a genuine access requirement. This saves memory resources and reduces the interference of low-frequency data on I / O and cache, thereby improving the overall system resource utilization.

[0074] Taking order data as an example, frequently accessed fields may include order status (such as "pending payment" or "completed"), order amount, user ID, etc. These fields are often used for real-time queries and reports; infrequently accessed fields may include order description, historical logs, etc., which are mainly used for auditing or occasional analysis.

[0075] According to embodiments of this disclosure, the method further includes: pre-allocating a first storage capacity to a first region, the first storage capacity being determined based on the storage capacity of a storage unit in a file storage system, the file storage system being a storage system corresponding to the second database; and pre-allocating a second storage capacity to a second region, the second storage capacity being greater than the first storage capacity.

[0076] In embodiments of this disclosure, the first storage capacity may refer to a fixed or upper limit space size pre-allocated for a first region in the target file, used to store frequently accessed fields. The file storage system may be the underlying storage system carrying the second database data file, such as a distributed file system or a block storage system, and the storage unit may specifically be the smallest or basic capacity granularity that can be independently allocated and managed in the file storage system.

[0077] Specifically, when dividing the first region for high-frequency fields, the basic storage unit size of the file storage system used by the data file carrying the second database will be referenced, and appropriate space capacity will be reserved for the first region at this granularity or an integer multiple thereof.

[0078] The second area set for low-frequency fields in the target file will reserve a larger storage space than the first area to centrally accommodate the compressed but larger-volume low-frequency data.

[0079] It is understood that, through the embodiments of this disclosure, by allocating a first storage capacity consistent with the storage unit of the underlying file storage system to the first region, the upper-layer data blocks can be aligned with the lower-layer storage blocks, reducing cross-block read / write and multiple I / O calls, and improving sequential scan efficiency; by allocating a larger second storage capacity to the second region, it can accommodate larger volumes of low-frequency data that are compressed and stored, reducing fragmentation and the number of files, lowering management overhead and storage costs, thereby optimizing disk access and space utilization overall.

[0080] According to an embodiment of this disclosure, loading data to be loaded into a first database includes: performing a conflict check between the first key value of each sub-data in the data to be loaded and the first key value in the first database; in response to a check result indicating no conflict, loading the target file into the corresponding partition in the first database; and in response to a check result indicating a conflict, recording the conflicting first key value and generating an alarm message.

[0081] In this embodiment of the disclosure, conflict checking specifically refers to comparing the key value of the data to be written with the key value already existing in the database before data loading, in order to determine whether there is a primary key duplication, unique index conflict or other disallowed data identifier conflict, thereby avoiding data overwriting, constraint abnormality or data consistency problems caused by key value duplication.

[0082] Specifically, before loading the target file into the corresponding partition of the first database, the first key value of each sub-database to be loaded can be checked one by one to see if a record with the same key value already exists in the existing partitions or the entire database. If the verification result is that there is no conflict, it means that the primary key or uniqueness constraint will not be violated, and the target file containing these sub-databases can be safely loaded into the corresponding partition of the first database. If the verification result is that there is a conflict, the specific first key value and related information of the conflict can be recorded, and an alarm message can be generated to prompt the operation and maintenance or upper-level business to perform manual processing or policy decision-making, so as to avoid erroneous data being directly written and causing overwriting or anomalies.

[0083] For conflicting data, different processing methods can be used. For example, in data update scenarios, historical data can be automatically overwritten. Alternatively, conflicting data can be retained and marked as "pending processing" for subsequent manual intervention by the business system or merged through scripts (e.g., taking the data with the latest timestamp).

[0084] Furthermore, when performing conflict checking, you can check only the newly added data. For example, first identify newly added data (e.g., data with the same primary key not found in the first database) and updated data (e.g., data with the same primary key found) according to business rules. For sub-data determined to be updated, write it directly as an update operation without performing primary key conflict checking; while for newly added sub-data, check whether its first key value is duplicated with an existing key value in the first database or other newly added key values ​​in this batch before loading.

[0085] It is understood that in this embodiment of the disclosure, conflict checking can ensure the uniqueness of key values, facilitate investigation and remediation, and balance import performance and data correctness. Instead of full-scale checking, conflict checking is performed on the first key value newly added in the target file and the key value already existing in the second database, which can significantly reduce the scale of the checked data and the time consumption, and improve data loading efficiency.

[0086] According to embodiments of this disclosure, loading data to be loaded into a first database includes: performing integrity verification and format verification on the data to be loaded; in response to a successful verification result, loading the data to be loaded into a second database; in response to a failed verification result, loading the verified sub-data from the data to be loaded into the second database, and generating alarm information based on the failed sub-data from the data to be loaded; wherein, the data to be loaded is newly added data obtained from the second database; the integrity verification includes at least one of the following: verifying the data volume of the newly added data and the data volume of the data to be loaded, and verifying the hash values ​​of key fields in the newly added data and the data to be loaded; the format verification includes verifying the table structure to which the newly added data belongs and the number of fields in the data to be loaded.

[0087] In this embodiment, newly added data refers to data records that are newly generated and written to the second database within a preset time window relative to existing historical data in the second database, and have not yet been synchronized or loaded into the first database. It should be noted that when the data to be loaded is newly added data obtained from the second database, historical data from the second database already exists in the first database.

[0088] Integrity checks are used to confirm the consistency of content between newly added data and data to be loaded. For example, comparing the number of records and bytes can detect whether some records were not successfully extracted, transmission was interrupted, or duplicate extraction occurred. Another example is calculating the hash value of pre-selected key fields in the newly added data and the data to be loaded, and comparing whether the two hash values ​​are consistent to identify whether there are problems such as tampering, truncation, or encoding abnormalities at the field level, thereby filtering out data with integrity risks before entering the first database.

[0089] Format validation focuses on structural correctness checks. For example, for the target table structure to which the new data belongs, it verifies whether the number, order, and type of fields in the data to be loaded match the definition of the target table. When the table structure defines several required fields, if the number of fields in the data to be loaded is insufficient or there are invalid fields, it can be determined that the format does not meet the requirements. Through this type of validation, loading failures or dirty data writing caused by reasons such as unsynchronized table structure changes or incorrect field mapping configurations can be avoided.

[0090] If all the above integrity and format checks pass, the entire batch of data to be loaded can be loaded into the target database, enabling one-time, batch writing of new data.

[0091] If the verification results show that only some sub-data in the data to be loaded has integrity or format issues, a partial pass / partial block strategy can be adopted. That is, sub-data that passes verification is loaded normally into the target database, while sub-data that fails verification is not loaded, and alarm information is generated based on these abnormal sub-data. The alarm information can include the specific sub-data identifier that erroneous, the corresponding verification failure type, and the reason. This facilitates timely identification of the problem source by operations and maintenance personnel or upper-level business logic, enabling manual verification and repair. This ensures overall loading progress while reducing the impact of abnormal data on the system and improving overall data quality.

[0092] Figure 3 A flowchart illustrating a data processing method according to another embodiment of the present disclosure is shown.

[0093] like Figure 3 As shown, the data processing method of this embodiment includes operations S310 to S350, and this data processing method can be executed, for example, on the server 105 described above.

[0094] When operating the S310, raw data and data processing scripts are obtained from the data integration interface.

[0095] In this embodiment, raw data can be obtained through a pre-configured data integration interface. This data integration interface employs a fine-grained access control mechanism, providing a dedicated access channel for the source data to the data warehouse. It can grant only read-only access and access permissions based on business tables, thus avoiding write risks and unauthorized access risks to the source business database at the interface level. Simultaneously, the interface layer can also incorporate built-in data pre-validation capabilities to perform field integrity checks and data type checks on the collected raw data. Integrity checks are used to check whether core business fields are empty or missing, while data type checks ensure that numeric fields do not contain character anomalies and that date fields conform to a unified format specification.

[0096] Simultaneously, data processing scripts corresponding to the original data can be obtained or determined based on the data integration interface. These scripts are pre-developed using a standardized scripting language, enabling three-level processing logic: cleaning, transformation, and integration. In the data cleaning stage, missing numerical values ​​are filled with the mean, missing categorical values ​​are filled with the pattern, and abnormal records exceeding preset business thresholds are removed. In the data transformation stage, business field mapping and encoding conversion are completed. In the data integration stage, data from multiple source tables are linked and integrated to form a structured, consistent data set.

[0097] In addition, the above scripts can be centrally managed through a code version control system, which records the person responsible for each modification, the content of the modification, and the timestamp. It supports version rollback and difference comparison, thereby ensuring that the data processing scripts used when performing operation S320 are traceable and controllable, and avoiding data processing deviations caused by accidental script modifications.

[0098] When operating the S320, the raw data is processed on the big data platform based on data processing scripts.

[0099] In this embodiment of the disclosure, the acquired raw data and corresponding data processing scripts can be deployed to a big data platform, where the raw data can be automatically processed based on the data processing scripts. For example, the big data platform first processes the raw data according to the pre-set cleaning, transformation, and integration process in the script to generate intermediate or final result datasets that meet the requirements.

[0100] Furthermore, data processing scripts on the big data platform are subject to unified scheduling, monitoring, and alarm management. For example, scheduling can be achieved by configuring upstream dependencies to ensure that the script is automatically triggered only after the source data synchronization is complete, and manual reruns or restarts are supported in cases of missing or abnormal data. Monitoring allows the big data platform to track the script's running status, execution time, and data volume in real time, automatically identifying execution failures, timeouts, or abnormal data volumes. Alarm management can push alarm information through multiple channels such as SMS and email once an anomaly is detected, while simultaneously recording detailed start and end times, processing volume, and error details in the logs, facilitating rapid problem location and troubleshooting within a limited time.

[0101] During operation S330, the processed data is stored in the second database.

[0102] In this formula embodiment, the processed business data can be written to the target table of the second database according to a pre-designed partitioning strategy, such as partitioning by date. During daily loading, only the current date partition is inserted or updated, avoiding scanning and rewriting the entire table, thereby improving the efficiency of daily batch loading. When querying historical data, only the partitions corresponding to the date range need to be scanned, thus narrowing the query range.

[0103] Furthermore, resource configuration can be based on two dimensions: data frequency and data volume. For example, when the business operates on a daily basis, daily incremental partitions can be designed accordingly. If it needs to be upgraded to hourly loading in the future, it can be further refined into hourly partitions and the scheduling cycle can be adjusted accordingly. As another example, based on the scale of newly added daily data and the volume of historical data migrated for the first time, table space and parallel computing resources can be reasonably estimated and allocated for the second database, reserving sufficient redundancy to cope with peak loads.

[0104] Furthermore, upon receiving a computational instruction, the second database utilizes its parallel computing capabilities to perform complex calculations and aggregation operations on the stored data within the database. These operations include multi-table joins, aggregation operations, and window function analysis, which are then used to generate report data or analytical metrics. For example, order data can be grouped and aggregated by user ID to calculate each user's total spending and average order amount.

[0105] The processing results are stored in the second database as temporary or intermediate result tables to avoid repeatedly reading and processing the original data. Simultaneously, data quality verification rules can be executed in the second database, such as checking that the amount field is not negative and that the order of the time field is logically sound, ensuring that the processed data stored in the second database is accurate and consistent in terms of business semantics.

[0106] During operation of S340, data from the second database is uploaded to the file storage system.

[0107] In this embodiment, pre-processed or processed data can be extracted from the second database and written to the file storage system according to a preset file format and partitioning rules. Specifically, the data exported from the second database can be fragmented and named based on the business date or partition key, and stored in a distributed file system or object storage in the form of text files, columnar files, etc., while recording the corresponding metadata (such as file path, generation time, etc.).

[0108] In operation S350, data is processed in the file storage system and the processed data is loaded into the first database.

[0109] In this embodiment of the disclosure, data uploaded to the file storage system can be generated to conform to the target table structure and business scope of the first database. After processing, this data can be loaded into the first database in accordance with the loading specifications of the first database through batch writing, partition insertion, or other methods. Necessary integrity checks, format checks, and constraint checks are performed before or during loading to ensure that the written data meets the requirements of the first database in terms of both structure and content.

[0110] Based on the above data processing method, this disclosure also provides a data processing apparatus. The following will be combined with... Figure 4 The device is described in detail.

[0111] Figure 4 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0112] like Figure 4 As shown, the data processing device 400 of this embodiment includes a query module 410 and a first processing module 420.

[0113] The query module 410 is used to query the data to be processed from the first database in response to receiving a data processing request. The data in the first database is loaded from the second database. The first key value of the data to be processed in the first database corresponds to the second key value of the data to be processed in the second database. The first key value is generated based on a pre-configured key value generation rule, and the storage location of the data in the first database is determined based on the partition range information corresponding to the first key value and the data access frequency. In one embodiment, the query module 410 can be used to execute the operation S210 described above, which will not be repeated here.

[0114] The query module 410 is used to respond to a data processing request, including a calculation operation, to determine the data to be processed in the second database according to the correspondence, and to perform the calculation operation in the second database. The response time for performing a query operation on data in the first database is shorter than that for the second database, and the response time for performing a calculation operation on data in the second database is shorter than that for the first database. In one embodiment, the query module 410 can be used to perform the operation S220 described above, which will not be repeated here.

[0115] According to an embodiment of this disclosure, the data processing apparatus 400 further includes a key-value generation module, a loading module, and a first determination module. The key-value generation module is used to generate a first key-value for each sub-data in the data to be loaded in the second database, based on a key-value generation rule pre-configured in the first database, using the second key-value of the sub-data in the second database. The loading module is used to load the data to be loaded into the first database based on the first key-value. The first determination module is used to determine the key-value correspondence between the first key-value and the second key-value based on the second key-value of each sub-data in the second database.

[0116] According to embodiments of this disclosure, the loading module includes an acquisition module, a second determination module, and a second processing module. The acquisition module is used to acquire partition range information for a first key value in the first database. The second determination module is used to determine the partition to which each sub-data belongs and the target file corresponding to the partition based on the first key value and partition range information of each sub-data. The second processing module stores each sub-data in the corresponding target file and loads the target file into the first database.

[0117] According to embodiments of this disclosure, the second processing module includes a third determining module, a first storage module, and a second storage module. The third determining module is used to determine the access frequency of each field in the sub-data based on historical access records in the first database and / or the second database. The first storage module is used to store the field in a first region of the target file for each sub-data in the data to be loaded, in response to the access frequency of the field being greater than a preset threshold. The fields in the first region are pre-loaded into memory for direct access when the access process starts. The second storage module is used to compress the field and store it in a second region of the target file in response to the access frequency of the field being less than or equal to the preset threshold. The fields in the second region remain unloaded when the access process starts.

[0118] According to an embodiment of this disclosure, the data processing device 400 further includes a first allocation module and a second allocation module. The first allocation module is used to pre-allocate a first storage capacity to a first region. The first storage capacity is determined based on the storage capacity of a storage unit in a file storage system, which is a storage system corresponding to the second database. The second allocation module is used to pre-allocate a second storage capacity to a second region. The second storage capacity is greater than the first storage capacity.

[0119] According to an embodiment of this disclosure, the loading module includes a first verification module, a first loading submodule, and an alarm information generation module. The first verification module is used to perform conflict verification between the first key value of each subdata in the data to be loaded and the first key value in the first database. The first loading submodule is used to load the target file into the corresponding partition in the first database in response to the verification result that there is no conflict. The alarm information generation module records the conflicting first key value and generates alarm information in response to the verification result that there is a conflict.

[0120] According to embodiments of this disclosure, the loading module includes a second verification module, a second loading submodule, and a third processing module. The second verification module is used to perform integrity verification and format verification on the data to be loaded. The second loading submodule is used to load the data to be loaded into a second database in response to a successful verification result. The third processing module is used to load the verified sub-data from the data to be loaded into the second database in response to a failed verification result, and to generate alarm information based on the failed sub-data from the data to be loaded. The data to be loaded is newly added data obtained from the second database. The integrity verification includes at least one of the following: verifying the amount of newly added data and the amount of data to be loaded, and verifying the hash values ​​of key fields in the newly added data and the data to be loaded. The format verification includes verifying the table structure to which the newly added data belongs and the number of fields in the data to be loaded.

[0121] According to embodiments of this disclosure, any plurality of modules in query module 410 and first processing module 420 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of query module 410 and first processing module 420 may be at least partially implemented as hardware circuitry, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of query module 410 and first processing module 420 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0122] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0123] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0124] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0125] According to embodiments of this disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0126] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0127] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0128] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this disclosure.

[0129] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0130] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0131] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0132] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0135] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, characterized in that, The method includes: In response to receiving a data processing request, the system queries the data to be processed from the first database, wherein the data in the first database is loaded from the second database, and the first key value of the data to be processed in the first database has a corresponding relationship with the second key value of the data to be processed in the second database, wherein the first key value is generated based on a pre-configured key value generation rule, and the storage location of the data in the first database is determined based on the partition range information corresponding to the first key value and the access frequency of the data. In response to the data processing request including a calculation operation, the data to be processed in the second database is determined according to the correspondence, and the calculation operation is performed in the second database. The response time for performing a query operation on the data in the first database is shorter than that for the second database, and the response time for performing a calculation operation on the data in the second database is shorter than that for the first database.

2. The method according to claim 1, characterized in that, The method further includes: For each sub-data in the data to be loaded in the second database, according to the key-value generation rules pre-configured in the first database, the first key-value of the sub-data is generated using the second key-value of the sub-data in the second database; Based on the first key value, the data to be loaded is loaded into the first database; Based on the second key value of each sub-data in the second database, the key-value correspondence between the first key value and the second key value is determined.

3. The method according to claim 2, characterized in that, The step of loading the data to be loaded into the first database according to the first key value includes: Obtain the partition range information for the first key value from the first database; Based on the first key value of each sub-data and the partition range information, determine the partition to which each sub-data belongs and the target file corresponding to the partition; Each sub-data is stored in a corresponding target file and the target file is loaded into the first database.

4. The method according to claim 3, characterized in that, The step of storing each sub-data into a corresponding target file includes: The access frequency of each field in the sub-data is determined based on the historical access records in the first database and / or the historical access records in the second database; For each sub-data in the data to be loaded, in response to the access frequency of the field being greater than a preset threshold, the field is stored in the first region of the target file. The fields in the first region are pre-loaded into memory for direct access when the access process starts. In response to the access frequency of the field being less than or equal to the preset threshold, the field is compressed and stored in the second region of the target file, wherein the fields in the second region remain unloaded when the access process starts.

5. The method according to claim 4, characterized in that, The method further includes: A first storage capacity is pre-allocated to the first region. The first storage capacity is determined based on the storage capacity of the storage units in the file storage system, which is the storage system corresponding to the second database. A second storage capacity is pre-allocated to the second region, and the second storage capacity is greater than the first storage capacity.

6. The method according to claim 2, characterized in that, The step of loading the data to be loaded into the first database includes: Perform conflict verification between the first key value of each sub-data in the data to be loaded and the first key value in the first database; In response to the verification result indicating no conflict, the target file is loaded into the corresponding partition of the first database; In response to the verification result indicating a conflict, the first key value with the conflict is recorded and an alarm message is generated.

7. The method according to claim 2, characterized in that, The step of loading the data to be loaded into the first database includes: Perform integrity and format checks on the data to be loaded; In response to the verification result being passed, the data to be loaded is loaded into the second database; In response to a verification failure, the verified sub-data in the data to be loaded is loaded into the second database, and an alarm message is generated based on the verified sub-data in the data to be loaded. The data to be loaded is newly added data obtained from the second database; The integrity verification includes at least one of the following: verifying the amount of data in the newly added data and the amount of data to be loaded; verifying the hash values ​​of key fields in the newly added data and the data to be loaded. The format validation includes validating the table structure to which the new data belongs and the number of fields in the data to be loaded.

8. A data processing apparatus, characterized in that, The device includes: The query module is used to respond to a received data processing request by querying data to be processed from a first database. The data in the first database is loaded from a second database. The first key value of the data to be processed in the first database corresponds to the second key value of the data to be processed in the second database. The first key value is generated based on a pre-configured key value generation rule. The storage location of the data in the first database is determined based on the partition range information corresponding to the first key value and the access frequency of the data. A first processing module is configured to respond to the data processing request, which includes a calculation operation, determine the data to be processed in the second database according to the correspondence, and perform the calculation operation in the second database, wherein the response time for performing a query operation on the data in the first database is shorter than that for the second database, and the response time for performing a calculation operation on the data in the second database is shorter than that for the first database.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images