A dynamic data access method and device, electronic equipment and storage medium

Abstract

The application relates to a dynamic data access method and device, electronic equipment and a storage medium. The dynamic data access method comprises the following steps: acquiring a data set pushed by a data pushing party; judging whether the format of each piece of pushing data is a preset standard format; for the pushing data in the standard format, scanning a full table field name of the pushing data, judging whether the pushing data can be associated with a registered resource type; performing table and partition processing on the pushing data, and obtaining a split table name and a partition block name corresponding to the pushing data according to the associated resource type; and storing each piece of pushing data in the data set according to the split table name and the partition block name, and completing data access. The dynamic data access method can automatically apply a table and partition rule in a data warehousing process, so that rapid data segmentation and input are realized.

Description

Technical Field

[0001] This invention relates to the field of data structure technology, and in particular to a method, apparatus, electronic device, and storage medium for accessing dynamic data. Background Technology

[0002] Current technical investigation operations often require the use of massive amounts of dynamic data. Dynamic data here refers to real-time updated data from telecommunications, transportation, logistics, and financial transactions, characterized by high update frequency, large update volume, long-term validity, large total proportion, prominent time attributes, and significant business application demands. While existing technologies provide systematic methods for the governance and application of massive dynamic data, these technologies focus on the entire process of governance, offering relatively general discussions but lacking specific optimization of governance sub-processes. In particular, the fundamental sub-process of data governance—data access—is not specifically optimized for the characteristics of dynamic data. In actual business operations, data departments often need to urgently access the latest and most comprehensive data as support. A batch of new data must be searchable and put into analysis and application before data processing. This places high demands on the speed of data access and rapid data extraction. The department that formulates data processing rules also needs to be able to view data conveniently and quickly at any time to save time in data processing. In addition, although the data source is traceable, due to the wide range of units and departments involved in the data and the diversity of main business areas, the data resource access standards of various data source departments are not uniform, and the naming of resource resources is not standardized, resulting in chaotic resource classification. For the same type of data, it is not possible to call up existing data examination results and governance methods in a timely manner, which increases the workload of data processing.

[0003] Existing technologies focus more on system governance methods and business collaboration for massive dynamic data (such as CN 108228693A and CN105554070A). They explore a series of data governance processes, including data access, processing, and monitoring, as well as organizational structure research to achieve core business operations. These studies emphasize overall management and specific service content, lacking specific optimization of data governance sub-processes and dedicated research on optimization methods for massive dynamic data access. Existing technologies that study data access from a table partitioning perspective (such as CN202110312972.4 and CN202010578314.5) have relatively cumbersome rule-building processes for massive dynamic data requiring rapid table partitioning and data access, and lack unified naming standards for resources. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a method, device, electronic device and storage medium for accessing dynamic data, and to provide a set of automatic table partitioning rules for the system, so that during the data entry process, it is not necessary to frequently perform specific observation and pre-analysis of different resource types of data and formulate table partitioning rules in a manual manner, but the system automatically applies the table partitioning rules of the time dimension to achieve rapid data segmentation and entry.

[0005] In a first aspect, the present invention provides a method for accessing dynamic data, comprising the following steps:

[0006] Obtain the dataset pushed by the data pusher, wherein the dataset consists of multiple push data entries, and each push data entry corresponds to an event time information entry;

[0007] Determine whether the format of each push data item is a preset standard format;

[0008] For the standard format push data, scan all the field names in the push data table to determine whether the push data can be associated with a registered resource type;

[0009] When the push data can be associated with a registered resource type, the push data is partitioned according to the preset table partitioning rules and partitioning rules, and the split table name and partition block name corresponding to the push data are obtained according to the associated resource type.

[0010] Each piece of push data in the dataset is stored according to the split table name and partition block name to complete the data access.

[0011] Furthermore, the table partitioning rules and partitioning rules include:

[0012] The event time is broken down into multiple time dimensions according to granularity;

[0013] Compare the amount of data corresponding to each time dimension with the preset threshold value;

[0014] The time dimension whose corresponding data volume is less than and closest to the threshold value is determined as the table partitioning dimension;

[0015] The next level of the aforementioned table partitioning dimension is used as the partitioning dimension.

[0016] Furthermore, after storing each piece of push data in the dataset according to the split table name and partition block name, the following steps are also included:

[0017] The stored dataset is named according to the preset data resource naming rules;

[0018] Obtain the review results from the reviewers regarding the naming of the resources;

[0019] If the review result is approved, the dataset and corresponding resource names will be uploaded to the designated database to complete the data access.

[0020] Furthermore, before determining whether each piece of pushed data meets the data entry criteria, the process also includes:

[0021] Randomly select several push data from the dataset as sample data, and scan each sample data to see if there is corresponding event time information;

[0022] If the sample data does not contain the corresponding event time information, the push process of the dataset is stopped, an error message is generated, and feedback is sent to the data pusher.

[0023] Furthermore, it also includes the following steps:

[0024] When it is detected that there is corresponding event time information in all sample data, scan the field null value rate and / or format logic error rate of the sample data;

[0025] When the field null value rate and / or format logic error rate exceed the first threshold, the data push process is terminated, and an error message is generated and fed back to the data pusher.

[0026] Furthermore, the preset standard formats include any of the following: csv, xls, json, xml, and raf.

[0027] Furthermore, it also includes the following steps:

[0028] When the pushed data cannot be associated with a registered resource type, the newly created resource type of the configuration personnel is obtained, and the newly created resource type can be associated with the pushed data.

[0029] Secondly, the present invention also provides a dynamic data access device, comprising:

[0030] The dataset acquisition module is used to acquire the dataset pushed by the data pusher. The dataset consists of multiple push data, and each push data corresponds to an event time information.

[0031] The format determination module is used to determine whether the format of each piece of pushed data is a preset standard format;

[0032] The registered resource determination module is used to scan the full table field names of the push data in the standard format and determine whether the push data can be associated with a registered resource type.

[0033] The table partitioning module is used to partition the push data according to preset table partitioning rules and partitioning rules when the push data can be associated with a registered resource type, and to obtain the split table name and partition block name corresponding to the push data according to the associated resource type.

[0034] The data storage module is used to store each piece of push data in the dataset according to the split table name and partition block name, thereby completing the data access.

[0035] Thirdly, the present invention also provides an electronic device, comprising:

[0036] At least one memory and at least one processor;

[0037] The memory is used to store one or more programs;

[0038] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a dynamic data access method as described in any of the first aspects of the present invention.

[0039] Fourthly, the present invention also provides a computer-readable storage medium.

[0040] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a dynamic data access method as described in any of the first aspects of the present invention.

[0041] This invention provides a method, apparatus, electronic device, and storage medium for accessing dynamic data. Addressing the data access process for massive amounts of dynamic data, it primarily offers a method to solve the problem of rapid data entry into the database due to the huge volume of real-time updates. This invention utilizes resource type registration to unify the resource types of data from multiple sources with the same structure but inconsistent naming. Compared to traditional technologies, it emphasizes the time-centric nature of dynamic data, providing a set of automatic table partitioning rules. This eliminates the need for frequent manual observation, pre-analysis, and rule formulation for different resource types during the data entry process. Instead, the system automatically applies time-based partitioning rules to achieve rapid data segmentation and entry. This avoids the significant time spent manually formulating partitioning rules and facilitates rapid data location for users in data mining operations based on time attributes and unified resource types.

[0042] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0043] Figure 1 A schematic diagram illustrating the steps of a dynamic data access method provided by the present invention;

[0044] Figure 2 This is a flowchart illustrating a preferred embodiment of the present invention in one instance;

[0045] Figure 3 This is a schematic diagram of the structure of a dynamic data access device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0047] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0048] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0049] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0050] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0051] To address the problems in the background art, embodiments of this application provide a method for accessing dynamic data, such as... Figure 1and Figure 2 As shown, the method includes the following steps:

[0052] S01: Obtain the dataset pushed by the data pusher. The dataset consists of multiple push data entries, each corresponding to an event time information entry.

[0053] In specific application scenarios, it is necessary to communicate with the data provider in advance to determine unified data push rules, formats, data transmission methods, and other standards. Following these unified standards, the data provider collects data along a specific time dimension (such as a day, month, year, etc.), categorizes it into datasets, and uploads them to the system along with basic data information.

[0054] Since the data provider may not adhere to a unified standard in data processing and transmission, a preferred embodiment further includes the following steps:

[0055] S11: Randomly select several push data from the dataset as sample data, and scan each sample data to see if there is corresponding event time information.

[0056] For example, the system randomly selects a certain number of sample data for each dataset based on the data generating institution (e.g., 500 samples each from institution A and institution B).

[0057] S12: When it is detected that there is no corresponding event time information in the sample data, the push process of the dataset is stopped, an error message is generated, and feedback is sent to the data pusher.

[0058] S13: When it is detected that there is corresponding event time information in all sample data, scan the field null value rate and / or format logic error rate of the sample data.

[0059] S14: When the field null value rate and / or format logic error rate exceed the first threshold, the data push process is terminated, and an error message is generated and fed back to the data pusher.

[0060] In one specific example, the first threshold is 30%. In other application scenarios, this threshold can be adjusted according to the actual situation.

[0061] After receiving the error feedback, the data pusher completes the event time field information for the dataset that was stopped from being pushed and re-pushes it until the system passes the sample data scan.

[0062] S02: Determine whether the format of each push data item is a preset standard format.

[0063] In a specific application scenario, the default standard formats include any of the following: csv, xls, json, xml, and raf.

[0064] Preferably, error messages are generated for data tables that do not conform to the standard format and fed back to the data entry personnel. The data entry personnel then use other format conversion tools to convert the format. The converted dataset is stored in a data mirror library (this mirror library does not provide open search functionality and only serves as a storage library for the original data).

[0065] S03: For the standard format push data, scan all the field names in the push data table to determine whether the push data can be associated with a registered resource type.

[0066] The system scans the full field names of newly stored datasets in the mirror repository, combines the dataset's data category with the registered resource types in the configuration repository, and associates the data resources.

[0067] There are two matching scenarios:

[0068] (1) Complete match, that is, all fields of the dataset completely match the field information of the registered resource type, and the "Push and Resource Type Association Table" in the configuration library already contains the corresponding association relationship, then the dataset will be automatically configured with the resource type according to this association relationship, and proceed to step 4 (e.g., if a department pushes a dataset of data category "highway", all its fields are the same as the registered resource type "highway passage record", and there is an association relationship between data category "highway" and resource type "highway passage record" in the "Push and Resource Type Association Table", then the resource type of this push dataset will be automatically configured as "highway passage record").

[0069] (2) If a complete match is not found, the system will compare the full field names of the dataset with the field information of the registered resource type and recommend resource types with a high degree of matching. The configuration personnel will determine the association between the dataset and the resource type through recommendations and data observation. If necessary, the resource type will be expanded and the association will be added to the "Push and Resource Type Association Table". After the data entry personnel approve the data, the association between the dataset and the resource type will be completed. (For example, if a department pushes a dataset of the data category "Highway", its full field information is highly matched with the field information of the resource type "Motor Vehicle Passage Record". The system will automatically recommend association. However, the configuration personnel will find through data observation that the data content of the dataset tends to be related to the resource type "Highway Passage Record". In this case, a new field list information of the resource type "Highway Passage Record" will be added to the configuration library and the association will be expanded in the "Push and Resource Type Association Table".)

[0070] When the pushed data cannot be associated with a registered resource type, the newly created resource type of the configuration personnel is obtained, and the newly created resource type can be associated with the pushed data.

[0071] S04: When the push data can be associated with a registered resource type, the push data is partitioned according to the preset table partitioning rules and partitioning rules, and the split table name and partition block name corresponding to the push data are obtained according to the associated resource type.

[0072] Preferably, the table partitioning rules and partitioning rules include:

[0073] (1) The event time is divided into multiple time dimensions according to the granularity.

[0074] For example, the event time can be broken down into dimensionality 1, dimensionality 2, dimensionality 3, etc., corresponding to year, month, week, etc. (If the event time is accurate to the date, the time dimension is broken down into four dimensions: year, month, week, and day. If the event time is accurate to the second, the time dimension is broken down into seven dimensions: year, month, week, day, hour, minute, and second).

[0075] (2) Compare the amount of data corresponding to each time dimension with the preset threshold value.

[0076] (3) The time dimension whose corresponding data volume is less than and closest to the threshold value is determined as the table partitioning dimension.

[0077] The determination of the threshold value needs to be determined through testing, and it is usually based on the database query efficiency (e.g., the database query efficiency requirement is that 5 million data can be queried in 100ms. A certain dataset in the mirror library has an average of 1,000 data per second, 60,000 data per minute, 3.5 million data per hour, and 90 million data per day across various time dimensions, from 3.5 million < 5 million < 90 million. This dataset is split into tables by hour).

[0078] (4) Use the next level of the table partitioning dimension as the partitioning dimension.

[0079] For example, a dataset partitioned by hour can be partitioned by minute. When the partitioning dimension is the smallest time dimension, partitions are made according to a specified amount of data, and each block is identified by a numerical sequence: for example, partitioning into blocks of 1 million data entries, and identifying each partition block in ascending order 0001, 0002, etc.

[0080] S05: Store each piece of push data in the dataset according to the split table name and partition block name to complete the data access.

[0081] For the dataset that has been partitioned, multiple data resources are created (abnormal data tables also need to be created as separate data resources), and each data resource is labeled using the resource naming method.

[0082] Detailed steps for naming resources:

[0083] (1) If the resource type registered by the data resource has successfully completed access before, the historical access naming rules shall be applied.

[0084] (2) When the resource type registered by the data resource is the first access resource type, configure the naming of each component, including but not limited to: the administrative region involved in the data, the list of generating institutions, the time dimension of the data table, and complete the naming of the data resource by comprehensively considering the resource type, such as "Guangdong Province Institution A Expressway Passage Record (January 2022)";

[0085] (3) The list of administrative regions and generating organizations that do not exist in the configuration library needs to be expanded before naming configuration.

[0086] In practical applications, data entry personnel manually review whether the naming of data resources is reasonable. If the review is successful, the data resources and their basic information are uploaded to the front-end raw layer database (a database that can provide shared queries), and the data access is completed. If the review fails, the naming components need to be reconfigured until the review is successful before the data access is completed.

[0087] Correspondingly, in a preferred embodiment, the following steps are also included:

[0088] S051: Name the stored dataset according to the preset data resource naming rules.

[0089] S052: Obtain the review results of the reviewers regarding the naming of the resources.

[0090] S053: When the audit result is approved, the dataset and corresponding resource names are uploaded to the designated database to complete the data access.

[0091] This application also provides a dynamic data access device, such as... Figure 3 As shown, the dynamic data access device 400 includes:

[0092] The dataset acquisition module 401 is used to acquire the dataset pushed by the data pusher. The dataset consists of multiple push data, and each push data corresponds to an event time information.

[0093] The format determination module 402 is used to determine whether the format of each push data item is a preset standard format;

[0094] The registered resource judgment module 403 is used to scan the full table field names of the push data in the standard format and determine whether the push data can be associated with the registered resource type.

[0095] The table partitioning module 404 is used to perform table partitioning on the push data according to preset table partitioning rules and partitioning rules when the push data can be associated with a registered resource type, and to obtain the split table name and partition block name corresponding to the push data according to the associated resource type.

[0096] The data storage module 405 is used to store each piece of push data in the dataset according to the split table name and partition block name, thereby completing the data access.

[0097] Preferably, the table partitioning rules and partitioning rules include:

[0098] The event time is broken down into multiple time dimensions according to granularity;

[0099] Compare the amount of data corresponding to each time dimension with the preset threshold value;

[0100] The time dimension whose corresponding data volume is less than and closest to the threshold value is determined as the table partitioning dimension;

[0101] The next level of the aforementioned table partitioning dimension is used as the partitioning dimension.

[0102] Preferably, the data storage module further includes:

[0103] The resource naming unit is used to name the stored dataset according to a preset data resource naming rule;

[0104] The naming review result acquisition unit is used to acquire the review results of the reviewers on the naming of the resources;

[0105] The data upload unit is used to upload the dataset and corresponding resource names to the designated database when the review result is approved, thereby completing the data access.

[0106] Preferred options also include:

[0107] The sample data scanning module is used to randomly select a number of push data as sample data from the dataset and scan each sample data to see if there is corresponding event time information.

[0108] The first push termination module is used to terminate the push process of the dataset and generate an error message when it is detected that the corresponding event time information is not found in the sample data.

[0109] Preferred options also include:

[0110] The field scanning module is used to scan the field null value rate and / or format logic error rate of the sample data when it is detected that the corresponding event time information exists in all sample data.

[0111] The second push termination module is used to terminate the push process of the dataset when the field null value rate and / or format logic error rate exceed the first threshold, and generate an error message to be fed back to the data pusher.

[0112] Preferably, the preset standard format includes any one of the following: csv, xls, json, xml, and raf.

[0113] Preferably, the registered resource determination module also includes:

[0114] A new resource type acquisition unit is used to acquire the new resource type created by the configuration personnel when the pushed data cannot be associated with the registered resource type. The new resource type can be associated with the pushed data.

[0115] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] This application also provides an electronic device, including:

[0117] At least one memory and at least one processor;

[0118] The memory is used to store one or more programs;

[0119] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the dynamic data access method as described above.

[0120] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0121] This application also provides a computer-readable storage medium.

[0122] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the dynamic data access method described above.

[0123] Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0124] This invention provides a method, apparatus, electronic device, and storage medium for accessing dynamic data. Addressing the data access process for massive amounts of dynamic data, it primarily offers a method to solve the problem of rapid data entry into the database due to the huge volume of real-time updates. This invention utilizes resource type registration to unify the resource types of data from multiple sources with the same structure but inconsistent naming. Compared to traditional technologies, it emphasizes the time-centric nature of dynamic data, providing a set of automatic table partitioning rules. This eliminates the need for frequent manual observation, pre-analysis, and rule formulation for different resource types during the data entry process. Instead, the system automatically applies time-based partitioning rules to achieve rapid data segmentation and entry. This avoids the significant time spent manually formulating partitioning rules and facilitates rapid data location for users in data mining operations based on time attributes and unified resource types.

[0125] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for accessing dynamic data, characterized in that, Includes the following steps: Obtain the dataset pushed by the data pusher, wherein the dataset consists of multiple push data entries, and each push data entry corresponds to an event time information entry; Randomly select several push data from the dataset as sample data, and scan each sample data to see if there is corresponding event time information; When it is detected that the corresponding event time information is not found in the sample data, the push process of the dataset is stopped, an error message is generated, and feedback is sent to the data pusher. Determine whether the format of each push data item is a preset standard format; For the standard format push data, scan all the field names in the push data table to determine whether the push data can be associated with a registered resource type; When the pushed data can be associated with a registered resource type, the pushed data is partitioned according to preset table partitioning and partitioning rules, and the corresponding split table name and partition block name are obtained according to the associated resource type; wherein, the table partitioning and partitioning rules include: The event time is broken down into multiple time dimensions according to granularity; Compare the amount of data corresponding to each time dimension with the preset threshold value; The time dimension whose corresponding data volume is less than and closest to the threshold value is determined as the table partitioning dimension; The next level below the aforementioned table partitioning dimension is used as the partitioning dimension; Each piece of push data in the dataset is stored according to the split table name and partition block name to complete the data access.

2. The method for accessing dynamic data according to claim 1, characterized in that, After storing each piece of push data in the dataset according to the split table name and partition block name, the method further includes the following steps: The stored dataset is named according to the preset data resource naming rules; Obtain the review results from the reviewers regarding the naming of the resources; If the review result is approved, the dataset and corresponding resource names will be uploaded to the designated database to complete the data access.

3. The method for accessing dynamic data according to claim 1, characterized in that, It also includes the following steps: When it is detected that there is corresponding event time information in all sample data, scan the field null value rate and / or format logic error rate of the sample data; When the field null value rate and / or format logic error rate exceed the first threshold, the data push process is terminated, and an error message is generated and fed back to the data pusher.

4. The method for accessing dynamic data according to claim 1, characterized in that: The default standard formats include any of the following: csv, xls, json, xml, and raf.

5. The method for accessing dynamic data according to claim 1, characterized in that, It also includes the following steps: When the pushed data cannot be associated with a registered resource type, the newly created resource type of the configuration personnel is obtained, and the newly created resource type can be associated with the pushed data.

6. A dynamic data access device, characterized in that, include: The dataset acquisition module is used to acquire the dataset pushed by the data pusher. The dataset consists of multiple push data, and each push data corresponds to an event time information. The sample data scanning module is used to randomly select a number of push data as sample data from the dataset and scan each sample data to see if there is corresponding event time information. The first push termination module is used to terminate the push process of the dataset and generate an error message when it is detected that the corresponding event time information is not found in the sample data; The format determination module is used to determine whether the format of each piece of pushed data is a preset standard format; The registered resource determination module is used to scan the full table field names of the push data in the standard format and determine whether the push data can be associated with a registered resource type. The table partitioning module is used to partition the push data according to preset table partitioning and partitioning rules when the push data can be associated with a registered resource type, and to obtain the split table name and partition block name corresponding to the push data based on the associated resource type; wherein, the table partitioning and partitioning rules include: The event time is broken down into multiple time dimensions according to granularity; Compare the amount of data corresponding to each time dimension with the preset threshold value; The time dimension whose corresponding data volume is less than and closest to the threshold value is determined as the table partitioning dimension; The next level below the aforementioned table partitioning dimension is used as the partitioning dimension; The data storage module is used to store each piece of push data in the dataset according to the split table name and partition block name, thereby completing the data access.

7. An electronic device, characterized in that, include: At least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a dynamic data access method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a dynamic data access method as described in any one of claims 1-5.

Images