Data processing method and device based on partition

Abstract

The present application provides a partition-based data processing method and device, which relates to the technical field of data warehouses. The method includes obtaining data to be stored, determining partition information of a target underlying table structure based on feature data of the data to be stored; wherein the feature data is used to characterize the creation features to which the data to be stored belongs; in response to a data query request, performing query condition preprocessing on the query conditions to obtain feature data to be queried; matching the corresponding target partition in the target underlying table structure according to the feature data to be queried, and querying the target data information matching the query condition in the target partition through the feature data to be queried. The present application greatly improves the data retrieval efficiency by improving the setting method of the underlying table structure and performing data retrieval through partitioning.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouses, and in particular to a partition-based data processing method and device. Background Art

[0002] For the massive amount of information collected from social media, news websites and other platforms, it is usually stored in the data warehouse of the information data platform. The underlying table structure is an important part of the data warehouse, which is related to the performance and stability of the data warehouse. The underlying table structure has a certain impact on the data storage, indexing, retrieval and query of the data warehouse. The existing underlying table structure is usually based on the user's retrieval needs and retrieved by the primary key ID. This method often consumes a lot of information for scenarios with large data volumes, and the retrieval efficiency is low. Summary of the invention

[0003] The purpose of this application is to provide a partition-based data processing method and device to alleviate the above-mentioned technical problems existing in the prior art.

[0004] In a first aspect, the present invention provides a partition-based data processing method, comprising:

[0005] Acquire the data to be stored, and determine the partition information of the target underlying table structure based on the characteristic data of the data to be stored; wherein the characteristic data is used to characterize the creation characteristics to which the data to be stored belongs;

[0006] In response to the data query request, query condition preprocessing is performed on the query condition to obtain the feature data to be queried;

[0007] The corresponding target partition is matched in the target underlying table structure according to the feature data to be queried, and the target data information matching the query condition is queried in the target partition through the feature data to be queried.

[0008] In an optional implementation, determining the partition information of the target underlying table structure based on the characteristic data of the data to be stored includes:

[0009] Processing the data to be stored to obtain time-related information of the data to be stored and a first cryptographic hash function characteristic value;

[0010] A data uniqueness check is performed based on the first cryptographic hash function characteristic value. If the check passes, a partition field is created based on time-related information of the data to be stored, so as to determine partition information of the target underlying table structure based on the partition field.

[0011] In an optional implementation, the data to be stored includes article data; processing the data to be stored to obtain time-related information and a first cryptographic hash function characteristic value of the data to be stored includes:

[0012] Extracting a first data attribute of the article data to determine the creation time of the article data;

[0013] The creation time of the article data is processed in a standard format to obtain time-related information;

[0014] A first cryptographic hash function characteristic value is generated based on the article data.

[0015] In an optional implementation, the time-related information includes the creation time of the data to be stored;

[0016] Create partition fields based on the time-related information of the data to be stored, and determine the partition information of the target underlying table structure based on the partition fields, including:

[0017] The creation time is divided into time granularities according to the hotspot frequency of the feature data to be queried and the data volume of the data to be stored;

[0018] Determine the time division period corresponding to the partition field based on the time granularity, and determine multiple partition fields according to the time division period;

[0019] Determine the partition information of the target underlying table structure based on the partition fields.

[0020] In an optional embodiment, the method further comprises:

[0021] Extracting a second data attribute from the article data to determine the text content of the article data;

[0022] The text content is segmented according to the preset segmentation rules, and the segmented data is organized into key-value pairs according to the JSON format requirements to generate a JSON object;

[0023] Store JSON objects according to partition information.

[0024] In an optional implementation, in response to a data query request, query condition preprocessing is performed on the query condition to obtain feature data to be queried, including:

[0025] In response to the data query request, extract the query information carried in the data query request; the query information includes at least one or more of user information, keyword information, query content information, and query time;

[0026] A second cryptographic hash function characteristic value and an ID primary key are determined based on the query information.

[0027] In an optional implementation, matching a corresponding target partition in a target underlying table structure according to the feature data to be queried, and searching the target partition for target data information matching the query condition through the feature data to be queried, including:

[0028] Match the corresponding target partition in the target underlying table structure according to the query time and the preset query period range;

[0029] The target data information matching the query condition is searched in the target partition by using the second cryptographic hash function characteristic value and the ID primary key.

[0030] In a second aspect, the present invention provides a partition-based data processing device, comprising:

[0031] A partition information determination module is used to obtain data to be stored and determine partition information of a target underlying table structure based on characteristic data of the data to be stored; wherein the characteristic data is used to characterize creation characteristics to which the data to be stored belongs;

[0032] A query condition processing module, used to perform query condition preprocessing on the query condition in response to the data query request to obtain the feature data to be queried;

[0033] The data matching module is used to match the corresponding target partition in the target underlying table structure according to the feature data to be queried, and to query the target data information matching the query condition in the target partition through the feature data to be queried.

[0034] In a third aspect, the present invention provides an electronic device including a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the partition-based data processing method of any one of the aforementioned implementations.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the partition-based data processing method of any one of the aforementioned implementations.

[0036] The partition-based data processing method and device provided by the present application have the following beneficial effects:

[0037] The data is partitioned by time information and stored according to the partitions. For large amounts of data, effective and fast data storage is possible. When querying data, the partition information is pre-locked and data is retrieved through the partitions, which greatly improves data retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flowchart of a partition-based data processing method provided in an embodiment of the present application;

[0040] Figure 2 A structural diagram of a partition-based data processing device provided in an embodiment of the present application;

[0041] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] The present application embodiment provides a partition-based data processing method, see Figure 1 As shown, the method can be applied to a database management system, and the method mainly includes the following steps:

[0046] Step S110, obtaining the data to be stored, and determining the partition information of the target underlying table structure based on the characteristic data of the data to be stored; wherein the characteristic data is used to characterize the creation characteristics to which the data to be stored belongs.

[0047] In one embodiment, the data to be stored refers to data in the application scenario of the present application solution, such as when applied to the storage and query of some text data, the data to be stored can be the original data corresponding to the text data. For example, when it is necessary to store some article data and query it after storage, the data to be stored is news articles or social software articles.

[0048] The creation characteristics of the characteristic data of the above-mentioned data to be stored refer to a series of attributes corresponding to when the data to be stored is created, such as the creation user, creation time, creation location (such as IP address), etc.

[0049] In one embodiment, the characteristic data of the data to be stored may include an MD5 value and a creation time of the data to be stored.

[0050] The underlying table structure is a very important part of the data warehouse, which directly affects the performance and stability of the data warehouse. The target underlying table structure involved in the embodiment of the present application is an improved table structure. In order to ensure the performance of data storage, index design, query performance and data quality, the present application partitions it according to the above-mentioned feature data, and the data to be stored can be stored in the corresponding partitions, so as to improve the efficient storage and access of data in the data warehouse through the target underlying table structure.

[0051] Step S120, in response to the data query request, preprocessing the query conditions to obtain the feature data to be queried.

[0052] The above-mentioned characteristic data to be queried may include the MD5 value corresponding to the query condition, the time in the query content, and the ID primary key.

[0053] After the data is partitioned and stored, when the stored data needs to be queried, preprocessing can be performed in advance according to the query conditions. In one embodiment, the preprocessing means first filtering by time (the time is not carried in the query condition, but a pre-set time range), and then filtering according to the user's specific conditions.

[0054] Step S130, matching the corresponding target partition in the target underlying table structure according to the feature data to be queried, and searching the target partition for target data information matching the query condition through the feature data to be queried.

[0055] In one embodiment, after the aforementioned feature data to be queried is determined, the time partition can be determined in advance according to a preset time range. After the time partition is determined, the corresponding data is queried in the partition according to the MD5 value and ID primary key corresponding to the feature data to be queried, thereby improving the query efficiency.

[0056] For ease of understanding, the partition-based data processing method provided in the embodiment of the present application is described in detail below.

[0057] In one implementation, the above-mentioned determination of the partition information of the target underlying table structure based on the characteristic data of the data to be stored may include the following steps 1.1 and 1.2 during specific implementation:

[0058] Step 1.1, processing the data to be stored to obtain time-related information and a first cryptographic hash function characteristic value of the data to be stored; the time-related information refers to time-related information in the data to be stored, such as the creation time of the data to be stored, the download time (crawling time) of the data to be stored, etc. In one example, when the data to be stored is web page information, the time-related information may also be the time when the web page content is published, the time when it is edited, the time when it is most recently edited, etc.

[0059] Step 1.2, performing data uniqueness verification based on the first cryptographic hash function characteristic value, if the verification passes, creating a partition field based on the time-related information of the data to be stored, so as to determine the partition information of the target underlying table structure based on the partition field.

[0060] In an application scenario, the above method provided in the embodiment of the present application can be applied to the partition storage and post-storage query of article data (such as news articles or social platform articles). Therefore, in one embodiment, the above data to be stored may include article data. The above processing of the data to be stored to obtain the time-related information and the first cryptographic hash function characteristic value of the data to be stored may include the following steps 2.1 to 2.3 in the specific implementation:

[0061] Step 2.1, extract the first data attribute of the article data to determine the creation time of the article data.

[0062] Step 2.2, format the creation time of the article data in a standard way to obtain time-related information.

[0063] Step 2.3, generating a first cryptographic hash function characteristic value based on the article data.

[0064] Further, when the time-related information is the creation time of the data to be stored, the above-mentioned creation of the partition field by the time-related information of the data to be stored to determine the partition information of the target underlying table structure based on the partition field may include the following steps 3.1 to 3.3 in specific implementation:

[0065] Step 3.1, divide the creation time into time granularities according to the hotspot frequency of the feature data to be queried and the data volume of the data to be stored;

[0066] Step 3.2, determining the time division period corresponding to the partition field based on the time granularity, and determining multiple partition fields according to the time division period;

[0067] Step 3.3, determine the partition information of the target underlying table structure based on the partition field.

[0068] Furthermore, in order to ensure the format requirements during data processing, the above method also includes:

[0069] Step 4.1, extracting the second data attribute of the article data to determine the text content of the article data;

[0070] Step 4.2, segment the text content according to the preset segmentation rules, organize the segmented data into key-value pairs according to the JSON format requirements, and generate a JSON object;

[0071] Step 4.3, store the JSON object according to the partition information. The JSON object here is the article data after format conversion.

[0072] In an optional implementation, in response to the data query request, query condition preprocessing is performed on the query condition to obtain the feature data to be queried. In a specific implementation, the following steps 5.1 and 5.2 may be included:

[0073] Step 5.1, in response to the data query request, extracting the query information carried in the data query request; the query information at least includes one or more of user information, keyword information, query content information, and query time;

[0074] Step 5.2: Determine the second cryptographic hash function characteristic value and the ID primary key based on the query information.

[0075] Furthermore, the above-mentioned matching of the corresponding target partition in the target underlying table structure according to the feature data to be queried, and searching the target data information matching the query condition in the target partition through the feature data to be queried, in specific implementation, may include the following steps 6.1 and 6.2:

[0076] Step 6.1, matching the corresponding target partition in the target underlying table structure according to the query time and the preset query period range;

[0077] Step 6.2: Search the target partition for target data information matching the query condition by using the second cryptographic hash function characteristic value and the ID primary key.

[0078] In summary, the present application obtains the data to be stored, pre-creates partitions for the data to be stored by using MD5 and the creation time of the data to be stored, and stores the data to be stored in each partition according to the data creation time; when querying, first filter by time according to the query conditions, and then filter by the user's specific conditions (if the user does not provide a time, the data is queried according to the default time), and then determine the partition according to the MD5 corresponding to the query conditions, the time in the query content, and the ID primary key, and then query from the partition according to the MD5 and ID primary key, thereby improving the query speed and query accuracy.

[0079] Based on the above method embodiment, the present application embodiment also provides a partition-based data processing device, see Figure 2 As shown, the device mainly includes the following parts:

[0080] The partition information determination module 210 is used to obtain the data to be stored and determine the partition information of the target underlying table structure based on the characteristic data of the data to be stored; wherein the characteristic data is used to characterize the creation characteristics to which the data to be stored belongs;

[0081] The query condition processing module 220 is used to perform query condition preprocessing on the query condition in response to the data query request to obtain the feature data to be queried;

[0082] The data matching module 230 is used to match the corresponding target partition in the target underlying table structure according to the feature data to be queried, and to search the target partition for target data information matching the query condition through the feature data to be queried.

[0083] The partition-based data processing device provided in the embodiment of the present application partitions the data according to the time information, and stores the data according to the partitions during data storage. For situations where the amount of data is large, effective and fast data storage can be performed. When querying data, the partition information is pre-locked, and then data is retrieved through the partitions, which greatly improves the data retrieval efficiency.

[0084] In a feasible implementation manner, the partition information determination module 210 is used to:

[0085] Processing the data to be stored to obtain time-related information of the data to be stored and a first cryptographic hash function characteristic value;

[0086] A data uniqueness check is performed based on the first cryptographic hash function characteristic value. If the check passes, a partition field is created based on time-related information of the data to be stored, so as to determine partition information of the target underlying table structure based on the partition field.

[0087] In a feasible implementation manner, the data to be stored includes article data; the partition information determination module 210 is used to:

[0088] Extracting a first data attribute of the article data to determine the creation time of the article data;

[0089] The creation time of the article data is processed in a standard format to obtain time-related information;

[0090] A first cryptographic hash function characteristic value is generated based on the article data.

[0091] In a feasible implementation manner, the time-related information includes the creation time of the data to be stored;

[0092] The partition information determination module 210 is used to:

[0093] The creation time is divided into time granularities according to the hotspot frequency of the feature data to be queried and the data volume of the data to be stored;

[0094] Determine the time division period corresponding to the partition field based on the time granularity, and determine multiple partition fields according to the time division period;

[0095] Determine the partition information of the target underlying table structure based on the partition fields.

[0096] In a feasible implementation manner, the above device further includes: a data format conversion and storage module, which is used to:

[0097] Extracting a second data attribute from the article data to determine the text content of the article data;

[0098] The text content is segmented according to the preset segmentation rules, and the segmented data is organized into key-value pairs according to the JSON format requirements to generate a JSON object;

[0099] Store JSON objects according to partition information.

[0100] In a feasible implementation manner, the query condition processing module 220 is used to:

[0101] In response to the data query request, extract the query information carried in the data query request; the query information includes at least one or more of user information, keyword information, query content information, and query time;

[0102] A second cryptographic hash function characteristic value and an ID primary key are determined based on the query information.

[0103] In a feasible implementation manner, the data matching module 230 is used to:

[0104] Match the corresponding target partition in the target underlying table structure according to the query time and the preset query period range;

[0105] The target data information matching the query condition is searched in the target partition by using the second cryptographic hash function characteristic value and the ID primary key.

[0106] The partition-based data processing device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for parts not mentioned in the embodiment of the partition-based data processing device, reference may be made to the corresponding contents in the aforementioned partition-based data processing method embodiment.

[0107] The present application also provides an electronic device, such as Figure 3 As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device 100 includes a processor 31 and a memory 30, the memory 30 stores computer executable instructions that can be executed by the processor 31, and the processor 31 executes the computer executable instructions to implement any of the above-mentioned partition-based data processing methods.

[0108] exist Figure 3 In the illustrated embodiment, the electronic device further includes a bus 32 and a communication interface 33 , wherein the processor 31 , the communication interface 33 and the memory 30 are connected via the bus 32 .

[0109] Among them, the memory 30 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 33 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 32 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 32 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] The processor 31 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 31. The above processor 31 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 31 reads the information in the memory and completes the steps of the partition-based data processing method of the above embodiment in combination with its hardware.

[0111] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned partition-based data processing method. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.

[0112] The computer program product of the partition-based data processing method and device provided in the embodiments of the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0113] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0114] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0115] In the description of the present application, it should be noted that the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A partition-based data processing method, characterized in that: include: Acquire the data to be stored, and determine the partition information of the target underlying table structure based on the characteristic data of the data to be stored; wherein the characteristic data is used to characterize the creation characteristics to which the data to be stored belongs; In response to the data query request, query condition preprocessing is performed on the query condition to obtain the feature data to be queried; Matching a corresponding target partition in the target underlying table structure according to the feature data to be queried, and searching the target partition for target data information matching the query condition through the feature data to be queried; The data to be stored includes article data; processing the data to be stored to obtain time-related information and a first cryptographic hash function characteristic value of the data to be stored, including: extracting a first data attribute from the article data to determine the creation time of the article data; processing the creation time of the article data in a format standard to obtain the time-related information; generating a first cryptographic hash function characteristic value based on the article data; The time-related information includes the creation time of the data to be stored; creating a partition field through the time-related information of the data to be stored, so as to determine the partition information of the target underlying table structure based on the partition field, including: dividing the creation time into time granularities according to the hot spot frequency of the feature data to be queried and the data volume of the data to be stored; determining the time division period corresponding to the partition field based on the time granularity, and determining multiple partition fields according to the time division period; determining the partition information of the target underlying table structure based on the partition field; Extracting a second data attribute from the article data to determine the text content of the article data; segmenting the text content according to a preset segmentation rule, organizing the segmented data into key-value pairs according to JSON format requirements, and generating a JSON object; storing the JSON object according to the partition information; In response to the data query request, query condition preprocessing is performed on the query condition to obtain the feature data to be queried, including: in response to the data query request, extracting the query information carried in the data query request; the query information at least includes one or more of user information, keyword information, query content information, and query time; determining the second cryptographic hash function feature value and the ID primary key based on the query information; Matching the corresponding target partition in the target underlying table structure according to the characteristic data to be queried, and querying the target data information matching the query condition in the target partition through the characteristic data to be queried, including: matching the corresponding target partition in the target underlying table structure according to the query time and a preset query time period range; querying the target data information matching the query condition in the target partition through the second cryptographic hash function characteristic value and the ID primary key.

2. The partition-based data processing method according to claim 1, characterized in that: Determining partition information of a target underlying table structure based on the characteristic data of the data to be stored includes: Processing the data to be stored to obtain time-related information and a first cryptographic hash function characteristic value of the data to be stored; A data uniqueness check is performed based on the first cryptographic hash function characteristic value. If the check passes, a partition field is created using the time-related information of the data to be stored, so as to determine partition information of the target underlying table structure based on the partition field.

3. A partition-based data processing device, characterized in that: For implementing the method according to claim 1, the device comprises: A partition information determination module, used to obtain data to be stored, and determine partition information of a target underlying table structure based on characteristic data of the data to be stored; wherein the characteristic data is used to characterize creation characteristics to which the data to be stored belongs; A query condition processing module, used to perform query condition preprocessing on the query condition in response to the data query request to obtain the feature data to be queried; The data matching module is used to match the corresponding target partition in the target underlying table structure according to the feature data to be queried, and to query the target data information matching the query condition in the target partition through the feature data to be queried.

4. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the partition-based data processing method according to claim 1 or 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the partition-based data processing method according to claim 1 or 2.

Images