Data conversion method and system

Abstract

The invention relates to the field of data processing, in particular to a data conversion method and system, and the method comprises the steps: obtaining a plurality of custom tags and data updating frequencies; according to the custom tag, screening corresponding data from original data of a source database table as source data; and based on source data, according to the data updating frequency, mapping to generate target data, and storing the target data. According to the method, the dependency on a source database table structure is reduced, the query efficiency of the user-defined tag data is improved, and a big data platform can process user tags more quickly and more effectively, so that the user experience and the service efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data conversion method and system. Background Art

[0002] A tag is a short word or phrase used to describe and classify things. In the Internet field, tags are often used to classify and describe users, products, content, etc., so as to make personalized recommendations and precision marketing. Taking user tags as an example, e-commerce platforms can label users with "post-80s", "female", "sports enthusiasts" and other tags based on their age, gender, purchase history and other information. These tags can help the platform better understand user characteristics and provide services that better meet user needs, such as customer relationship management, personalized recommendations, precision marketing, etc. Through user tags, companies can better understand users and provide products and services that better meet user needs, thereby improving user satisfaction and corporate profits.

[0003] However, for platforms with large amounts of data, existing tag management technologies have some problems. A common approach is to create a table in the database and store tags as fields in the table. For example, fields such as age, gender, and shopping preferences can all be used as user tags. However, this approach has problems with expansion and query efficiency. When a new tag needs to be added, the database table structure needs to be modified, which is a time-consuming operation that may affect system stability. At the same time, when the number of fields in the table increases, the speed of reading specific fields will drop significantly, which is unacceptable for application scenarios that require real-time or fast response. Summary of the invention

[0004] To solve the above problems, the present invention provides a data conversion method and system.

[0005] The first aspect of the present invention discloses a data conversion method, comprising:

[0006] Get several custom tags and data update frequency;

[0007] According to the custom tag, corresponding data is filtered from the original data in the source database table as source data;

[0008] Based on the source data and according to the data update frequency, the target data is mapped and generated and saved.

[0009] Furthermore, the data conversion method further includes: storing the target data in a target database table, wherein the source database table is a HIVE database table and the target database table is a Doris database table.

[0010] Furthermore, the step of selecting corresponding data from the original data in the source database table as source data according to the custom tag includes:

[0011] Acquire a specified field of the original data according to the custom tag;

[0012] Based on the Quartz scheduler, the data of the designated field and the identifier data are regularly filtered from the original data as the source data.

[0013] Furthermore, the step of mapping and generating target data based on the source data and according to the data update frequency and saving the target data comprises:

[0014] According to the data update frequency, the custom label and the identifier of the source data are added to the corresponding fields of the Bitmap table of the Doris database table;

[0015] Based on the Quartz scheduler, the source data is mapped and loaded into the Bitmap table of the Doris database table through Broker Load to generate the target data.

[0016] Furthermore, the scheduled tasks of the Quartz scheduler are configured through SimpleScheduleBuilder or CronScheduleBuilder.

[0017] Further, the step of adding the custom tag and the identifier of the source data to the corresponding field of the Bitmap table of the Doris database table according to the data update frequency includes:

[0018] When the data update frequency does not exceed the preset threshold:

[0019] Extracting an identifier from each piece of the source data and generating multiple primary keys;

[0020] Add the plurality of primary keys and the custom tags to corresponding fields of the Bitmap table.

[0021] Furthermore, the step of mapping and loading the source data into the Bitmap table of the Doris database table through Broker Load based on the Quartz scheduler to generate the target data includes:

[0022] Analyze each field data of the source data, and filter out the corresponding source data according to the custom label;

[0023] The filtered source data is loaded into the corresponding field of the Bitmap table according to the primary key to generate the target data.

[0024] Further, the step of adding the custom tag and the identifier of the source data to the corresponding field of the Bitmap table of the Doris database table according to the data update frequency includes:

[0025] When the data update frequency exceeds the preset threshold:

[0026] According to the custom tag, multiple primary keys are generated as corresponding fields of the Bitmap table.

[0027] Furthermore, the step of mapping and loading the source data into the Bitmap table of the Doris database table through Broker Load based on the Quartz scheduler to generate the target data includes:

[0028] According to the primary key, the source data is screened based on the field data of the source data;

[0029] Combining the identifier data of the source data corresponding to each of the primary keys into an identifier data set;

[0030] Multiple identifier data sets are bitmap encoded and loaded into the field content of the corresponding field of the Bitmap table to generate the target data.

[0031] A second aspect of the present invention discloses a data conversion system, comprising:

[0032] The acquisition module is used to obtain several custom tags and data update frequencies;

[0033] A screening module, used to screen corresponding data from the original data in the source database table as source data according to the custom tag;

[0034] The mapping module is used to map and generate target data based on source data and according to the data update frequency, and save the target data.

[0035] The present invention proposes a data conversion method, which stores the data of custom tags in a database table different from the original data. By storing the data of custom tags separately from the basic data, the dependence on the source database table structure is reduced and the query efficiency of custom tag data is improved. When querying a specific tag, since the tag data is stored in an independent table, the query efficiency is higher. This method improves the flexibility and efficiency of tag management, allowing the big data platform to process user tags more quickly and effectively, thereby improving user experience and business efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a flowchart of a data conversion method disclosed in an embodiment of the present invention;

[0038] Figure 2 It is a structural schematic diagram of a data conversion system disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, or product end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] With the rapid development of Internet technology and big data, many platforms, such as online shopping platforms, have accumulated massive amounts of user data. These data are of immeasurable value for analyzing user behavior, optimizing services and promoting products. However, on the one hand, due to the need for data security and privacy protection, these data cannot be frequently accessed and processed. On the other hand, due to the huge amount of data, frequent data processing consumes a lot of computing resources and time, affecting the efficiency of the system. Therefore, how to efficiently and accurately generate custom tags based on these user data while ensuring data security and efficiency has become an important and challenging problem. The following is based on the original data stored in the Hive database and the custom tag data stored in the Doris library to specifically explain the specific steps of the data conversion method provided by the present invention. It can be understood by those skilled in the art that when the amount of original data is small, the security requirements of the data are not high, or the custom tag update frequency is low, the original data and the custom tag data can also be stored in different database tables of the same database.

[0043] See also Figure 1 As shown, Figure 1 FIG. 1 is a flow chart of a data conversion method disclosed in an embodiment of the present invention. Figure 1 As shown, the data conversion method may include the following operations:

[0044] S101, obtaining a number of custom tags and data update frequencies;

[0045] In this optional embodiment, the custom label may be created by a platform maintenance personnel, or may be created by a terminal user based on a label custom page provided by the platform, and the embodiment of the present invention does not impose any limitation thereto.

[0046] An example of using a label customization page is as follows:

[0047] After the end user passes the identity authentication, he / she enters the label customization page, which displays the corresponding functional modules according to the user's permissions.

[0048] The label customization page provides a friendly user interface to display the table structure in the Hive library to the end user. The end user can select any required field in the Hive library through the drop-down menu or input box in the label customization page.

[0049] End users can set the name, description, and other properties of custom tags according to their needs. The tag customization page provides input boxes and options for tag properties. End users can describe the purpose and definition of custom tags in detail. End users can set a unique name for custom tags and add description information to help other end users understand the meaning and purpose of the custom tag.

[0050] The label customization page verifies the accuracy of custom labels in real time and provides a preview function. End users can view the preview results of custom labels to ensure that the custom label logic is correct. The label customization page will perform syntax and logic checks on the entered SQL or selected fields to prompt end users of possible errors. The preview function allows end users to view the application effect of custom labels before saving them to ensure the correctness of custom label definitions.

[0051] S102, according to the custom tag, selecting corresponding data from the original data in the source database table as source data;

[0052] In an optional embodiment, the step of selecting corresponding data from the original data in the source database table as source data according to the custom tag includes:

[0053] Acquire a specified field of the original data according to the custom tag;

[0054] Based on the Quartz scheduler, the data of the designated field and the identifier data are regularly filtered from the original data as the source data.

[0055] In this optional embodiment, each custom label corresponds to data in one or more fields of the original data in the source database table. For example, the custom label "age group" corresponds to the "age" field of the source database table. For reasons that can be understood by those skilled in the art, the custom label "age group" also corresponds to an identifier field of the source database table, such as the "user ID" field, to identify a user corresponding to the "age group" data.

[0056] The Quartz scheduler is a component for managing and scheduling tasks, which is configured and initialized by SchedulerFactory. It is responsible for managing all tasks and triggers, and scheduling the execution of tasks according to the definition of the trigger. The working principle of the scheduler is to start or stop the tasks associated with it according to the preset trigger schedule. In this optional embodiment, the associated task is to filter the data and identifier data of the specified field from the original data as the source data. It can be understood by those skilled in the art that the associated task ultimately needs to map the source data to generate the target data, which will be introduced in the subsequent section.

[0057] S103: Based on the source data and according to the data update frequency, the target data is mapped and generated, and the target data is saved.

[0058] In an optional embodiment, the data conversion method further includes: storing the target data in a target database table, wherein the source database table is a HIVE database table and the target database table is a Doris database table.

[0059] Hive is a data warehouse infrastructure built on Hadoop. Hive provides an SQL interface for querying and analyzing data stored in Hadoop, called HiveQL. It supports the storage, query, and analysis of structured data, and can convert SQL-type queries into MapReduce tasks for execution. The main goal of Hive is to provide users with big data with convenient, fast, and scalable data analysis tools. Doris is an open source analytical database with a massively parallel processing architecture that supports fast, highly concurrent complex analytical queries. Through column storage and vectorized execution, Doris can quickly return random, real-time OLAP query results, while also providing efficient batch data loading capabilities. The main goal of Doris is to provide high-concurrency, low-latency complex query capabilities for online analysis to meet most data analysis needs.

[0060] In an optional embodiment, the step of mapping and generating target data based on the source data and according to the data update frequency and saving the target data comprises:

[0061] According to the data update frequency, the custom label and the identifier of the source data are added to the corresponding fields of the Bitmap table of the Doris database table;

[0062] Based on the Quartz scheduler, the source data is mapped and loaded into the Bitmap table of the Doris database table through Broker Load to generate the target data.

[0063] In this optional embodiment, Broker Load is a tool for loading data in the Doris database. It supports loading data from multiple data sources such as Hadoop HDFS, Apache Hadoop, AWS S3, etc. into the Doris database. Broker Load can handle the loading of large amounts of data, is suitable for importing batch data, and supports multiple data formats, such as CSV, Parquet, etc. Broker Load processes data in parallel in a distributed manner to improve the efficiency of data processing. The Bitmap table in the Doris database is a special type of table that uses the Bitmap data type to store data. The Bitmap data type is mainly used to store and query large amounts of discrete data. In the Bitmap table, each column usually stores a Bitmap, and each Bitmap corresponds to a specific value. This structure can effectively store and query large-scale multi-value discrete data, such as user behavior data. The Bitmap table can greatly improve the efficiency of data queries, especially for queries involving large amounts of discrete data.

[0064] It can be seen that this optional embodiment can automatically perform data conversion and loading tasks at a fixed time without manual operation, which greatly improves the efficiency of data processing. In addition, by adding custom tags and identifiers of source data to the corresponding fields of Doris's Bitmap table, this can better organize and manage target data and improve the efficiency of data query.

[0065] In another optional embodiment, the scheduled task of the Quartz scheduler is configured through SimpleScheduleBuilder or CronScheduleBuilder.

[0066] In this optional embodiment, for tasks that require complex scheduling strategies to be set according to Cron expressions, the scheduled tasks of the Quartz scheduler can be configured through CronScheduleBuilder; for simple, one-time or repeated tasks at specified intervals, the scheduled tasks of the Quartz scheduler can be configured through SimpleScheduleBuilder.

[0067] The following explains in detail how to import the source data through the Quartz scheduler

[0068] 1). Implement the Quartz Job interface as the data mapping task execution unit to map the source data to generate the target data and store it in the Doris database table. Implement the following functions in the instance of the Job interface:

[0069] Use HDFS API or other file reading tools to read the source data; after reading the source data, map the data in the source data to the corresponding fields in the Doris table according to the custom tags, and perform necessary data type conversion to obtain the target data.

[0070] Ensure that the target data complies with the expected format and scope to avoid erroneous data affecting subsequent queries and analysis;

[0071] Based on Doris' Broker Load, the target data is imported into the Doris database table;

[0072] 2). Generate task trigger units based on Quartz's SimpleScheduleBuilder or CronScheduleBuilder;

[0073] 3). Create a scheduler based on Doris SchedulerFactory;

[0074] 4) Registering the data mapping task execution unit and the task triggering unit to the scheduler;

[0075] 5). Start the scheduler to call the data mapping task execution unit according to the task triggering unit.

[0076] In another optional embodiment, the step of adding the custom tag and the identifier of the source data to the corresponding field of the Bitmap table of the Doris database table according to the data update frequency includes:

[0077] When the data update frequency does not exceed the preset threshold:

[0078] Extracting an identifier from each piece of the source data and generating multiple primary keys;

[0079] Add the plurality of primary keys and the custom tags to corresponding fields of the Bitmap table.

[0080] In this optional embodiment, the identifier is a field in the data used to uniquely identify each record. In a database, each record usually has an identifier consisting of one or more fields that uniquely identify the record. The identifier can be any data that can uniquely identify a record, such as a user ID, order number, etc.

[0081] The primary key is a field or combination of fields used to uniquely identify each record in a database table. Each table should have a primary key, and the value of the primary key must be unique in the table, without duplication or null values. In a relational database, the primary key is usually a field, such as a user ID, order number, etc. However, in some cases, a table may use a composite primary key composed of multiple fields. In this optional embodiment, the identifier of each source data is used as the primary key of a database data related to a custom tag.

[0082] In a further optional embodiment, the step of mapping and loading the source data into the Bitmap table of the Doris database table through Broker Load based on the Quartz scheduler to generate the target data includes:

[0083] Analyze each field data of the source data, and filter out the corresponding source data according to the custom label;

[0084] The filtered source data is loaded into the corresponding field of the Bitmap table according to the primary key to generate the target data.

[0085] In this optional embodiment, analyzing each field data of the source data and filtering out the corresponding source data according to the custom tag means filtering each field data of the source data to obtain data corresponding to the custom tag.

[0086] For example: In the source data, the age of user 1 is 15, the age of user 2 is 27, the age of user 3 is 56, and the age of user 4 is 34; the custom labels are "Age range 0-30" and "Age range 31-60". The data corresponding to the custom labels are:

[0087] Primary key: 1, data: age range 0-30;

[0088] Primary key: 2, data: age range 0-30;

[0089] Primary key: 3, data: age range 31-60;

[0090] Primary key: 4, data: age range 31-60.

[0091] In an optional embodiment, the step of adding the custom tag and the identifier of the source data to the corresponding field of the Bitmap table of the Doris database table according to the data update frequency includes:

[0092] When the data update frequency exceeds the preset threshold:

[0093] According to the custom tag, multiple primary keys are generated as corresponding fields of the Bitmap table.

[0094] In this optional embodiment, generating multiple primary keys according to the custom tags means generating a primary key for each custom tag. For example, for the custom tags "age group 0-30", "age group 31-60", and "age group 60+", three primary keys "age group 0-30", "age group 31-60", and "age group 60+" are generated.

[0095] In an optional embodiment, the step of mapping and loading the source data into the Bitmap table of the Doris database table through Broker Load based on the Quartz scheduler to generate the target data includes:

[0096] According to the primary key, the source data is screened based on the field data of the source data;

[0097] Combining the identifier data of the source data corresponding to each of the primary keys into an identifier data set;

[0098] Multiple identifier data sets are bitmap encoded and loaded into the field content of the corresponding field of the Bitmap table to generate the target data.

[0099] In this optional embodiment, filtering the source data according to the primary key and based on the field data of the source data means filtering each source data and corresponding it to a custom label.

[0100] The identifier data of the source data corresponding to each primary key is formed into an identifier data set, specifically, the identifiers in the source data corresponding to each primary key are formed into an identifier data set, and the identifier data set is used as the data corresponding to the primary key.

[0101] For example: In the source data, the age of user 1 is 15, the age of user 2 is 27, the age of user 3 is 56, and the age of user 4 is 34; the custom labels are "Age range 0-30" and "Age range 31-60". The data corresponding to the custom labels are:

[0102] Primary key: age range 0-30, data: 1,2;

[0103] Primary key: age range 31-60, data: 3,4;

[0104] In an optional embodiment, the data forwarding method further provides a task monitoring and management function, and displays the execution status, execution log and execution result of the data import task through a display terminal.

[0105] The present invention also provides a data conversion system, comprising:

[0106] The acquisition module 201 is used to acquire a number of custom tags and data update frequencies;

[0107] A screening module 202 is used to screen corresponding data from the original data in the source database table as source data according to the custom tag;

[0108] The mapping module 203 is used to map and generate target data based on the source data according to the data update frequency, and save the target data.

[0109] For the specific definition of the data conversion system, please refer to the definition of the data conversion method above, which will not be repeated here. Each module in the above data conversion system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the electronic device in hardware format, or can be stored in the memory in the electronic device in software format, so that the processor can call the operations corresponding to the above modules.

[0110] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0111] In summary, the data conversion method and system disclosed in the present invention not only reduces the dependence on the source database table structure, but also improves the query efficiency of custom tag data, so that the big data platform can process user tags more quickly and effectively, thereby improving user experience and business efficiency. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0112] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A data conversion method, characterized in that: The method comprises: Get several custom tags and data update frequency; According to the custom tag, corresponding data is filtered from the original data in the source database table as source data; Based on the source data and according to the data update frequency, the target data is mapped and generated and saved.

2. A data conversion method according to claim 1, characterized in that: The data conversion method further includes: storing the target data in a target database table, wherein the source database table is a HIVE database table and the target database table is a Doris database table.

3. A data conversion method according to claim 2, characterized in that: The step of selecting corresponding data from the original data in the source database table as source data according to the custom tag includes: Acquire a specified field of the original data according to the custom tag; Based on the Quartz scheduler, the data of the designated field and the identifier data are regularly filtered from the original data as the source data.

4. A data conversion method according to claim 2, characterized in that: The step of mapping and generating target data based on the source data and according to the data update frequency and saving the target data comprises: According to the data update frequency, the custom label and the identifier of the source data are added to the corresponding fields of the Bitmap table of the Doris database table; Based on the Quartz scheduler, the source data is mapped and loaded into the Bitmap table of the Doris database table through Broker Load to generate the target data.

5. A data conversion method according to claim 3 or 4, characterized in that: Configure the scheduled tasks of the Quartz scheduler through SimpleScheduleBuilder or CronScheduleBuilder.

6. A data conversion method according to claim 4, characterized in that: The step of adding the custom tag and the identifier of the source data to the corresponding fields of the Bitmap table of the Doris database table according to the data update frequency includes: When the data update frequency does not exceed the preset threshold: Extracting an identifier from each piece of the source data and generating multiple primary keys; Add the plurality of primary keys and the custom tags to corresponding fields of the Bitmap table.

7. A data conversion method according to claim 6, characterized in that: The step of mapping and loading the source data into the Bitmap table of the Doris database table through Broker Load based on the Quartz scheduler to generate the target data includes: Analyze each field data of the source data, and filter out the corresponding source data according to the custom label; The filtered source data is loaded into the corresponding field of the Bitmap table according to the primary key to generate the target data.

8. A data conversion method according to claim 4, characterized in that: The step of adding the custom tag and the identifier of the source data to the corresponding fields of the Bitmap table of the Doris database table according to the data update frequency includes: When the data update frequency exceeds the preset threshold: According to the custom tag, multiple primary keys are generated as corresponding fields of the Bitmap table.

9. A data conversion method according to claim 8, characterized in that: The step of mapping and loading the source data into the Bitmap table of the Doris database table through Broker Load based on the Quartz scheduler to generate the target data includes: According to the primary key, the source data is screened based on the field data of the source data; Combining the identifier data of the source data corresponding to each of the primary keys into an identifier data set; Multiple identifier data sets are bitmap encoded and loaded into the field content of the corresponding field of the Bitmap table to generate the target data.

10. A data conversion system, characterized in that: include: The acquisition module is used to obtain several custom tags and data update frequencies; A screening module, used to screen corresponding data from the original data in the source database table as source data according to the custom tag; The mapping module is used to map and generate target data based on source data and according to the data update frequency, and save the target data.

Images