Label data processing method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, discloses a tag data processing method, device and equipment based on a column database bitmap coded data structure and a medium, and aims to solve the problem of low tag data processing efficiency in the prior art. The method comprises the following steps: customizing a bitmap tag data set corresponding to a tag data demand based on preset structured information tag data, determining a target bitmap tag data set corresponding to the tag data demand according to the bitmap tag data set, and carrying out corresponding data processing on the target bitmap tag data set according to the tag data demand to obtain a target tag data set; according to the method and the device, the tag data processing required by the preset business is met, the corresponding bitmap tag data set is self-defined according to the business requirement, the tag data processing efficiency is improved, and the method and the device can be applied to but not limited to the financial field, the insurance field and the medical health field.

Description

Technical Field

[0001] This invention relates to the fields of data processing technology and financial technology, and in particular to a method, apparatus, device and medium for processing tag data based on a columnar database bitmap encoded data structure. Background Technology

[0002] Data labeling refers to the process of adding labels to raw data, that is, the transformation from unlabeled to labeled data. Data labels are specific annotations or classification identifiers attached to the raw data; they are the labeling results of the data labeling process. For example, if an image is the raw data, the process of labeling an image as "cat" or "dog" is the process of labeling the image, i.e., image labeling. "Cat" or "dog" are the data labels for the image. Labeled data processing (i.e., tag data processing) refers to the process of querying, processing, analyzing, or modeling data that already has labels (such as category labels, annotation labels, etc.). For example, training an image classification model on a dataset of images labeled "cat / dog". Here, labeled data refers to data that has undergone label processing; it is a complete dataset that already contains labels, a collection of "labeled data," i.e., labeled data.

[0003] In big data analytics scenarios, data tagging is a crucial means of enhancing data interpretability and query efficiency through structured metadata. In other words, tagging raw data is essential for achieving accurate queries and analyses. For example, in the financial industry, financial risk management and fraud detection require querying and analyzing financial transaction information, including but not limited to deposits, withdrawals, and loans of relevant financial users. Similarly, in the insurance industry, automating insurance claims and combating fraud requires querying and analyzing insurance information, including but not limited to basic customer information, insurance product information, and claims information. In the healthcare industry, clinical decision support requires querying and analyzing patient information, including but not limited to basic patient information, laboratory test results, electronic medical records, and related patient information. All these data queries and analyses in different scenarios require tagging the raw data first, and then further processing the tagged data for querying and analysis.

[0004] In traditional technologies, different technical solutions can be adopted to implement data tagging. For example, in one example, a relational database table structure is used to store tag data. In this case, tagging data processing generally involves retrieving the data set with the corresponding tags through multi-table join queries. In another example, a columnar database bitmap encoded data structure (such as the ClickHouse Bitmap data structure) is used to store tag data. In this case, tagging data processing generally involves querying and analyzing the corresponding tag data by manipulating a large number of Boolean values ​​or integer sets. Here, ClickHouse is a columnar distributed database management system (DBMS), and the ClickHouse Bitmap data structure is a Bitmap data structure based on ClickHouse.

[0005] However, the inventors realized that traditional technologies for data tagging and the corresponding tag data processing have significant drawbacks: 1) When storing and querying tagged data based on relational database table structures, the efficiency of multi-table join queries is low. As the amount of data increases, query performance drops sharply, making it difficult to meet the real-time requirements of big data analysis scenarios. At the same time, storing and managing tagged data requires a large amount of database space, resulting in high data maintenance costs; 2) When storing and querying tagged data based on columnar database bitmap encoded data structures (such as ClickHouse Bitmap data structures), tagged data is often stored as a conventional data type, i.e., structured data processing is performed using wide tables. This means that data from multiple join tables are pre-merged into one table, making it difficult to meet the needs of rapid processing and flexible analysis of tagged data under large data volumes, and also making it difficult to meet the real-time requirements of big data analysis scenarios.

[0006] Therefore, improving the processing efficiency of tagged data has become an urgent technical problem to be solved in big data analysis scenarios with high real-time requirements. Summary of the Invention

[0007] This invention provides a label data processing method, apparatus, computer equipment, and medium based on a columnar database bitmap encoded data structure to solve the technical problem of low label data processing efficiency in traditional technologies.

[0008] Firstly, a tag data processing method based on a columnar database bitmap encoded data structure is provided, comprising: responding to a tag data requirement corresponding to a preset business, determining whether a corresponding bitmapped tag dataset exists; if no corresponding bitmapped tag dataset exists, customizing a bitmapped tag dataset corresponding to the tag data requirement based on preset structured information tag data; determining a target bitmapped tag dataset corresponding to the tag data requirement based on the bitmapped tag dataset; and performing corresponding data processing on the target bitmapped tag dataset according to the tag data requirement.

[0009] Secondly, a tag data processing device based on a columnar database bitmap encoded data structure is provided, comprising: a first judgment module, used to respond to tag data requirements corresponding to a preset business and determine whether a corresponding bitmapped tag dataset exists; a tag data customization module, used to customize a bitmapped tag dataset corresponding to the tag data requirement based on preset structured information tag data when a corresponding bitmapped tag dataset does not exist; a first determination module, used to determine a target bitmapped tag dataset corresponding to the tag data requirement based on the bitmapped tag dataset; and a first tag data processing module, used to perform corresponding data processing on the target bitmapped tag dataset according to the tag data requirement.

[0010] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0011] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method.

[0012] In the aforementioned scheme implemented by the tag data processing method, apparatus, computer equipment, and storage medium based on a columnar database bitmap-encoded data structure, the method, in the absence of a corresponding bitmap-encoded tag dataset, customizes a bitmap-encoded tag dataset corresponding to the tag data requirements based on preset structured information tag data. Then, based on the bitmap-encoded tag dataset, it determines the target bitmap-encoded tag dataset corresponding to the tag data requirements. Finally, according to the tag data requirements, it performs corresponding data processing on the target bitmap-encoded tag dataset to meet the tag data processing needs of preset business operations. This achieves the customization of corresponding columnar storage bitmap-encoded tag datasets based on preset structured information tag data according to the different tag data processing needs corresponding to different business operations. Furthermore, it performs corresponding data queries, calculations, and analyses based on the bitmap-encoded tag datasets, thereby storing the structured information tag data... This technology combines data with a columnar database-based bitmap-encoded data structure, leveraging the strengths of this structure in high-speed querying and large-scale data analysis. Especially in real-time big data analytics scenarios, it improves the efficiency of data querying and computation, enabling rapid response to tagging needs across different business scenarios. Furthermore, it allows for the creation of new bitmap-encoded tag datasets or the adjustment of existing ones based on the columnar database bitmap-encoded data structure, without requiring large-scale modifications to the underlying data storage structure of the original, pre-defined structured information tag data. This enhances the processing efficiency, flexibility, and scalability of tag data for different businesses, thereby improving tag data processing efficiency. It can be applied to fields including, but not limited to, finance, insurance, and healthcare. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating the tag data processing method based on a columnar database bitmap encoded data structure provided in an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of the first sub-process of the tag data processing method based on the columnar database bitmap encoded data structure provided in an embodiment of the present invention;

[0016] Figure 3This is a schematic diagram of the second sub-process of the tag data processing method based on the columnar database bitmap encoded data structure provided in an embodiment of the present invention;

[0017] Figure 4 This is a schematic block diagram of a tag data processing device based on a columnar database bitmap encoded data structure provided in an embodiment of the present invention.

[0018] Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0019] Figure 6 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] This invention provides a tag data processing method based on a columnar database bitmap encoded data structure. The method can be applied to computer devices including but not limited to desktop computers, servers, and cloud platforms, and can be used in tag data processing in fields including but not limited to finance, insurance, and healthcare.

[0023] To address the low efficiency of traditional tag data processing technologies, the inventors propose a tag data processing method based on a columnar database bitmap-encoded data structure. The core idea of ​​this invention is to respond to tag requirements in different business scenarios, and based on stored structured information data (e.g., structured information data stored in a relational database), customize the tag data corresponding to the columnar storage based on the columnar database bitmap-encoded data structure—that is, a customized bitmap-encoded tag dataset. Then, based on this customized bitmap-encoded tag dataset, perform corresponding data queries, calculations, and analyses, thereby using the stored structured information data as a data source. Furthermore, based on this, and according to requirements, flexibly customize the columnar storage based on the columnar database bitmap-encoded data structure... By combining the corresponding tag data, and leveraging the bitmap-encoded data structure based on a columnar database, which excels in high-speed querying and large-scale data analysis (i.e., the Bitmap data structure), especially in big data analysis scenarios with high real-time requirements, the efficiency of corresponding data query and operation is improved. It can quickly respond to tag requirements in different business scenarios. Furthermore, new tag data based on the columnar database bitmap-encoded data structure can be created or existing tag data based on the columnar database bitmap-encoded data structure can be adjusted at any time according to the actual needs of different businesses, without modifying the underlying data storage structure of the original structured information. This improves the system's adaptability and scalability to different businesses, thereby improving the processing efficiency of tag data.

[0024] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating a tag data processing method based on a columnar database bitmap encoded data structure provided in an embodiment of the present invention. Figure 1 As shown, the method includes, but is not limited to, the following steps S11-S15:

[0026] S11. Respond to the tag data requirements corresponding to the preset business and determine whether there is a corresponding bitmap tag dataset.

[0027] Explaining this, processing the preset business involves processing the corresponding tag data, i.e., generating tag data requirements corresponding to the preset business. Tag data requirements indicate the processing of the relevant tag data corresponding to the preset business, including but not limited to querying, calculating, and analyzing. Therefore, in response to the tag data requirements corresponding to the preset business, it is determined whether a corresponding bitmapped tag dataset exists, i.e., whether the bitmapped tag dataset required by the preset business exists. A bitmapped tag dataset is a collection of Bitmap tag data, also known as bitmap indexed tag data or bitmap column tag data. It represents a set of tag data represented by a columnar database bitmap-encoded data structure, i.e., a set of tag data stored based on a columnar database bitmap-encoded data structure. Bitmapped represents the process of converting tag data into a bitmap-encoded structure, used to express and distinguish different tag data, not to limit the corresponding tag data. A bitmapped tag dataset is a set of bitmap-encoded tag data corresponding to columnar storage based on a columnar database bitmap-encoded data structure. The columnar storage database is ClickHouse or a database system with the same columnar storage characteristics. For example, in ClickHouse, Bitmaps are stored as independent columns. Each Bitmap column is physically independent, following ClickHouse's basic principle of columnar storage. Bitmap columns use a specialized bitmap compression algorithm, which significantly reduces space usage compared to traditional relational database storage or simple array list storage. When using columnar database bitmap-encoded data structure columns in big data analysis, identifying the content information stored in each Bitmap column can be done by checking, but not limited to, using the BitmapHas function, to see if a value exists in the Bitmap. For related technologies of columnar database bitmap-encoded data structures, please refer to existing related technologies, which will not be elaborated here.

[0028] For example, in the financial sector, in loan risk control, when querying "credit score < 600" + "recent multiple loan applications" + "low job stability," the corresponding tag data requirements for loan risk control include credit score tag data corresponding to "credit score < 600," loan frequency tag data corresponding to "recent multiple loan applications," and job stability tag data corresponding to "low job stability." This requires querying, calculating, and analyzing the aforementioned credit score tag data, loan frequency tag data, and job stability tag data to obtain the corresponding loan risk control results. In this scenario, it is necessary to determine whether Bitmap tag data corresponding to the credit score tag data, loan frequency tag data, and job stability tag data exists.

[0029] Similarly, in the insurance sector, in targeted insurance product promotion, when it involves querying "login count of a certain insurance APP more than 3 times in the past 7 days" + "number of long-term insurance policies held by the policyholder > 2", the tag data requirements for targeted insurance product promotion are the APP login count tag data corresponding to "login count of a certain insurance APP in the past 7 days" and the long-term insurance policy count tag data corresponding to "number of long-term insurance policies held by the policyholder". It is necessary to query, intersect, and analyze the above-mentioned APP login count tag data and long-term insurance policy count tag data to obtain the corresponding results of targeted insurance product promotion. In this scenario, it is necessary to determine whether there is Bitmap tag data corresponding to the APP login count tag data and the long-term insurance policy count tag data of a certain insurance customer.

[0030] Similarly, in the healthcare field, in the chronic disease management grouping business, when it involves querying "diabetes" + "hypertension" + "age > 60 years old", the corresponding tag data requirements for the chronic disease management grouping business are diabetes tag data corresponding to "diabetes disease information", hypertension tag data corresponding to "hypertension disease information", and age information tag data corresponding to "age > 60 years old". It is necessary to query, intersect, and analyze the above-mentioned diabetes tag data, hypertension tag data, and age information tag data to obtain the corresponding results of chronic disease management grouping. In this scenario, it is necessary to determine whether there is Bitmap tag data corresponding to the diabetes tag data, hypertension tag data, and age information tag data of a certain patient.

[0031] S12. If a corresponding bitmapped label dataset exists, determine the target bitmapped label dataset corresponding to the label data requirement based on the bitmapped label dataset, and perform corresponding data processing on the target bitmapped label dataset according to the label data requirement.

[0032] Explained, assuming a corresponding bitmapped label dataset exists—that is, if the corresponding bitmapped label dataset is found based on the label data required by the preset business—then the target bitmapped label dataset corresponding to the label data requirement is determined. The target bitmapped label dataset is a subset of the original bitmapped label dataset. For example, in ClickHouse, the Bitmap column supports both full operations and partial content extraction, providing flexible data access methods that can be flexibly configured as needed. The methods for obtaining the content of the ClickHouse Bitmap column can be found in existing technologies and will not be elaborated upon here.

[0033] Then, based on the tag data requirements, the target bitmap-based tag dataset is subjected to corresponding data processing including but not limited to intersection, union, and difference to meet the tag data processing needs of the preset business.

[0034] S13. In the absence of a corresponding bitmapped label dataset, based on the preset structured information label data, customize the bitmapped label dataset corresponding to the label data requirement.

[0035] Explained, pre-set structured information tag data, i.e., preset structured information tag data, represents tag data based on structured information representation. Pre-set structured information tag data can be obtained through methods including but not limited to the following two: 1) Offline access to structured data (such as database tables, CSV, JSON, and other big data warehouses) through data warehouse tools such as Hive and Spark; 2) Real-time access to semi-structured data (such as JSON-formatted user behavior tracking analysis data, logs, sensor data, etc.) through real-time data stream acquisition tools such as Kafka and Flink. For semi-structured data, key information can be extracted using JSON functions (such as JSONExtract) or regular expression functions (such as match) in ClickHouse, including but not limited to, and converted into a structured format for storage. JSONExtract is a tool in JMeter used to extract specific fields or values ​​from JSON-formatted data. The JSONPath syntax is used to specify the data fields to be extracted. Thus, data acquisition and standardization of multi-source data access are achieved, and structured information tag data is obtained.

[0036] Based on the above concept and setup, in the absence of a corresponding bitmapped label dataset, the target structured information label data is extracted from the preset structured information label data. The target structured information label data is a subset of the preset structured information label data, and the target structured information label data at least includes the structured information label data required by the label data requirement. That is, the structured information label data required by the label data requirement is a subset of the target structured information label data. The target structured information label data is then converted into a corresponding bitmapped label dataset to obtain the bitmapped label dataset corresponding to the label data requirement.

[0037] Because the target structured information tag data is extracted and transformed according to requirements from the preset structured information tag data, the custom construction of the bitmap tag dataset corresponding to the tag data requirements is realized, thus achieving the creation of a bitmap tag dataset corresponding to the custom tag data requirements. Furthermore, the bitmap tag dataset corresponds to a preset business; therefore, different businesses correspond to different bitmap tag datasets. That is, for different businesses, a corresponding bitmap tag dataset is customized, and the tag data required by the preset business is no longer directly related to the preset structured information tag data. This improves the efficiency, flexibility, and scalability of tag data processing for different businesses.

[0038] S14. Based on the bitmapped label dataset, determine the target bitmapped label dataset corresponding to the label data requirement;

[0039] S15. Based on the label data requirements, perform corresponding data processing on the target bitmap-based label dataset.

[0040] Explained as described above, when a corresponding bitmapped label dataset exists, that is, when the corresponding bitmapped label dataset is found based on the label data required by the preset business, the target bitmapped label dataset corresponding to the label data requirement is determined based on the bitmapped label dataset. The target bitmapped label dataset is a subset of the bitmapped label dataset. Then, according to the label data requirement, the target bitmapped label dataset is subjected to corresponding data processing including but not limited to intersection, union, and difference to meet the label data processing required by the preset business.

[0041] It should be noted that the corresponding bitmapped label dataset involved in step S12 above is also customized and constructed based on the processing method of the custom bitmapped label dataset corresponding to step S13, except that it already existed before the judgment in step 11.

[0042] This invention, in its embodiments, responds to the tag data requirements corresponding to a preset service, determines whether a corresponding bitmapped tag dataset exists, and if not, customizes a bitmapped tag dataset corresponding to the tag data requirement based on preset structured information tag data. Then, based on the bitmapped tag dataset, it determines the target bitmapped tag dataset corresponding to the tag data requirement. Finally, according to the tag data requirement, it performs corresponding data processing on the target bitmapped tag dataset to meet the tag data processing needs of the preset service. This achieves the following: based on preset structured information tag data, it customizes a corresponding columnar storage bitmapped tag dataset corresponding to a columnar database bitmap encoded data structure, and performs corresponding data queries, calculations, and analyses based on the bitmapped tag dataset, thereby storing... The system combines stored structured information tag data with a columnar database-based bitmap-encoded data structure. Leveraging the Bitmap data structure, which excels in high-speed querying and large-scale data analysis, this approach significantly improves the efficiency of data querying and computation, especially in real-time big data analysis scenarios. It can quickly respond to tag requirements across different business scenarios. Furthermore, it allows for the creation of new bitmap-encoded tag datasets or the adjustment of existing ones based on the columnar database bitmap-encoded data structure, without requiring large-scale modifications to the underlying data storage structure of the original, pre-defined structured information tag data. This enhances the processing efficiency, flexibility, and scalability of tag data for different business needs, thereby improving the overall tag data processing efficiency.

[0043] Please see Figure 2 , Figure 2 This is a schematic diagram of the first sub-process of the tag data processing method based on a columnar database bitmap encoded data structure provided in an embodiment of the present invention. Figure 2 As shown, in this embodiment, based on preset structured information tag data, a bitmap-based tag dataset corresponding to the customized tag data requirements is created, including:

[0044] S21. Determine the target structured information tag data corresponding to the tag data requirement based on the preset structured information tag data;

[0045] S22. Determine the user primary key corresponding to the target structured information tag data, and sort all the user primary keys to obtain a user primary key-sorted ordered mapping structure;

[0046] S23. Associate the target structured information tag data with the user primary key-sorted ordered mapping structure to obtain the user tag data model;

[0047] S24. Based on a preset data storage conversion method, the user tag data model is converted into a columnar storage data structure based on a columnar database bitmap encoding data structure and stored to obtain the bitmapped tag dataset corresponding to the tag data requirement.

[0048] Explained, based on preset structured information tag data, structured information tag data related to tag data requirements is determined from the preset structured information tag data, thereby determining the target structured information tag data corresponding to the tag data requirements, realizing the target structured information tag data corresponding to the custom tag data requirements, and further realizing the custom bitmap-based tag dataset corresponding to the columnar storage based on the columnar database bitmap encoded data structure. Here, the target structured information tag data is a subset of the preset structured information tag data. In one example, the target structured information tag data is the full amount of preset structured information tag data. The target structured information tag data contains several field names and information of the columns corresponding to the field names. The field names are the names used to identify columns in the structured data.

[0049] For example, as shown in the example above, in the financial sector, in loan risk control, when querying "credit score < 600" + "recent multiple loan applications" + "low job stability," the three field names and their corresponding column information for "credit score < 600" + "recent multiple loan applications" + "low job stability" are credit score, number of recent loan applications, and job stability. Similarly, in the insurance sector, in targeted insurance product promotion, when querying "login to a certain insurance APP more than 3 times in the past 7 days" + "policyholder holds more than 2 long-term insurance policies," the three field names and their corresponding column information are involved. The query "Login to an insurance app more than 3 times in the past 7 days" + "Number of long-term insurance policies held by the policyholder > 2" involves the two field names and their corresponding column information for "Number of logins to an insurance app in the past 7 days" and "Number of long-term insurance policies held by the policyholder". In the healthcare field, in the chronic disease management group business, when querying "diabetes" + "hypertension" + "age > 60 years old", the query involves the three field names and their corresponding column information for "diabetes disease information", "hypertension disease information", and "age" corresponding to the age information label data.

[0050] The user primary key is determined for the target structured information tag data. The user primary key represents the field or combination of fields in the database table corresponding to the predefined structured information tag data, uniquely identifying each row of data. The user primary key is usually designed and determined by relevant personnel, and its design process must consider business requirements and technical specifications. The user primary key is the core basis for linking different data. The user primary key is generally the field name or combination of field names mentioned above. All user primary keys are then sorted, typically by grouping and cumulatively sorting, resulting in a user primary key-sorted ordered mapping structure, i.e., a user primary key-sorted Map mapping dataset. The user primary key-sorted Map represents a two-layer structure that maps ordered primary keys to the physical storage location of a columnar database bitmap encoded data structure (such as ClickHouseBitmap). It belongs to the database index Map. The primary key layer is used to maintain the ordered primary key sequence (usually integer IDs), and the bitmap layer is used for each primary key to correspond to a compressed Bitmap data block. It should be noted that the Map in Bitmap represents a bit mapping relationship, used for indexing and table lookup, and has a different meaning from the Map in the user primary key-sorted Map.

[0051] Then, the target structured information tag data is associated with the user primary key-sorted ordered mapping structure to obtain the user tag data model. Based on a preset data storage transformation method, such as through functions including but not limited to ClickHousegroupBitmapState, the user tag data model is transformed into an efficient, compressed columnar database bitmap encoded data structure and stored to obtain the bitmap tag dataset corresponding to the custom tag data requirements. This enables the flexible creation of custom tag data according to business needs, thus providing a flexible custom tag mechanism that can quickly respond to changes in tag requirements under different business scenarios. Users can create new tags or adjust the existing tag system at any time according to actual business needs without making large-scale modifications to the underlying data storage structure, improving the system's adaptability and scalability to business needs. Furthermore, the Bitmap data structure adopts a bit vector storage method, which significantly reduces the space occupied compared to traditional relational database storage or simple array list storage.

[0052] This invention, through its embodiments, transforms preset structured information tag data to customize and construct a bitmap-based tag dataset corresponding to tag data requirements. This allows for the creation of customized bitmap-based tag datasets based on columnar database bitmap-encoded data structures, tailored to the different tag data needs of various business operations. Furthermore, it enables data querying, computation, and analysis based on these bitmap-based tag datasets. By combining the stored structured information tag data with the columnar database bitmap-encoded data structure, and leveraging the Bitmap data structure's strengths in high-speed querying and large-scale data analysis, the invention improves processing efficiency, flexibility, and scalability for different business operations, thereby enhancing the overall tag data processing efficiency.

[0053] In one embodiment, after customizing the bitmapped label dataset corresponding to the label data requirement, the method further includes:

[0054] In response to the update of the target structured information label data, the bitmapped label dataset is updated synchronously.

[0055] Explained, this method involves updating the target structured information tag data while simultaneously updating the bitmap tag dataset. This achieves synchronized updates between the target structured information tag data and the bitmap tag dataset. In other words, in response to updates to the target structured information tag data, the bitmap tag dataset is updated synchronously. This can be achieved through real-time data synchronization using distributed streaming and batch processing frameworks, including but not limited to Flink, Spark, and Kafka. The corresponding tag data is then built and refreshed in real-time into the corresponding Bitmap table in ClickHouse columnar storage. This ensures that the target structured information tag data and the bitmap tag dataset are updated synchronously, allowing relevant business processes to use the latest tag data. In big data analysis scenarios with high real-time requirements, the latest tag data can be used in real time, enabling rapid response to tag needs in different business scenarios.

[0056] This invention, through synchronous updates of target structured information tag data and bitmap tag dataset, achieves real-time synchronous updates of the bitmap tag dataset. Based on the updated bitmap tag dataset, it responds to the tag data requirements of corresponding preset business scenarios. Leveraging the columnar database-based bitmap encoded data structure, which excels in high-speed querying and large-scale data analysis, the Bitmap data structure, especially in big data analysis scenarios with high real-time requirements, further improves the efficiency of corresponding data querying and computation, quickly responding to tag requirements in different business scenarios, thereby improving the processing efficiency of tag data.

[0057] Please see Figure 3 , Figure 3This is a schematic diagram of the second sub-process of the tag data processing method based on a columnar database bitmap encoded data structure provided in an embodiment of the present invention. Figure 3 As shown, in this embodiment, in response to the tag data requirement corresponding to the preset service, determining whether a corresponding bitmapped tag dataset exists includes:

[0058] S31. Determine the target data label corresponding to the label data requirement;

[0059] S32. Based on the constructed bitmapped label dataset, determine whether there exists a bitmapped label dataset corresponding to the target data label;

[0060] S33. If the above judgment is true, it is determined that there is a corresponding bitmapped label dataset;

[0061] S34. If the above judgment is not true, it is determined that there is no corresponding bitmapped label dataset.

[0062] Explainingly, the target data label corresponding to the label data requirement is determined. Based on the constructed bitmapped label dataset, it is determined whether a bitmapped label dataset corresponding to the target data label exists. That is, the constructed bitmapped label dataset is queried to see if a bitmapped label dataset corresponding to the target data label exists. If the above judgment is yes, it is determined that a corresponding bitmapped label dataset exists. If the above judgment is no, it is determined that a corresponding bitmapped label dataset does not exist. Thus, in response to the label data requirement corresponding to the preset business, it is determined whether a corresponding bitmapped label dataset exists.

[0063] In this embodiment of the invention, by determining whether a corresponding bitmapped tag dataset exists based on the target data tag, the Bitmap data structure, which is based on a columnar database and is good at high-speed querying and large-scale data analysis, can quickly determine whether a corresponding bitmapped tag dataset exists for the tag data requirements corresponding to specific preset business, thereby quickly responding to tag requirements under different business scenarios and improving the processing efficiency of tag data.

[0064] In one embodiment, determining the target bitmapped label dataset corresponding to the label data requirement based on the bitmapped label dataset includes:

[0065] Determine the data tags corresponding to the aforementioned tag data requirements;

[0066] Based on the bitmapped label dataset, query the label data corresponding to the data label to obtain the target bitmapped label dataset corresponding to the label data requirement.

[0067] Interpretatively, determine the data labels corresponding to the label data requirements, and query the label data corresponding to the data labels based on the bitmapped label dataset to obtain the target bitmapped label dataset corresponding to the label data requirements.

[0068] For example, in the financial sector, in loan risk control, when it comes to querying "credit score < 600" + "recent multiple loan applications" + "low job stability", the target Bitmap tag data corresponding to "credit score < 600" + "recent multiple loan applications" + "low job stability" is obtained by querying the Bitmap tag data based on Bitmap tag data.

[0069] Similarly, in the insurance sector, in targeted promotion of insurance products, when it comes to querying "login to a certain insurance APP more than 3 times in the past 7 days" + "policyholder holding more than 2 long-term insurance policies", based on Bitmap tag data, query the tag data corresponding to "login to a certain insurance APP more than 3 times in the past 7 days" + "policyholder holding more than 2 long-term insurance policies", and obtain the target Bitmap tag data corresponding to "login to a certain insurance APP more than 3 times in the past 7 days" + "policyholder holding more than 2 long-term insurance policies".

[0070] Similarly, in the healthcare field, in the chronic disease management grouping business, when it comes to querying "diabetes" + "hypertension" + "age > 60 years old", based on the Bitmap tag data, query the tag data corresponding to "diabetes" + "hypertension" + "age > 60 years old" to obtain the target Bitmap tag data corresponding to "diabetes" + "hypertension" + "age > 60 years old".

[0071] In this embodiment of the invention, by transforming the preset structured information tag data accordingly, a bitmap-based tag dataset corresponding to the tag data requirements is customized and constructed. Based on the bitmap-based tag dataset, corresponding data queries, operations, and analyses are performed. By leveraging the Bitmap data structure, which is based on a columnar database and excels in high-speed querying and large-scale data analysis, the filtering of single tag data and complex combination operations of multiple tag data can be completed in a very short time, meeting the needs of real-time big data analysis and response, thereby improving the processing efficiency of tag data.

[0072] In one embodiment, before customizing the bitmap-based tag dataset corresponding to the tag data requirements based on preset structured information tag data, the method further includes:

[0073] Based on the target data tag, determine whether the preset structured information tag data contains the target structured information tag data corresponding to the tag data requirement;

[0074] If the above judgment is correct, execute the step of "customizing the bitmap-based tag dataset corresponding to the tag data requirements based on the preset structured information tag data";

[0075] If the above judgment is incorrect, an error message will be issued.

[0076] Explained, based on the target data tags, it is determined whether the preset structured information tag data contains the target structured information tag data corresponding to the tag data requirement. If the above determination is yes, that is, if the preset structured information tag data contains the target structured information tag data corresponding to the tag data requirement, it indicates that the preset structured information tag data contains relevant data, and the step of "customizing the bitmap tag dataset corresponding to the tag data requirement based on the preset structured information tag data" is executed to perform the corresponding customization of the bitmap tag dataset. If the above determination is no, that is, if the preset structured information tag data does not contain the target structured information tag data corresponding to the tag data requirement, it indicates that the preset structured information tag data does not contain relevant data, and there is no need to perform the corresponding customization of the bitmap tag dataset, and an error message is issued.

[0077] In this embodiment of the invention, by determining whether the preset structured information tag data exists based on the target data tag, and then determining whether to customize the bitmap tag dataset corresponding to the tag data requirement, it is possible to respond more quickly to the tag requirements under different business scenarios and further improve the processing efficiency of tag data.

[0078] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0079] In one embodiment, a tag data processing apparatus based on a columnar database bitmap encoded data structure is provided. This tag data processing apparatus corresponds one-to-one with the tag data processing method based on a columnar database bitmap encoded data structure described in the above embodiments. Please refer to [link to previous document]. Figure 4 , Figure 4 This is a schematic block diagram of a tag data processing device based on a columnar database bitmap encoded data structure, provided in an embodiment of the present invention. Figure 4As shown, the tag data processing device 40 based on the columnar database bitmap encoded data structure includes a first judgment module 41, a tag data customization module 42, a first determination module 43, and a first tag data processing module 44. The above functional modules are described in detail below:

[0080] The first judgment module 41 is used to respond to the tag data requirements corresponding to the preset business and determine whether there is a corresponding bitmap tag dataset.

[0081] The tag data customization module 42 is used to customize the bitmapped tag dataset corresponding to the tag data requirement based on the preset structured information tag data when there is no corresponding bitmapped tag dataset.

[0082] The first determining module 43 is used to determine the target bitmapped label dataset corresponding to the label data requirement based on the bitmapped label dataset.

[0083] The first tag data processing module 44 is used to perform corresponding data processing on the target bitmap tag dataset according to the tag data requirements.

[0084] In one embodiment, the tag data customization module 42 includes:

[0085] The first determining submodule is used to determine the target structured information tag data corresponding to the tag data requirement based on the preset structured information tag data;

[0086] The sorting submodule is used to determine the user primary key corresponding to the target structured information tag data, and sort all the user primary keys to obtain a user primary key-sorted ordered mapping structure;

[0087] The association submodule is used to associate the target structured information tag data with the user primary key-sorted ordered mapping structure to obtain the user tag data model;

[0088] The conversion submodule is used to convert the user tag data model into a columnar storage data structure based on a columnar database bitmap encoded data structure and store it, based on a preset data storage conversion method, to obtain the bitmapped tag dataset corresponding to the tag data requirements.

[0089] In one embodiment, the tag data processing device 40 further includes:

[0090] The bitmapped label dataset update module is used to respond to the update of the target structured information label data and synchronously update the bitmapped label dataset.

[0091] In one embodiment, the tag data processing device 40 further includes:

[0092] The second tag data processing module is used to determine the target bitmapped tag dataset corresponding to the tag data requirement based on the bitmapped tag dataset when a corresponding bitmapped tag dataset exists, and to perform corresponding data processing on the target bitmapped tag dataset according to the tag data requirement.

[0093] In one embodiment, the first determining module 41 includes:

[0094] The second determining submodule is used to determine the target data label corresponding to the label data requirement;

[0095] The first judgment submodule is used to determine whether there is a bitmapped label dataset corresponding to the target data label based on the constructed bitmapped label dataset.

[0096] The first determination submodule is used to determine the existence of a corresponding bitmapped label dataset if the above determination is true.

[0097] The second determination submodule is used to determine that there is no corresponding bitmapped label dataset if the above determination is negative.

[0098] In one embodiment, the first determining module 43 includes:

[0099] The third determining submodule is used to determine the data tags corresponding to the tag data requirements;

[0100] The query submodule is used to query the tag data corresponding to the data tag based on the bitmapped tag dataset, and obtain the target bitmapped tag dataset corresponding to the tag data requirement.

[0101] In one embodiment, the tag data processing device 40 further includes:

[0102] The second judgment module is used to determine, based on the target data tag, whether the preset structured information tag data exists in the target structured information tag data corresponding to the tag data requirement;

[0103] The execution module is used to execute the step of "customizing the bitmap-based tag dataset corresponding to the tag data requirements based on the preset structured information tag data" if the above judgment is true.

[0104] This invention provides a tag data processing device based on a columnar database bitmap-encoded data structure. By responding to tag data requirements corresponding to preset business needs, it determines whether a corresponding bitmapped tag dataset exists. If no such dataset exists, it customizes a bitmapped tag dataset corresponding to the tag data requirement based on preset structured information tag data. Then, based on the bitmapped tag dataset, it determines the target bitmapped tag dataset corresponding to the tag data requirement. Finally, it performs corresponding data processing on the target bitmapped tag dataset to meet the tag data processing needs of the preset business. This allows for the customization of corresponding columnar storage bitmapped tag datasets based on a columnar database bitmap-encoded data structure, based on preset structured information tag data, to meet the different tag data processing requirements of different businesses. This system enables data querying, computation, and analysis, combining stored structured information tag data with a columnar database-based bitmap-encoded data structure. Leveraging the Bitmap data structure's strengths in high-speed querying and large-scale data analysis, especially in real-time big data analytics scenarios, it improves the efficiency of data querying and computation. It can quickly respond to tag requirements in different business scenarios. Furthermore, it allows for the creation of new bitmap-encoded tag datasets or the adjustment of existing ones based on the columnar database bitmap-encoded data structure, without requiring large-scale modifications to the underlying data storage structure of the original, pre-defined structured information tag data. This improves the efficiency, flexibility, and scalability of tag data processing for different businesses, thereby enhancing the overall tag data processing efficiency.

[0105] Specific limitations regarding the tag data processing device based on a columnar database bitmap encoded data structure can be found in the limitations of the tag data processing method based on this structure mentioned above, and will not be repeated here. Each module in the aforementioned tag data processing device based on a columnar database bitmap encoded data structure can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0106] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a tag data processing method based on a columnar database bitmap encoded data structure on the server side.

[0107] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a tag data processing method based on a columnar database bitmap encoded data structure.

[0108] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0109] Responding to the tag data requirements corresponding to the preset business, determine whether there is a corresponding bitmap tag dataset;

[0110] In the absence of a corresponding bitmapped label dataset, a bitmapped label dataset corresponding to the label data requirement is customized based on the preset structured information label data.

[0111] Based on the bitmapped label dataset, determine the target bitmapped label dataset corresponding to the label data requirement;

[0112] Based on the stated label data requirements, the target bitmapped label dataset is processed accordingly.

[0113] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0114] Responding to the tag data requirements corresponding to the preset business, determine whether there is a corresponding bitmap tag dataset;

[0115] In the absence of a corresponding bitmapped label dataset, a bitmapped label dataset corresponding to the label data requirement is customized based on the preset structured information label data.

[0116] Based on the bitmapped label dataset, determine the target bitmapped label dataset corresponding to the label data requirement;

[0117] Based on the stated label data requirements, the target bitmapped label dataset is processed accordingly.

[0118] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0121] The software tools or components not belonging to our company that appear in the embodiments of this invention are merely illustrative examples and do not represent actual use.

[0122] The data collection in this embodiment of the invention complies with the requirements of relevant laws and regulations, such as China's Personal Information Protection Law, GDPR (General Data Protection Regulation of the European Union), or information security standards of other countries and regions.

[0123] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A label data processing method based on columnar database bitmap encoding data structure, characterized in that, The method comprises the following steps: in response to a label data requirement corresponding to a preset service, determining whether a corresponding bitmapized label data set exists; in the case where the corresponding bitmapized label data set does not exist, defining a bitmapized label data set corresponding to the label data requirement based on preset structured information label data; determining a target bitmapized label data set corresponding to the label data requirement according to the bitmapized label data set; performing corresponding data processing on the target bitmapized label data set according to the label data requirement.

2. The label data processing method based on a column database bitmap encoding data structure according to claim 1, wherein, defining a bitmapized label data set corresponding to the label data requirement based on preset structured information label data comprises the following steps: determining target structured information label data corresponding to the label data requirement according to the preset structured information label data; determining user primary keys corresponding to the target structured information label data, and sorting all the user primary keys to obtain a user primary key-sorting ordered mapping structure; associating the target structured information label data with the user primary key-sorting ordered mapping structure to obtain a user label data model; based on a preset data storage conversion mode, converting the user label data model into a columnar storage data structure based on a columnar database bitmap encoding data structure and storing to obtain the bitmapized label data set corresponding to the label data requirement.

3. The method of claim 1, wherein the bitmap encoding data structure is based on a columnar database. after defining the bitmapized label data set corresponding to the label data requirement, the method further comprises the following step: in response to updating of the target structured information label data, synchronously updating the bitmapized label data set.

4. The label data processing method based on a columnar database bitmap encoding data structure according to any one of claims 1-3, characterized in that, The method further comprises the following steps: in the case where the corresponding bitmapized label data set exists, determining a target bitmapized label data set corresponding to the label data requirement according to the bitmapized label data set, and performing corresponding data processing on the target bitmapized label data set according to the label data requirement.

5. The label data processing method based on a columnar database bitmap encoding data structure according to any one of claims 1-4, characterized in that, in response to a label data requirement corresponding to a preset service, determining whether a corresponding bitmapized label data set exists comprises the following steps: determining target data label corresponding to the label data requirement; determining whether a bitmapized label data set corresponding to the target data label exists according to the bitmapized label data set that has been constructed; if the determination is yes, it is determined that the corresponding bitmapized label data set exists; if the determination is no, it is determined that the corresponding bitmapized label data set does not exist.

6. The method of claim 5, wherein the bitmap encoding data structure is based on a columnar database. determining a target bitmapized label data set corresponding to the label data requirement according to the bitmapized label data set comprises the following steps: determining data label corresponding to the label data requirement; querying label data corresponding to the data label according to the bitmapized label data set to obtain a target bitmapized label data set corresponding to the label data requirement.

7. The tag data processing method based on a column database bitmap encoding data structure according to claim 5, wherein, before defining a bitmapized label data set corresponding to the label data requirement based on preset structured information label data, the method further comprises the following step: determining whether target structured information label data corresponding to the label data requirement exists in the preset structured information label data according to the target data label; If the determination is yes, a step of "defining a bitmapped label data set corresponding to the label data requirement based on preset structured information label data" is performed.

8. A label data processing apparatus based on a columnar database bitmap encoding data structure, characterized by, The method comprises the following steps: A first determination module is configured to determine whether a corresponding bitmapped label data set exists in response to a label data requirement corresponding to a preset service. A label data definition module is configured to define a bitmapped label data set corresponding to the label data requirement based on preset structured information label data in the case that no corresponding bitmapped label data set exists. A first determination module is configured to determine a target bitmapped label data set corresponding to the label data requirement according to the bitmapped label data set. A first label data processing module is configured to perform corresponding data processing on the target bitmapped label data set according to the label data requirement.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the label data processing method based on the columnar database bitmapped encoding data structure according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the label data processing method based on the columnar database bitmapped encoding data structure according to any one of claims 1 to 7.

Images