Label preprocessing method and device, computer device and storage medium

Abstract

This application relates to a tag preprocessing method, apparatus, computer device, and storage medium. The method includes: configuring relevant data information according to the task type of the preprocessed tags; reading first configuration information from the relevant data information to obtain Aviator expressions and preprocessed tags; parsing the preprocessed tags to obtain atomic tags; establishing a key-value relationship between the preprocessed tags and atomic tags in a local caching application; constructing a user-scope query database based on the key-value relationship in the local caching application; performing user-scope queries using a stream-batch integrated structured query language engine to obtain user-scope data; calculating and obtaining the values ​​of the preprocessed tags based on the Aviator expressions and user-scope data; and storing the calculated values ​​of the preprocessed tags according to a preset storage mechanism. This application can solve the timeout problem of existing tagging systems, reduce the complexity of developing user tags, greatly improve data processing capabilities, and significantly increase the storage space of storage institutions.

Description

Technical Field

[0001] This application relates to the technical field of big data processing, and in particular to a tag preprocessing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of internet applications, data analysis is playing an increasingly important role in business decision-making. Data analysis typically requires preprocessing operations such as tag extraction and filtering of the data to be analyzed. However, data tagging involves numerous data sources, diverse tagging rules, and a complex network environment. Traditional data tagging methods require excessive manual intervention, have low automation, are complex, and result in low efficiency. Furthermore, they are not applicable to tagging various data sources, leading to poor versatility and low accuracy. To address the need for manual intervention in existing technologies, this paper proposes an intelligent data processing platform, the VSP Tagging System (Vipshop Financial Tagging System). However, the existing VSP Tagging System still suffers from the following problems:

[0003] 1. Due to the large number of tag dependency levels, the real-time calculation time will be too long, resulting in a large number of timeouts for hot and complex tags, which poses a great risk to the stable operation of the business.

[0004] 2. The Storm (distributed real-time big data processing system) cluster has high maintenance costs and is about to be decommissioned, so there is an urgent need to replace it with a Flink (an open-source big data processing framework under the Apache Foundation) cluster.

[0005] 3. With rapid business development and an excessively fast increase in the number of tags, the processing time for Storm cluster jobs is getting longer and longer for processing hundreds of millions of user data.

[0006] 4. The system's Redis (Remote Dictionary Server) has extremely high memory usage, and memory usage is growing too fast as the business develops.

[0007] Therefore, there is an urgent need to propose a stable and efficient tag preprocessing method, device, computer equipment, and storage medium that can realize massive data processing and storage. Summary of the Invention

[0008] Therefore, it is necessary to provide a stable and efficient tag preprocessing method, apparatus, computer equipment, and storage medium that can handle and store massive amounts of data, addressing the aforementioned technical problems.

[0009] On the one hand, a label preprocessing method is provided, the method comprising:

[0010] Step A: Configure relevant data information according to the task type of the preprocessing tags;

[0011] Step B: Read the first configuration information from the relevant data information to obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application;

[0012] Step C: Construct a user-scoped query database based on the key-value pair relationships in the local cache application, and use a stream-batch integrated structured query language engine to perform user-scoped queries to obtain user-scoped data;

[0013] Step D: Based on the Aviator expression and the user-scoped data, calculate and obtain the value of the preprocessed label, and store the calculated value of the preprocessed label according to a preset storage mechanism.

[0014] In one embodiment, the method further includes: the relevant data information includes the first configuration information and the second configuration information, specifically: the first configuration information includes a self-service user management table, a user configuration management table, and an attribute configuration management table; when adding / modifying the first configuration information, a tag preprocessing option is added, and a preprocessing status viewing option is added to the list operation column; the second configuration information includes a monitoring table, and the configuration process of the monitoring table includes: configuring interactive functions: adding a preset queue field to the list; adding / editing the monitoring table, adding a preset queue item; adding a business data extraction table name field to the list; adding / editing the monitoring table function, adding a business data extraction table name item; initializing the configuration: connecting to the preset queue item of the existing table, updating the monitoring table record, and updating the business data extraction table name of the existing record.

[0015] In one embodiment, the method further includes: reading the first configuration information from the relevant data information to obtain Aviator expressions and preprocessed tags, parsing the preprocessed tags to obtain atomic tags, and establishing a key-value pair relationship between the preprocessed tags and the atomic tags in the local cache application. This includes: reading the attribute configuration management table as basic data; reading the preprocessed tags and expression rules of the preprocessed tags from the self-service population management table and the population configuration management table; parsing the expression rules of the preprocessed tags to obtain atomic tags; establishing key-value pair objects of the atomic tags, establishing correspondence key-value pair objects between the atomic tags and the preprocessed tags, and establishing tag rule key-value pair objects of the preprocessed tags in the local cache application.

[0016] In one embodiment, the method further includes: constructing a user-scoped query database based on the key-value pair relationships in the local cache application, and performing user-scoped queries using a stream-batch integrated structured query language engine to obtain user-scoped data. This includes: querying the atomic tags of the preprocessed tags based on the key-value pair relationships in the local cache application; querying the corresponding monitoring table records through the atomic tags, and reading the configured data warehouse tool table; constructing data warehouse tool statements based on the configuration of the atomic tags and the data warehouse tool table; defining the current preprocessed tag's status as initializing, and generating the user-scoped query database; and performing user-scoped queries on the user-scoped query database using the stream-batch integrated structured query language engine to obtain the user-scoped data.

[0017] In one embodiment, the method further includes: calculating and obtaining the value of the preprocessing tag based on the Aviator expression and the user scope data, which includes: consuming incremental data messages and filtering messages according to the configuration information of the atomic tags; reading the filtered message data and performing calculations according to the calculation rules of the atomic tags to generate atomic tag values; querying the correspondence key-value pair object between the atomic tags and preprocessing tags from the local cache application and comparing it with the atomic tag values; if the atomic tag values ​​have changed, then traversing multiple preprocessing tags.

[0018] In one embodiment, the method further includes: querying the key value of a non-message atomic tag in the user-scoped data based on the atomic tag of the preprocessed tag; and executing the Aviator expression of the preprocessed tag to generate the value of the preprocessed tag.

[0019] In one embodiment, the method further includes: storing the calculated value of the preprocessed label according to a preset storage mechanism includes: using a compressed bitmap mechanism to store the value of the preprocessed label: storing the identifier of the binarized population label as a bit in the compressed bitmap object, serializing it into a string, and then storing it in a key-value storage database.

[0020] On the other hand, a label preprocessing apparatus is provided, the apparatus comprising:

[0021] The information configuration module is used to configure relevant data information according to the task type of the preprocessed tags;

[0022] The relationship establishment module is used to read the first configuration information in the relevant data information, obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application;

[0023] The user-scope data acquisition module is used to construct a user-scope query database based on the key-value pair relationship in the local cache application, and to perform user-scope queries using a stream-batch integrated structured query language engine to obtain user-scope data.

[0024] The calculation and storage module is used to calculate and obtain the value of the preprocessed label based on the Aviator expression and the user-scoped data, and store the calculated value of the preprocessed label according to a preset storage mechanism.

[0025] In another aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0026] Step A: Configure relevant data information according to the task type of the preprocessing tags;

[0027] Step B: Read the first configuration information from the relevant data information to obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application;

[0028] Step C: Construct a user-scoped query database based on the key-value pair relationships in the local cache application, and use a stream-batch integrated structured query language engine to perform user-scoped queries to obtain user-scoped data;

[0029] Step D: Based on the Aviator expression and the user-scoped data, calculate and obtain the value of the preprocessed label, and store the calculated value of the preprocessed label according to a preset storage mechanism.

[0030] In another aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0031] Step A: Configure relevant data information according to the task type of the preprocessing tags;

[0032] Step B: Read the first configuration information from the relevant data information to obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application;

[0033] Step C: Construct a user-scoped query database based on the key-value pair relationships in the local cache application, and use a stream-batch integrated structured query language engine to perform user-scoped queries to obtain user-scoped data;

[0034] Step D: Based on the Aviator expression and the user-scoped data, calculate and obtain the value of the preprocessed label, and store the calculated value of the preprocessed label according to a preset storage mechanism.

[0035] The aforementioned tag preprocessing method, apparatus, computer equipment, and storage medium include the following steps: configuring relevant data information according to the task type of the preprocessed tags; reading first configuration information from the relevant data information to obtain an Aviator expression and preprocessed tags, parsing the preprocessed tags to obtain atomic tags, and establishing a key-value pair relationship between the preprocessed tags and the atomic tags in a local cache application; constructing a user-scope query database based on the key-value pair relationship in the local cache application, performing user-scope queries using a stream-batch integrated structured query language engine to obtain user-scope data; calculating and obtaining the value of the preprocessed tags based on the Aviator expression and the user-scope data, and storing the calculated value of the preprocessed tags according to a preset storage mechanism. This application uses a stream-batch integrated framework and a distributed processing engine to calculate and process preprocessed tags in real time, solving the tag timeout problem in existing systems, reducing the complexity of developing user tags, greatly improving data processing capabilities, and using a compressed bitmap mechanism to store values, significantly increasing the storage space of the storage institution. Attached Figure Description

[0036] Figure 1 This is a diagram illustrating the application environment of a label preprocessing method in one embodiment.

[0037] Figure 2 This is a flowchart illustrating a tag preprocessing method in one embodiment;

[0038] Figure 3 This is another flowchart illustrating the label preprocessing method in one embodiment;

[0039] Figure 4 This is a schematic diagram of the timing flow of a tag query request in a tag preprocessing method in one embodiment.

[0040] Figure 5 This is a schematic diagram of the real-time calculation timing flow of tag data in a tag preprocessing method in one embodiment;

[0041] Figure 6 This is a schematic diagram of the physical system deployment of the tag preprocessing method in one embodiment;

[0042] Figure 7 This is a schematic diagram illustrating a data storage example of a tag preprocessing method in one embodiment;

[0043] Figure 8 This is a structural block diagram of a label preprocessing device in one embodiment;

[0044] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] The label preprocessing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with a data processing platform located on server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, and tablets, and server 104 can be a standalone server or a server cluster consisting of multiple servers.

[0047] Example 1

[0048] In one embodiment, such as Figures 2-7 As shown, a tag preprocessing method is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0049] S1: Configure relevant data information based on the task type of the preprocessing label.

[0050] It should be noted that the preprocessing tag is the appendix. Figure 2 The real-time B tag in the configuration, the preprocessing tag task type can be message access and offline table query, and the relevant configuration data information includes the first configuration information and the second configuration information, specifically:

[0051] The first configuration information includes a self-service user management table, a user configuration management table, and an attribute configuration management table. When adding / modifying the first configuration information, a tag preprocessing option is added, and a preprocessing status viewing option is added to the list operation column.

[0052] The second configuration information includes a monitoring table, and the configuration process of the monitoring table includes:

[0053] Configure interactive functions: Add a preset queue field to the list; add / edit a monitoring table and add a preset queue item; add a business data extraction table name field to the list; add / edit a monitoring table and add a business data extraction table name item. For example, configure a Flink VDP (VIPShop Data Pump, which provides incremental data subscription and consumption tools based on database incremental log parsing, abbreviated as VDP) queue to provide an entry point for subsequent VDP message access.

[0054] Initialization configuration: Connect to the preset queue items of the existing table and update the monitoring table records. Update the business data extraction table name of the existing records. For example, configure the business data extraction table to initialize the data batch processing of Flink SQL (a development language that conforms to standard structured query semantics) and provide an offline table query entry point.

[0055] S2: Read the first configuration information from the relevant data information to obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application.

[0056] It should be noted that the attribute configuration management table, self-service user management table, and user configuration management table in the first configuration information are read. Among them, the attribute configuration management table is read as the basic data. The self-service user management table and the user configuration management table contain preprocessing tags and expression rules for the preprocessing tags. The expression mentioned here is the Aviator expression. The Aviator expression is a high-performance, lightweight expression evaluation engine implemented in Java, which is mainly used for dynamic evaluation of various expressions.

[0057] Parsing the expression rules of the preprocessed tags yields atomic tags, which are then... Figures 2-4 Real-time A tags in the middle;

[0058] In the local cache application (localCache), the atomic tag key-value pair object is established, the correspondence key-value pair object between the atomic tag and the preprocessed tag is established, and the tag rule key-value pair object of the preprocessed tag is established.

[0059] S3: Construct a user-scoped query database based on the key-value pair relationships in the local cache application, and use a stream-batch integrated structured query language engine to perform user-scoped queries to obtain user-scoped data.

[0060] It should be noted that, based on steps S1 and S2, a user-wide query SQL is constructed and performed using the Flink SQL engine (a stream-batch integrated structured query language engine). After retrieving the query data, the next step is executed, specifically:

[0061] Based on the key-value pair relationship in the local cache application, query the atomic tags of the preprocessed tags;

[0062] The corresponding monitoring table records are queried using the atomic tags, and the configured data warehouse tool (hive-data warehouse tool) table is read.

[0063] Based on the configuration of the atomic tags and the data warehouse tool table, construct data warehouse tool statements;

[0064] Define the current preprocessed tag status as initializing and generate a user-wide query database;

[0065] The user-scope query database is used to perform a user-scope query to obtain the user-scope data by using a stream-batch integrated structured query language engine.

[0066] S4: Based on the Aviator expression and the user range data, calculate and obtain the value of the preprocessed tag, and store the calculated value of the preprocessed tag according to a preset storage mechanism.

[0067] It should be noted that the calculation of the preprocessed label value based on the Aviator expression and the user-scoped data includes:

[0068] The incremental data consumption message (VDP message) is filtered according to the configuration information of the atomic tag;

[0069] According to the calculation rules of the atomic tags, the filtered message data is read and calculated to generate atomic tag values. The calculation rules here are expression rules.

[0070] The local cache application queries the key-value pair object corresponding to the atomic tag and the preprocessing tag, and compares it with the atomic tag value. If the atomic tag value changes, since multiple preprocessing tags are configured and all depend on the same atomic tag, then...

[0071] Traverse multiple of the preprocessed tags;

[0072] The purpose of the above steps is that when the atomic tag value changes, the upstream tags need to be recalculated. It is necessary to know which upstream tags depend on this atomic tag, so it is necessary to find the key-value pair objects and compare them.

[0073] Furthermore, based on the atomic tags of the preprocessed tags, query the key values ​​of the non-message atomic tags in the user-scoped data, that is, the Redis values ​​of the non-message atomic tags. For example, a tag A may depend on tag B, tag C, or tag D. A real-time message in a business table may only be associated with tag B. In this case, the value of tag B needs to be updated with the real-time message to update the stored value, and tags C and D need to continue to read the original stored values. At this time, the non-message atomic tags refer to tags C and D.

[0074] The Aviator expression of the preprocessed tag is executed to generate the value of the preprocessed tag.

[0075] Furthermore, storing the calculated values ​​of the preprocessed tags according to a preset storage mechanism includes:

[0076] The values ​​of the preprocessed tags are stored using a compressed bitmap (roaring bitmap) mechanism.

[0077] The identifier of the binarized crowd label is stored as a bit in the compressed bitmap object, serialized into a string, and then stored in the key-value storage database.

[0078] For example, such as Figure 7 As shown, the preprocessed label calculated values ​​are stored in Redis and MySQL (relational database):

[0079] Redis part: For binary (containing only true and false values) user tags, which usually describe the user's hit rate, the data is relatively stable. The id is stored as a bit in the rising bitmap in the rBitmap object, and after being serialized into a string, it is stored in the value.

[0080] offset: the primary key ID of the audience (for example, the ID corresponding to ug09030460 is 2044).

[0081] v: Crowd value (Example 1). If the crowd does not belong, remove it.

[0082] Redis storage structure: gb:u:Vipshop userid (Vipshop user primary key id);

[0083] Database section: The database is partitioned by userId, with 64 databases applied: schem_01, schem_02, ..., schem_n. Each database has 4 tables: Real-time A population table, Real-time A attribute table, Real-time B attribute table, and Real-time B population table.

[0084] Database sharding logic: userId%64 confirms which database the user belongs to, and confirms with the DBA via dropdown. This requires 2 machines, 4 instances on each machine, and 8 databases on each instance.

[0085] The aforementioned tag preprocessing method includes: configuring relevant data information according to the task type of the preprocessed tag; reading the first configuration information from the relevant data information to obtain the AVIATOR expression and the preprocessed tag, parsing the preprocessed tag to obtain atomic tags, and establishing a key-value pair relationship between the preprocessed tag and the atomic tag in a local cache application; constructing a user-scope query database based on the key-value pair relationship in the local cache application, performing user-scope queries using a stream-batch integrated structured query language engine to obtain user-scope data; calculating and obtaining the value of the preprocessed tag based on the AVIATOR expression and the user-scope data, and storing the calculated value of the preprocessed tag according to a preset storage mechanism. This application uses a stream-batch integrated framework and a distributed processing engine to calculate and process preprocessed tags in real time, solving the tag timeout problem in existing systems, reducing the complexity of developing user tags, greatly improving data processing capabilities, and using a compressed bitmap mechanism to store values, significantly increasing the storage space of the storage institution.

[0086] It should be understood that, although Figure 2-5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-5 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0087] Example 2

[0088] In one embodiment, such as Figure 8 As shown, a tag preprocessing device is provided, comprising: an information configuration module, a relationship establishment module, a user-scope data acquisition module, and a calculation and storage module, wherein:

[0089] The information configuration module is used to configure relevant data information according to the task type of the preprocessed tags;

[0090] The relationship establishment module is used to read the first configuration information in the relevant data information, obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application;

[0091] The user-scope data acquisition module is used to construct a user-scope query database based on the key-value pair relationship in the local cache application, and to perform user-scope queries using a stream-batch integrated structured query language engine to obtain user-scope data.

[0092] The calculation and storage module is used to calculate and obtain the value of the preprocessed label based on the Aviator expression and the user-scoped data, and store the calculated value of the preprocessed label according to a preset storage mechanism.

[0093] The relevant data information includes the first configuration information and the second configuration information, specifically:

[0094] The first configuration information includes a self-service user management table, a user configuration management table, and an attribute configuration management table. When adding / modifying the first configuration information, a tag preprocessing option is added, and a preprocessing status viewing option is added to the list operation column.

[0095] The second configuration information includes a monitoring table, and the configuration process of the monitoring table includes:

[0096] Configure interactive functions: Add a preset queue field to the list; add a preset queue item to the new / edit monitoring table; add a business data extraction table name field to the list; add a business data extraction table name item to the new / edit monitoring table function.

[0097] Initialization configuration: Connect to the preset queue items of the existing table, update the monitoring table records, and update the business data extraction table name of the existing records.

[0098] In a preferred embodiment of the present invention, the relationship establishment module is specifically used for:

[0099] Read the attribute configuration management table as the basic data, and read the preprocessed tags and expression rules of the preprocessed tags from the self-service population management table and the population configuration management table;

[0100] The expression rules of the preprocessed tags are parsed to obtain atomic tags;

[0101] In the local caching application, the atomic tag key-value pair object, the correspondence key-value pair object between the atomic tag and the preprocessed tag, and the tag rule key-value pair object of the preprocessed tag are established.

[0102] In a preferred embodiment of the present invention, the user-range data acquisition module is specifically used for:

[0103] Based on the key-value pair relationship in the local cache application, query the atomic tags of the preprocessed tags;

[0104] The corresponding monitoring table records are queried using the atomic tags, and the configured data warehouse tool table is read.

[0105] Based on the configuration of the atomic tags and the data warehouse tool table, construct data warehouse tool statements;

[0106] Define the current preprocessed tag status as initializing and generate a user-wide query database;

[0107] The user-scope query database is used to perform a user-scope query to obtain the user-scope data by using a stream-batch integrated structured query language engine.

[0108] In a preferred embodiment of the present invention, the computing and storage module is specifically used for:

[0109] The incremental consumption data message is filtered according to the configuration information of the atomic tag;

[0110] According to the calculation rules of the atomic tags, the filtered message data is read and calculated to generate atomic tag values;

[0111] The system queries the local cache application for the correspondence between the atomic tags and preprocessed tags, compares the corresponding key-value pairs with the atomic tag values, and if the atomic tag values ​​have changed, then...

[0112] Traverse multiple of the preprocessed tags;

[0113] Based on the atomic tags of the preprocessed tags, query the key values ​​of non-message atomic tags in the user-scoped data;

[0114] The Aviator expression of the preprocessed tag is executed to generate the value of the preprocessed tag.

[0115] In a preferred embodiment of the present invention, the computing and storage module is further configured to:

[0116] The step of storing the calculated values ​​of the preprocessed tags according to a preset storage mechanism includes:

[0117] The values ​​of the preprocessed tags are stored using a compressed bitmap mechanism:

[0118] The identifier of the binarized crowd label is stored as a bit in the compressed bitmap object, serialized into a string, and then stored in the key-value storage database.

[0119] Specific limitations regarding the label preprocessing device can be found in the limitations of the label preprocessing method above, and will not be repeated here. Each module in the aforementioned label preprocessing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0120] Example 3

[0121] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a tag preprocessing method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0122] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0124] Step A: Configure relevant data information according to the task type of the preprocessing tags;

[0125] Step B: Read the first configuration information from the relevant data information to obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application;

[0126] Step C: Construct a user-scoped query database based on the key-value pair relationships in the local cache application, and use a stream-batch integrated structured query language engine to perform user-scoped queries to obtain user-scoped data;

[0127] Step D: Based on the Aviator expression and the user-scoped data, calculate and obtain the value of the preprocessed label, and store the calculated value of the preprocessed label according to a preset storage mechanism.

[0128] The relevant data information includes the first configuration information and the second configuration information, specifically:

[0129] The first configuration information includes a self-service user management table, a user configuration management table, and an attribute configuration management table. When adding / modifying the first configuration information, a tag preprocessing option is added, and a preprocessing status viewing option is added to the list operation column.

[0130] The second configuration information includes a monitoring table, and the configuration process of the monitoring table includes:

[0131] Configure interactive functions: Add a preset queue field to the list; add a preset queue item to the new / edit monitoring table; add a business data extraction table name field to the list; add a business data extraction table name item to the new / edit monitoring table function.

[0132] Initialization configuration: Connect to the preset queue items of the existing table, update the monitoring table records, and update the business data extraction table name of the existing records.

[0133] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0134] Read the attribute configuration management table as the basic data, and read the preprocessed tags and expression rules of the preprocessed tags from the self-service population management table and the population configuration management table;

[0135] The expression rules of the preprocessed tags are parsed to obtain atomic tags;

[0136] In the local caching application, the atomic tag key-value pair object, the correspondence key-value pair object between the atomic tag and the preprocessed tag, and the tag rule key-value pair object of the preprocessed tag are established.

[0137] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0138] Based on the key-value pair relationship in the local cache application, query the atomic tags of the preprocessed tags;

[0139] The corresponding monitoring table records are queried using the atomic tags, and the configured data warehouse tool table is read.

[0140] Based on the configuration of the atomic tags and the data warehouse tool table, construct data warehouse tool statements;

[0141] Define the current preprocessed tag status as initializing and generate a user-wide query database;

[0142] The user-scope query database is used to perform a user-scope query to obtain the user-scope data by using a stream-batch integrated structured query language engine.

[0143] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0144] The incremental consumption data message is filtered according to the configuration information of the atomic tag;

[0145] According to the calculation rules of the atomic tags, the filtered message data is read and calculated to generate atomic tag values;

[0146] The system queries the local cache application for the correspondence between the atomic tags and preprocessed tags, compares the corresponding key-value pairs with the atomic tag values, and if the atomic tag values ​​have changed, then...

[0147] Traverse multiple of the preprocessed tags;

[0148] Based on the atomic tags of the preprocessed tags, query the key values ​​of non-message atomic tags in the user-scoped data;

[0149] The Aviator expression of the preprocessed tag is executed to generate the value of the preprocessed tag.

[0150] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0151] The step of storing the calculated values ​​of the preprocessed tags according to a preset storage mechanism includes:

[0152] The values ​​of the preprocessed tags are stored using a compressed bitmap mechanism:

[0153] The identifier of the binarized crowd label is stored as a bit in the compressed bitmap object, serialized into a string, and then stored in the key-value storage database.

[0154] Example 4

[0155] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0156] Step A: Configure relevant data information according to the task type of the preprocessing tags;

[0157] Step B: Read the first configuration information from the relevant data information to obtain the Aviator expression and preprocessing tag, parse the preprocessing tag to obtain the atomic tag, and establish the key-value pair relationship between the preprocessing tag and the atomic tag in the local cache application;

[0158] Step C: Construct a user-scoped query database based on the key-value pair relationships in the local cache application, and use a stream-batch integrated structured query language engine to perform user-scoped queries to obtain user-scoped data;

[0159] Step D: Based on the Aviator expression and the user-scoped data, calculate and obtain the value of the preprocessed label, and store the calculated value of the preprocessed label according to a preset storage mechanism.

[0160] The relevant data information includes the first configuration information and the second configuration information, specifically:

[0161] The first configuration information includes a self-service user management table, a user configuration management table, and an attribute configuration management table. When adding / modifying the first configuration information, a tag preprocessing option is added, and a preprocessing status viewing option is added to the list operation column.

[0162] The second configuration information includes a monitoring table, and the configuration process of the monitoring table includes:

[0163] Configure interactive functions: Add a preset queue field to the list; add a preset queue item to the new / edit monitoring table; add a business data extraction table name field to the list; add a business data extraction table name item to the new / edit monitoring table function.

[0164] Initialization configuration: Connect to the preset queue items of the existing table, update the monitoring table records, and update the business data extraction table name of the existing records.

[0165] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0166] Read the attribute configuration management table as the basic data, and read the preprocessed tags and expression rules of the preprocessed tags from the self-service population management table and the population configuration management table;

[0167] The expression rules of the preprocessed tags are parsed to obtain atomic tags;

[0168] In the local caching application, the atomic tag key-value pair object, the correspondence key-value pair object between the atomic tag and the preprocessed tag, and the tag rule key-value pair object of the preprocessed tag are established.

[0169] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0170] Based on the key-value pair relationship in the local cache application, query the atomic tags of the preprocessed tags;

[0171] The corresponding monitoring table records are queried using the atomic tags, and the configured data warehouse tool table is read.

[0172] Based on the configuration of the atomic tags and the data warehouse tool table, construct data warehouse tool statements;

[0173] Define the current preprocessed tag status as initializing and generate a user-wide query database;

[0174] The user-scope query database is used to perform a user-scope query to obtain the user-scope data by using a stream-batch integrated structured query language engine.

[0175] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0176] The incremental consumption data message is filtered according to the configuration information of the atomic tag;

[0177] According to the calculation rules of the atomic tags, the filtered message data is read and calculated to generate atomic tag values;

[0178] The system queries the local cache application for the correspondence between the atomic tags and preprocessed tags, compares the corresponding key-value pairs with the atomic tag values, and if the atomic tag values ​​have changed, then...

[0179] Traverse multiple of the preprocessed tags;

[0180] Based on the atomic tags of the preprocessed tags, query the key values ​​of non-message atomic tags in the user-scoped data;

[0181] The Aviator expression of the preprocessed tag is executed to generate the value of the preprocessed tag.

[0182] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0183] The step of storing the calculated values ​​of the preprocessed tags according to a preset storage mechanism includes:

[0184] The values ​​of the preprocessed tags are stored using a compressed bitmap mechanism:

[0185] The identifier of the binarized crowd label is stored as a bit in the compressed bitmap object, serialized into a string, and then stored in the key-value storage database.

[0186] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0187] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A label pre-treatment method characterized by, The method comprises: According to the task type of the preprocessing label, configure related data information, which includes first configuration information and second configuration information; Read the first configuration information in the related data information to obtain an Aviator expression and a preprocessing label, parse the preprocessing label to obtain an atomic label, and establish a key-value pair relationship between the preprocessing label and the atomic label in a local cache application; Based on the key-value pair relationship in the local cache application, construct a user range query database, use a stream batch integrated structured query language engine to perform user range query, and obtain user range data; Based on the Aviator expression and the user range data, calculate and obtain the value of the preprocessing label, and store the calculated value of the preprocessing label according to a preset storage mechanism; The first configuration information includes a self-service crowd management table, a crowd configuration management table and an attribute configuration management table, when the first configuration information is added / modified, a label preprocessing option is added, and a preprocessing state viewing option is added in a list operation column; The second configuration information includes a monitoring table, and the configuration process of the monitoring table comprises: Configure an interactive function: add a preset queue field on the list; add / edit a monitoring table, add a preset queue item; add a business number table name field on the list; add / edit a monitoring table function, and add a business number table name item; Initialize configuration: interface the preset queue item of the existing table, update the monitoring table record, and update the business number table name of the existing record; The storage of the value of the preprocessing label according to the preset storage mechanism comprises: A compression bitmap mechanism is used to store the value of the preprocessing label: The identifier of the binary crowd label is stored as a bit in a bitmap object, and after being serialized into a string, it is stored in a key-value storage database.

2. The label pretreatment method according to claim 1, characterized in that, The reading of the first configuration information in the related data information to obtain an Aviator expression and a preprocessing label, the parsing of the preprocessing label to obtain an atomic label, and the establishment of a key-value pair relationship between the preprocessing label and the atomic label in a local cache application comprise: Read the attribute configuration management table as basic data, read the preprocessing label in the self-service crowd management table and the crowd configuration management table, and read the expression rule of the preprocessing label; Parse the expression rule of the preprocessing label to obtain an atomic label; In the local cache application, establish the atomic label key-value pair object, establish the corresponding relationship key-value pair object between the atomic label and the preprocessing label, and establish the label rule key-value pair object of the preprocessing label.

3. The label pretreatment method according to claim 1, characterized by, Based on the key-value pair relationship in the local cache application, construct a user range query database, use a stream batch integrated structured query language engine to perform user range query, and obtain user range data, which comprises: Query the atomic label of the preprocessing label based on the key-value pair relationship in the local cache application; Query the corresponding monitoring table record through the atomic label, and read the configured data warehouse tool table; Based on the configuration of the atomic label and the data warehouse tool table, construct a data warehouse tool statement; define a state of a current preprocessing label as being in initialization, and generate a user-scoped query database; perform a user-scoped query on the user-scoped query database using a stream-batch integrated structured query language engine to obtain the user-scoped data.

4. The label pretreatment method according to claim 1, characterized by, the calculating and obtaining of the value of the preprocessing label based on the Aviator expression and the user-scoped data includes: consume incremental data messages, and perform message screening according to configuration information of the atomic label; read the screened message data and perform calculation according to a calculation rule of the atomic label to generate an atomic label value; query a corresponding relationship key-value pair object of the atomic label and the preprocessing label from the local cache application and compare the atomic label value, and if the atomic label value has changed, then, traverse a plurality of preprocessing labels.

5. The label pretreatment method according to claim 4, characterized in that, further include: query a key of a non-message atomic label in the user-scoped data according to an atomic label of the preprocessing label; execute an Aviator expression of the preprocessing label to generate a value of the preprocessing label.

6. A label pre-processing apparatus for implementing the label pre-processing method as claimed in claim 1, characterized by The apparatus includes: an information configuration module configured to configure related data information according to a task type of a preprocessing label; a relationship establishment module configured to read first configuration information in the related data information, obtain an Aviator expression and a preprocessing label, parse the preprocessing label to obtain an atomic label, and establish a key-value pair relationship between the preprocessing label and the atomic label in a local cache application; a user-scoped data acquisition module configured to construct a user-scoped query database based on the key-value pair relationship in the local cache application, perform a user-scoped query using a stream-batch integrated structured query language engine, and obtain user-scoped data; a calculation storage module configured to calculate and obtain a value of the preprocessing label based on the Aviator expression and the user-scoped data, and store the calculated value of the preprocessing label according to a preset storage mechanism.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images