Data tag generation method and device and electronic equipment

Abstract

The embodiment of the invention provides a data tag generation method and device and electronic equipment, and relates to the technical field of data processing. The method comprises the following steps: receiving a data change notification sent by a message queue, and analyzing notification content carried by the data change notification to obtain data change information; according to a preset label screening rule and the data change information, a target label list is determined, and target labels contained in the target label list are labels which correspond to the data change notification and need to be calculated; and executing a tag calculation task corresponding to the target tag list to obtain at least one target tag value, and storing the at least one target tag value in a tag result storage list in a column type storage mode. In this way, time driving is replaced by event driving, and the real-time performance of data label calculation is improved; invalid calculation of irrelevant labels can be avoided, the calculation efficiency is improved, and resources are saved; and the label result storage table supports one-time query to obtain all labels of the entity, so that the query performance is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data label generation method and device and electronic equipment. BACKGROUND

[0002] A data label refers to a label or mark used to identify, classify, describe or explain each data point or data set in a data set, providing contextual information about the data, which helps users understand the meaning and purpose of the data.

[0003] In the prior art, a fixed-batch computing method and a full-scan computing method are used for data label calculation.

[0004] The fixed-batch computing method specifically refers to that the system performs a batch label calculation task at fixed time intervals (such as every hour or every day). However, the fixed-batch computing method uses a passive timing trigger mechanism and has timeliness problems, resulting in large data label calculation delays and failing to meet the real-time requirements of businesses.

[0005] The full-scan computing method specifically refers to scanning all data for data label calculation each time data label calculation is performed. This method can ensure the accuracy of the results, but the computing resources are consumed greatly, and even if only a small amount of data changes, all labels need to be recalculated, resulting in a large amount of invalid calculation, low efficiency and serious resource waste. SUMMARY

[0006] The embodiments of the present application provide a data label generation method, device and electronic equipment to improve the real-time performance and efficiency of data label calculation and save resources.

[0007] In a first aspect, the embodiments of the present application provide a data label generation method, which comprises: receiving a data change notification sent by a message queue and parsing the notification content carried by the data change notification to obtain data change information; determining a target label list according to a preset label filtering rule and the data change information, wherein the target labels included in the target label list are labels corresponding to the data change notification that need to be calculated; performing a label calculation task corresponding to the target label list to obtain at least one target label value, and storing the at least one target label value in a label result storage table using a columnar storage method.

[0008] In an optional embodiment, the data change notification sent by the message queue comprises: real-time monitoring of a message queue from a real-time data warehouse through a message listener to receive a data change notification, wherein the data change notification is a change message generated when data in the real-time data warehouse changes.

[0009] In an optional embodiment, the target label list is determined according to the preset label screening rule and the data change information, and includes the following steps: The candidate label list is screened from the initial label list based on the first label screening condition and the data change information. The target label list is screened from the candidate label list based on the second label screening condition and the data change information.

[0010] In an optional embodiment, the candidate label list is screened from the initial label list based on the first label screening condition and the data change information, and includes the following steps: According to the association relationship between each initial label and the data table, each candidate label associated with the data table identifier contained in the data change information in the initial label list is determined, and each candidate label is taken as the candidate label list.

[0011] In an optional embodiment, the target label list is screened from the candidate label list based on the second label screening condition and the data change information, and includes the following steps: For each candidate label contained in the candidate label list, the following operations are respectively performed: if the current field value of at least one change field contained in the data change information meets the label calculation condition corresponding to one candidate label, one candidate label is taken as the target label. The obtained at least one target label is taken as the target label list.

[0012] In an optional embodiment, the label calculation task corresponding to the target label list is executed, and includes the following steps: The label calculation task corresponding to the target label list is asynchronously executed by using an asynchronous parallel processing architecture.

[0013] In an optional embodiment, the first calculation time is embedded in each target label value, and the first calculation time is the calculation time of the corresponding target label value. The at least one target label value is stored into the label result storage table by using the columnar storage mode, and includes the following steps: For the at least one target label, the following operations are respectively performed: The first calculation time of one target label is compared with the second calculation time of the stored label value corresponding to one target label value in the label result storage table. If the first calculation time of one target label is greater than the second calculation time, one target label value is stored into the label result storage table by using the columnar storage mode.

[0014] In an optional embodiment, the primary key of the label result storage table is a composite primary key, and the composite primary key includes the entity identifier, the business domain and the entity key.

[0015] In a second aspect, the embodiments of the present application further provide a data label generation device, the device comprising: The message receiving module is configured to receive the data change notification sent by the message queue, and parse the notification content carried by the data change notification to obtain data change information. The determining module is configured to determine a target label list according to the preset label screening rule and the data change information, wherein the target labels included in the target label list are labels corresponding to the data change notification that need to be calculated. The computing and storing module is configured to execute a label calculation task corresponding to the target label list, obtain at least one target label value, and store the at least one target label value in the label result storage table in a columnar storage manner.

[0016] In an optional embodiment, when receiving the data change notification sent by the message queue, the message receiving module is further configured to: The message listener is configured to listen to the message queue from the real-time data warehouse in real time to receive the data change notification, wherein the data change notification is a change message generated when the data in the real-time data warehouse changes.

[0017] In an optional embodiment, when determining the target label list according to the preset label screening rule and the data change information, the determining module is further configured to: The determining module is configured to screen a candidate label list from the initial label list based on the first label screening condition and the data change information. The determining module is configured to screen the target label list from the candidate label list based on the second label screening condition and the data change information.

[0018] In an optional embodiment, when screening the candidate label list from the initial label list based on the first label screening condition and the data change information, the determining module is further configured to: The determining module is configured to determine each candidate label in the initial label list that is associated with the data table identifier included in the data change information according to the association relationship between each initial label and the data table, and take each candidate label as the candidate label list.

[0019] In an optional embodiment, when screening the target label list from the candidate label list based on the second label screening condition and the data change information, the determining module is further configured to: The determining module is configured to perform the following operations on each candidate label included in the candidate label list: if the current field value of at least one changed field included in the data change information meets the label calculation condition corresponding to one candidate label, then take one candidate label as a target label. The determining module is configured to take the obtained at least one target label as the target label list.

[0020] In an optional embodiment, when performing the tag calculation task corresponding to the target tag list, the calculation and storage module is further configured to: An asynchronous parallel processing architecture is adopted to asynchronously execute the tag calculation tasks corresponding to the target tag list.

[0021] In one optional embodiment, a first computation time is embedded in each target label value, the first computation time being the computation time of the corresponding target label value; When storing at least one target label value in the label result storage table using a columnar storage method, the calculation and storage module is also used for: For at least one target label, perform the following operations respectively: Compare the first calculation time of a target label with the second calculation time of the stored label value corresponding to a target label value in the label result storage table; If the first calculation time for a target label is longer than the second calculation time, then the target label value will be stored in the label result storage table using columnar storage.

[0022] In one optional embodiment, the primary key of the tag result storage table is a composite primary key, which includes: entity identifier, business domain, and entity key.

[0023] Thirdly, embodiments of this application also provide an electronic device, including: Processor; and Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the data tag generation method as described in the first aspect.

[0024] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the data tag generation method as described in the first aspect.

[0025] Fifthly, this application provides a computer program product that, when invoked by a computer, causes the computer to execute the data tag generation method steps as described in the first aspect.

[0026] The beneficial effects of this application are as follows: In the method for generating a data label provided in the embodiments of the present application, a data change notification sent by a message queue is received, and notification content carried by the data change notification is parsed to obtain data change information. Then, according to a preset label screening rule and the data change information, a target label list is determined, the target label in the target label list is a label corresponding to the data change notification and needing to be calculated. Finally, a label calculation task corresponding to the target label list is executed to obtain at least one target label value, and the at least one target label value is stored in a label result storage table in a columnar storage mode. In this way, the real-time performance of data label calculation can be improved, event driving is used to replace time driving, and a change from passive calculation to active calculation is realized. Compared with a traditional timing polling mode, a response speed of milliseconds can be realized. According to the preset label screening rule and the data change information, the target label list is determined, and invalid calculation of irrelevant labels can be avoided, so that the calculation efficiency is significantly improved and resources are saved. The label result is stored in the columnar storage mode, and the label result storage table supports one-time query to obtain all labels of an entity, so that the query performance is greatly improved, and the throughput is significantly improved, especially in a high-concurrency query scenario.

[0027] In addition, other features and advantages of the present application will be set forth in the description that follows, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings described here are used to provide further understanding of the present application and form a part of the present application. However, the present application is not limited by the inappropriate limitations in the drawings. Figure 1 An optional system architecture schematic diagram applicable to the embodiments of the present application; Figure 2 An implementation flowchart of a data label generation method provided by the embodiments of the present application; Figure 3 A label result storage table schematic diagram provided by the embodiments of the present application; Figure 4 A data label generation system architecture schematic diagram provided by the embodiments of the present application; Figure 5 A structure schematic diagram of a data label generation device provided by the embodiments of the present application; Figure 6 A structure schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0029] Embodiments of the present application will be described in more detail with reference to the drawings. While several embodiments of the application are shown in the drawings, it is not intended that the application be limited to the embodiments shown, and it is to be understood that the application can be carried out by way of various forms.

[0030] It should be understood that each of the steps in the method embodiments of the present application can be performed in a different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present application is not limited in this respect.

[0031] The term "comprising" and variations thereof as used herein are used inclusively, i.e., "comprising but not limited to." The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related terms are defined in the following description. It should be noted that references in the application to "first", "second", and the like indicate different apparatuses, modules, or units, and do not imply that the functions of these apparatuses, modules, or units are limited to the order or interdependence implied by the terminology.

[0032] It should be noted that the terms "one", "multiple", and the like used in the present application are illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0033] The names of the messages or information exchanged between the devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0034] Some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0035] (1) Message queue: an asynchronous communication mechanism for delivering messages in a distributed system to achieve system decoupling.

[0036] (2) Columnar storage: a storage method for organizing data by column, facilitating compression and query optimization.

[0037] (3) Business domain: a data classification method according to business type, such as transaction domain, credit domain, and marketing domain.

[0038] Based on the above explanations of terms and related terminology, the design concept of the embodiments of this application will be briefly introduced below: Data labels are tags or markers used to identify, classify, describe, or interpret individual data points or datasets within a dataset. They provide contextual information about the data and help users understand its meaning and purpose.

[0039] In existing technologies, data tag calculation is performed using timed batch calculation and full scan calculation methods, while the tag results are stored using traditional relational database storage methods.

[0040] The scheduled batch calculation method involves the system executing batch tag calculation tasks at fixed time intervals (such as hourly or daily). However, this method, which uses a passive timed triggering mechanism, suffers from timeliness issues, resulting in significant delays in data tag calculation and failing to meet the real-time requirements of the business.

[0041] The full-scan calculation method involves scanning all data for label calculation each time a label is calculated. While this method ensures accuracy, it consumes enormous computational resources. Even if only a small amount of data changes, all labels need to be recalculated, making it impossible to identify which labels truly require recalculation. This results in a large amount of invalid calculations, low efficiency, and significant resource waste.

[0042] Furthermore, traditional relational databases use a standard row-based table structure to store tag results, with each tag as a row. While this structure is simple, it performs poorly in high-concurrency query scenarios, requiring multiple database accesses to retrieve multiple tags for a single entity.

[0043] In view of this, embodiments of this application propose a method, apparatus, and electronic device for generating data tags, which may specifically include: receiving a data change notification sent by a message queue, parsing the notification content carried by the data change notification to obtain data change information, then determining a target tag list according to preset tag filtering rules and data change information, wherein the target tags included in the target tag list are the tags that need to be calculated corresponding to the data change notification, and finally executing the tag calculation task corresponding to the target tag list to obtain at least one target tag value, and storing at least one target tag value in a tag result storage table in a columnar storage manner.

[0044] In the above manner, the data change notification sent by the message queue is received, that is, the real-time triggering mechanism of message driving, which can improve the real-time performance of data tag calculation. The event driving is used to replace the time driving, and the change from passive calculation to active calculation is realized. Compared with the traditional timing polling mode, the response speed of millisecond level can be realized. According to the preset tag screening rule and the data change information, the target tag list is determined, which can avoid invalid calculation of irrelevant tags, significantly improve the calculation efficiency and save resources. The tag result is stored in the column storage mode, the tag result storage table supports querying all tags of an entity at a time, which greatly improves the query performance, especially in the high concurrency query scene, the throughput is significantly improved.

[0045] In particular, the preferred embodiments of the present application are described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0046] Referring to Figure 1 As shown in the figure, it is an optional system architecture diagram applicable to the embodiments of the present application. The system architecture can include: terminal device (101a, 101b) and server 102. The terminal device (101a, 101b) and the server 102 can exchange information through a communication network, wherein the communication network can adopt a communication mode including a wireless communication mode and a wired communication mode. Exemplarily, the terminal device (101a, 101b) can access the network through a cellular mobile communication technology and communicate with the server 102. Wherein, the cellular mobile communication technology, such as, includes the fifth generation mobile communication (5th generation mobile networks, 5G) technology or the next generation mobile communication technology. Optionally, the terminal device (101a, 101b) can access the network through a short distance wireless communication mode and communicate with the server 102. Wherein, the short distance wireless communication mode, such as, includes wireless fidelity (wireless fidelity, Wi-Fi) technology.

[0047] The number of communication devices involved in the above system architecture is not limited in the embodiments of the present application. For example, the above system architecture can include more terminal devices, or can include fewer terminal devices, or can also include other network devices. For example, Figure 1 As shown in the figure, only the terminal device (101a, 101b) and the server 102 are described as an example, and the functions of the above communication devices and their respective functions are briefly introduced below.

[0048] The terminal device (101a, 101b) is a device that can provide voice and / or data connectivity for a user, and can be a device that supports wired and / or wireless connection modes.

[0049] For example, the terminal device (101a, 101b) can include, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a palm computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal device in industrial control, a wireless terminal device in unmanned driving, a wireless terminal device in smart power grids, a wireless terminal device in transportation safety, a wireless terminal device in smart cities, or a wireless terminal device in smart homes, etc.

[0050] In addition, the terminal device (101a, 101b) can be installed with a related client, which can be software such as an application (APP), a browser, a short video software, etc., or a webpage, an applet, etc. It should be noted that the terminal device (101a, 101b) in the embodiments of the present application can be a client related to the generation of a data tag.

[0051] The server 102 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0052] The data tag generation method provided by the exemplary embodiments of the present application will be described below in combination with the above-mentioned system architecture and in reference to the accompanying drawings. It should be noted that the above-mentioned system architecture is only shown for the purpose of facilitating understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

[0053] Referring to Figure 2 As shown in the figure, it is an implementation flowchart of a data tag generation method provided by the embodiments of the present application, and the execution subject is taken as an example of a server. The specific implementation process of the method is as follows: S20: receiving the data change notification sent by the message queue, and parsing the notification content carried by the data change notification to obtain data change information.

[0054] The data change information includes a change type, a data table identifier, a current field value of a changed field, a primary key value, a change time, and the like.

[0055] In the embodiment of the application, a message listener is used to listen to a message queue from a real-time data warehouse in real time, and receive data change notifications.

[0056] The data change notification is a change message generated when data in the real-time data warehouse is changed.

[0057] In the embodiment of the application, when data in the real-time data warehouse is changed, a data change notification is automatically generated by using a database log analysis technology, and the data change notification is asynchronously sent to a server through a message queue, and then the server receives the data change notification in real time through a message listener.

[0058] In this way, the decoupling of a data source (the real-time data warehouse) and label calculation is achieved through the integration of the message listener and the message queue. The message queue serves as a buffer layer, can cope with traffic peaks, prevent data loss, and improve the fault tolerance of the system. The real-time performance of label calculation is improved, and the reliable delivery of data change notifications is ensured.

[0059] In addition, in the embodiment of the application, when the notification content carried by the data change notification is parsed, parsing of multiple message formats (such as JSON and Avro) is supported, so that the method can adapt to the implementation of different real-time data warehouses.

[0060] Optionally, in the embodiment of the application, the time characteristics of the received multiple data change notifications can be analyzed to determine notification traffic in different time periods, and the computing resources required in different time periods can be determined according to the notification traffic in different time periods, so as to use different computing resources in different time periods, ensure the efficiency of label calculation, and reduce resource utilization. Specifically, by using machine learning, the notification traffic in different time periods in the future is learned according to the time characteristics of historical data change notifications. If the notification traffic in a time period is low, the time period is allocated with low computing resources. If the notification traffic in a time period is high, the time period is allocated with high computing resources.

[0061] S21: determining a target label list according to a preset label filtering rule and data change information.

[0062] The target label in the target label list is a label that needs to be calculated corresponding to the data change notification. The preset label filtering rule includes a first label filtering condition and / or a second label filtering condition. The first label filtering condition is a data table level label filtering condition, and the second label filtering condition is a field level label filtering condition.

[0063] Optionally, in the embodiments of the present application, the target label list can be determined according to the first label filtering condition and the data change information, the target label list can also be determined according to the second label filtering condition and the data change information, and the target label list can also be determined according to the first label filtering condition, the second label filtering condition and the data change information. In order to achieve the filtering mechanism from coarse to fine, the embodiments of the present application preferably determine the target label list according to the first label filtering condition, the second label filtering condition and the data change information.

[0064] Optionally, in the embodiments of the present application, a possible implementation is provided for determining the target label list according to the first label filtering condition and the second label filtering condition and the data change information, and the specific steps are as follows: S210: filtering a candidate label list from the initial label list based on the first label filtering condition and the data change information.

[0065] In the embodiments of the present application, each candidate label associated with the data table identifier contained in the data change information in the initial label list is determined according to the association relationship between each initial label and the data table, and each candidate label is taken as the candidate label list.

[0066] The data table identifier can be a data table name or a data table number, and the embodiments of the present application do not limit this.

[0067] For example, it is assumed that the data table identifier contained in the data change information is the identifier of the user transaction table, and the labels associated with the user transaction table are "user monthly transaction total amount" and "whether it is a large transaction user", then the candidate label list includes "user monthly transaction total amount" and "whether it is a large transaction user".

[0068] S211: filtering a target label list from the candidate label list based on the second label filtering condition and the data change information.

[0069] In the embodiments of the present application, for each candidate label contained in the candidate label list, the following operations are performed: if the current field value of at least one changed field contained in the data change information meets the label calculation condition corresponding to one candidate label, then one candidate label is taken as a target label, and finally the obtained at least one target label is taken as the target label list.

[0070] Different label calculation conditions are set for different labels to achieve accurate triggering, and the label calculation condition can be configured as a numerical comparison, an enumeration matching or a logical combination.

[0071] For example, assuming that a candidate label is "whether it is a large transaction user", the label calculation condition of "whether it is a large transaction user" is that the transaction type field is equal to "large transfer", at this time, the at least one changed field contained in the data change information includes the transaction type field, and the current field value of the transaction type field is large transfer, and then it is determined that "whether it is a large transaction user" is a target label. Assuming that a candidate label is "user monthly transaction total amount", the label calculation condition of "user monthly transaction total amount" is that the transaction amount field is changed, at this time, the at least one changed field contained in the data change information includes the transaction amount field, and then it is determined that "user monthly transaction total amount" is a target label.

[0072] In this way, the screening process is divided into two levels, forming a filtering mechanism from coarse to fine. The first level quickly excludes irrelevant labels, which can quickly reduce the range of candidate labels from the full amount of labels to a subset related to a specific table, laying the foundation for subsequent high-cost label calculation condition judgment, avoiding the overhead of traversing all labels. The second level accurately determines whether to calculate, significantly reducing unnecessary calculation tasks. Avoiding full scan or invalid triggering, saving CPU and memory resources, while ensuring that only labels that are truly affected by data changes will be triggered for calculation, improving the relevance and accuracy of label updates.

[0073] In addition, it is worth noting that in the embodiments of the present application, the priority of the label calculation task can also be configured. Optionally, the priority of the label calculation task corresponding to each label can be determined according to the influence degree of the label generation speed of each label on the business, and the priority of the label with a greater influence degree of the label generation speed on the business is higher; optionally, the priority of the label calculation task corresponding to each label can also be determined according to the importance of each data table, and the priority of the label corresponding to the data table with higher importance is higher.

[0074] S22: Perform the label calculation task corresponding to the target label list, obtain at least one target label value, and store the at least one target label value in the label result storage table in a columnar storage manner.

[0075] In the embodiments of the present application, an asynchronous parallel processing architecture is adopted to asynchronously execute the label calculation task corresponding to the target label list.

[0076] The asynchronous parallel processing architecture can be a thread pool supporting parallel calculation of multiple target labels.

[0077] In this way, the blocking problem of synchronous serial processing is avoided, and the processing capacity of the system when a large amount of data flows in is significantly improved, ensuring that the real-time performance is not dragged down by the calculation task.

[0078] Optionally, in the embodiment of the present application, when the label calculation task corresponding to the target label list is executed, a retry mechanism and an error handling mechanism can be introduced to ensure reliable execution of the label calculation task. The asynchronous mechanism decouples the label calculation task from the message receiving process, so that even if the calculation module has a short delay, the continuity of message receiving will not be affected. In combination with the retry mechanism and the error handling mechanism, the fault tolerance and reliability of the system are improved.

[0079] In the embodiment of the present application, the primary key of the label result storage table is a composite primary key, which includes an entity identifier, a business domain, and an entity key. The entity identifier is used to distinguish different business entities (such as users, orders, etc.), the business domain is used to classify labels of different business types (such as transaction domain, credit domain, marketing domain, etc.), and the entity key is used to specifically locate entity instances (such as monthly transaction amount, consumption frequency, etc.). Each label corresponds to a data column.

[0080] Referring to FIG. 2, Figure 3 FIG. 2 is a schematic diagram of a label result storage table in the embodiment of the present application, in which "001" and "002" are entity identifiers, "transaction domain" and "operation domain" are business domains, "v1", "v2",..., "v200" are entity keys, "x1", "y2",..., "z1" are label values, "x2", "y2",..., "z2" are label values, and "x3", "y3",..., "z3" are label values.

[0081] In this way, the composite primary key structure of the label result storage table enables efficient organization of data in the label result storage table according to entities and business domains. In combination with columnar storage, all labels of an entity in a specific business domain can be obtained through one query, greatly reducing database access operations.

[0082] Optionally, in the embodiment of the present application, a first calculation time is embedded in each target label value, and the first calculation time is the calculation time of the corresponding target label value.

[0083] In which, the storage format of each target label value is "label value ## first calculation time".

[0084] Optionally, in the embodiment of the present application, when at least one target label value is stored in the label result storage table in the columnar storage manner, the following operations are respectively performed for at least one target label: S220: comparing the first calculation time of one target label with the second calculation time of a stored label value corresponding to one target label value in the label result storage table.

[0085] S221: if the first calculation time of one target label is greater than the second calculation time, then one target label value is stored in the label result storage table in the columnar storage manner.

[0086] In the embodiments of the present application, it is judged whether the first calculation time of a target label is greater than the second calculation time. If yes, the value of a target label is stored in the label result storage table in a columnar storage mode. Otherwise, the stored label value is retained.

[0087] In this way, by embedding the calculation time and comparing the time sequence, it is ensured that the latest calculated label value always covers the old value, avoiding the data inconsistency problem caused by network delay or out-of-order in a distributed environment, and being suitable for high-precision data scenes such as finance. Moreover, the calculation time is embedded in the label value itself, simplifying the consistency management.

[0088] Based on the above embodiments, refer to Figure 4 Fig. 1 shows a system architecture diagram of a data label generation system provided in the embodiments of the present application. For example, refer to Figure 4 The data label generation system includes a real-time data warehouse 401, a message queue 402, a message receiving module 403, a trigger judgment module 404, a calculation execution module 405, a storage management module 406, and a label result storage table 407.

[0089] The message receiving module 403 is responsible for listening to the message queue from the real-time data warehouse and receiving data change notifications. This module realizes decoupling with the data source through the message queue subscription mechanism, supports parsing and processing of multiple message formats. The sub-modules include a message listener, a message parser, and a format converter.

[0090] The trigger judgment module 404 judges whether the received data change information needs to trigger label calculation according to the pre-configured trigger rules (including preset label filtering rules and priority rules). This module contains a rule matcher, a condition evaluator, and a priority sorter, which can intelligently identify the target label list and determine the calculation priority.

[0091] The calculation execution module 405 is responsible for executing specific label calculation tasks. This module adopts an asynchronous parallel processing architecture, including a task scheduler, an asynchronous calculator, and a result encapsulator, which can efficiently process a large number of label calculation tasks.

[0092] The storage management module 406 is responsible for persisting the calculation results (label values) according to the optimized storage structure. This module contains a storage engine, a version controller, and a consistency manager, which supports high-concurrency write and query operations.

[0093] The modules cooperate with each other through asynchronous message passing and shared storage to form a complete real-time data label generation system.

[0094] Further, based on the same technical concept, the application provides a data label generation device for implementing the method flow of the application. Figure 5 As shown in FIG. 5, the data label generation device 500 can include a message receiving module 501, a determination module 502, and a calculation and storage module 503. The message receiving module 501 is configured to receive a data change notification sent by a message queue and parse the notification content carried by the data change notification to obtain data change information. The determination module 502 is configured to determine a target label list according to a preset label screening rule and the data change information, wherein the target labels included in the target label list are labels corresponding to the data change notification that need to be calculated. The calculation and storage module 503 is configured to perform a label calculation task corresponding to the target label list to obtain at least one target label value, and store the at least one target label value in a columnar storage manner in a label result storage table.

[0095] In an optional embodiment, when receiving the data change notification sent by the message queue, the message receiving module 501 is further configured to: The message receiving module 501 is further configured to listen to the message queue from the real-time data warehouse in real time through a message listener to receive the data change notification, wherein the data change notification is a change message generated when the data in the real-time data warehouse changes.

[0096] In an optional embodiment, when determining the target label list according to the preset label screening rule and the data change information, the determination module 502 is further configured to: The determination module 502 is further configured to screen a candidate label list from the initial label list based on a first label screening condition and the data change information. The determination module 502 is further configured to screen the target label list from the candidate label list based on a second label screening condition and the data change information.

[0097] In an optional embodiment, when screening the candidate label list from the initial label list based on the first label screening condition and the data change information, the determination module 502 is further configured to: The determination module 502 is further configured to determine each candidate label in the initial label list that is associated with a data table identifier included in the data change information according to an association relationship between each initial label and the data table, and take each candidate label as the candidate label list.

[0098] In an optional embodiment, when screening the target label list from the candidate label list based on the second label screening condition and the data change information, the determination module 502 is further configured to: For each candidate label included in the candidate label list, the following operations are performed respectively: if the current field value of at least one changed field included in the data change information satisfies the label calculation condition corresponding to one candidate label, the one candidate label is taken as a target label; The obtained at least one target label is taken as a target label list.

[0099] In an optional embodiment, when performing the label calculation task corresponding to the target label list, the calculation and storage module 503 is further configured to: The label calculation task corresponding to the target label list is performed asynchronously in an asynchronous parallel processing architecture.

[0100] In an optional embodiment, the first calculation time is embedded in each target label value, and the first calculation time is the calculation time of the corresponding target label value; When the at least one target label value is stored into the label result storage table in the column storage mode, the calculation and storage module 503 is further configured to: For each target label, the following operations are performed respectively: The first calculation time of one target label is compared with the second calculation time of the stored label value corresponding to one target label value in the label result storage table. If the first calculation time of one target label is greater than the second calculation time, one target label value is stored into the label result storage table in the column storage mode.

[0101] In an optional embodiment, the primary key of the label result storage table is a composite primary key, and the composite primary key includes the entity identifier, the business domain and the entity key.

[0102] Based on the description of the method embodiments and the device embodiments, the exemplary embodiments of the present application further provide an electronic device, including at least one processor, and a memory connected with the at least one processor in communication. The memory stores a computer program which can be executed by the at least one processor, and the computer program is used to make the electronic device execute the method according to the embodiments of the present application when being executed by the at least one processor.

[0103] The embodiments of the present application further provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program is used to make a computer execute the method according to the embodiments of the present application when being executed by a processor of the computer.

[0104] The embodiments of the present application further provide a computer program product including a computer program, wherein the computer program is used to make a computer execute the method according to the embodiments of the present application when being executed by a processor of the computer.

[0105] Refer toFigure 6 As shown, a structural block diagram of an electronic device 600 that can serve as a server or client of the present application will now be described, which is an example of a hardware device that can be applied to aspects of the present application. The electronic device is intended to represent a wide variety of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent a wide variety of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0106] As shown, the electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604. Figure 6

[0107] Various components in the electronic device 600 are connected to the I / O interface 605, including an input unit 606, an output unit 607, the storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 607 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a worldwide interoperability for microwave access (WiMax) device, a cellular communication device, and / or the like. ​

[0108] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processors, controllers, microcontrollers, and the like. The computing unit 601 performs various methods and processes described above. For example, in some embodiments, the above-described method of generating a data tag can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 can be configured to perform the above-described method of generating a data tag by any other appropriate means, such as by means of firmware.

[0109] Program code for carrying out the methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be embodied in whole or in part within a machine, executed by a machine, executed partially by a machine and partially by a remote machine, or entirely by a remote machine or server.

[0110] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store program code for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include a lined paper, a portable computer diskette, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] As used in this application, the terms "machine-readable medium" and "computer- readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0112] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0113] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0114] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0115] Moreover, it is to be understood that the above-described embodiments are only exemplary of the application, and that a person skilled in the art can make various changes and modifications without departing from the scope of the application. Therefore, the scope of the application should be determined by the following claims, rather than by the detailed description, and equivalents to the claims should be covered.

Claims

1. A method for generating data tags, characterized in that, include: Receive data change notifications sent from the message queue, and parse the notification content carried by the data change notifications to obtain data change information; Based on the preset tag filtering rules and the data change information, a target tag list is determined. The target tags included in the target tag list are the tags that need to be calculated corresponding to the data change notification. Execute the tag calculation task corresponding to the target tag list to obtain at least one target tag value, and store the at least one target tag value in the tag result storage table in a columnar storage manner.

2. The method as described in claim 1, characterized in that, The data change notification sent by the message queue includes: The system listens to the message queue from the real-time data warehouse in real time via a message listener and receives data change notifications, which are change messages generated when data in the real-time data warehouse changes.

3. The method as described in claim 1, characterized in that, The step of determining the target tag list based on preset tag filtering rules and the data change information includes: Based on the first tag filtering criteria and the data change information, a candidate tag list is filtered from the initial tag list; Based on the second tag filtering criteria and the data change information, a target tag list is filtered from the candidate tag list.

4. The method as described in claim 3, characterized in that, The step of filtering a candidate tag list from the initial tag list based on the first tag filtering criteria and the data change information includes: Based on the association between each initial label and the data table, each candidate label in the initial label list that is associated with the data table identifier contained in the data change information is determined, and each candidate label is used as a candidate label list.

5. The method as described in claim 3, characterized in that, The step of filtering the target tag list from the candidate tag list based on the second tag filtering criteria and the data change information includes: For each candidate tag included in the candidate tag list, the following operations are performed respectively: if the current field value of at least one changed field included in the data change information satisfies the tag calculation condition corresponding to the candidate tag, then the candidate tag is taken as the target tag; At least one target label obtained is used as a target label list.

6. The method as described in claim 1, characterized in that, The execution of the tag calculation task corresponding to the target tag list includes: An asynchronous parallel processing architecture is adopted to asynchronously execute the tag calculation tasks corresponding to the target tag list.

7. The method as described in claim 1, characterized in that, Each target label value is embedded with a first calculation time, which is the calculation time of the corresponding target label value; The step of storing the at least one target label value in a columnar storage method to a label result storage table includes: For each of the at least one target label, perform the following operations: Compare the first calculation time of a target label with the second calculation time of the stored label value corresponding to the target label value in the label result storage table; If the first calculation time of a target label is greater than the second calculation time, then the target label value is stored in the label result storage table using a columnar storage method.

8. The method as described in claim 1, characterized in that, The primary key of the tag result storage table is a composite primary key, which includes: entity identifier, business domain, and entity key.

9. A data tag generation apparatus, characterized in that, include: The message receiving module is used to receive data change notifications sent by the message queue, and parse the notification content carried by the data change notification to obtain data change information; The determination module is used to determine a target tag list based on preset tag filtering rules and the data change information. The target tags included in the target tag list are the tags that need to be calculated corresponding to the data change notification. The calculation and storage module is used to execute the tag calculation task corresponding to the target tag list, obtain at least one target tag value, and store the at least one target tag value in the tag result storage table in a columnar storage manner.

10. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1-8.

Images