User tag data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a user tag data processing method and device, computer equipment and a storage medium. The user tag data processing method is applied to a user tag data processing system, and comprises the following steps: converting initial user tag data acquired under different data access channels to obtain multiple pieces of standard user tag data under a preset data structure; according to the multiple pieces of standard user tag data and operation state indexes of the user tag data processing system, health feature values of the user tag data processing system are detected, and the health feature values represent the data processing capacity of the user tag data processing system; and processing the plurality of pieces of standard user tag data according to the health feature values. By adopting the method, the processing effect of processing the user tag data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, apparatus, computer equipment, and storage medium for processing user tag data. Background Art

[0002] With the acceleration of the digitalization process, the scale of customers reached by enterprises has grown exponentially. In this context, the systematic labeling of diverse user resources has become the core foundation of refined operations. Among them, user labels are abstract descriptions of user characteristics in the form of structured fields by integrating multi-dimensional data such as user basic attributes, behavioral data, and preference characteristics.

[0003] At present, in the process of processing user tag data to obtain user tags, user tag data is usually collected by interface reporting, and the collected user tag data is processed based on a timed trigger method to finally obtain user tags. However, due to the variety of source channels of user tag data, the richness of user tag data is insufficient. At the same time, due to the differences in data traffic and data type distribution in different processing scenarios, the timeliness and accuracy of tag calculation are insufficient. Therefore, the current processing effect of user tag data is poor. Summary of the Invention

[0004] Based on this, it is necessary to provide a user tag data processing method, device, computer equipment and storage medium to improve the processing effect of user tag data in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a user tag data processing method, which is applied to a user tag data processing system, and the method includes:

[0006] Convert the initial user tag data collected under different data access channels to obtain multiple standard user tag data under a preset data structure;

[0007] detecting a health characteristic value of the user tag data processing system based on the plurality of standard user tag data and an operating status indicator of the user tag data processing system, wherein the health characteristic value represents a data processing capability of the user tag data processing system;

[0008] The plurality of pieces of standard user tag data are processed according to the health characteristic values.

[0009] In one embodiment, the initial user tag data collected through different data access channels is converted to obtain multiple pieces of standard user tag data under a preset data structure, including:

[0010] Convert the initial user tag data collected under different data access channels to obtain multiple user tag structure data under a preset data structure;

[0011] According to the user identity identifier of each user tag structure data, the plurality of user tag structure data are respectively mapped to corresponding candidate user tag data;

[0012] At least one non-repeating candidate user tag data is selected from the plurality of candidate user tag data as the standard user tag data.

[0013] In one embodiment, detecting the health characteristic value of the user tag data processing system based on the plurality of standard user tag data and the operating status indicator of the user tag data processing system includes:

[0014] Determine a maximum time consumption index for processing a piece of standard user tag data and an average time consumption index for processing each piece of standard user tag data, and obtain a data backlog of the plurality of pieces of standard user tag data in a preset message queue;

[0015] A health characteristic value of the user tag data processing system is determined according to the maximum time consumption index, the average time consumption index, the data backlog and the operating status index.

[0016] In one embodiment, before processing the plurality of standard user tag data according to the health characteristic values, the method further includes:

[0017] Determining a data backlog of to-be-processed user tag data among the plurality of pieces of user tag data;

[0018] The user tag data to be processed is cached according to the data backlog.

[0019] In one embodiment, caching the to-be-processed user tag data according to the data backlog includes:

[0020] When it is detected that the data backlog is less than or equal to a first preset backlog threshold, caching the to-be-processed user tag data in a local memory queue;

[0021] When it is detected that the data backlog is greater than the first preset backlog threshold and less than the second preset backlog threshold, caching the to-be-processed user tag data in a preset intermediate database;

[0022] When it is detected that the data backlog is greater than or equal to the second preset backlog threshold, the to-be-processed user tag data is cached to the current disk.

[0023] In one embodiment, the processing of the plurality of standard user tag data according to the health characteristic value includes:

[0024] determining a target data processing capacity of the user tag data processing system according to the health characteristic value;

[0025] According to the cache priority of the user tag data to be processed, selecting the user tag data to be processed corresponding to the target data processing amount from all the user tag data to be processed as the standard user tag data;

[0026] The plurality of pieces of standard user tag data are processed.

[0027] In one embodiment, the method further comprises:

[0028] Acquire an initial data processing rate for processing the plurality of pieces of standard user tag data;

[0029] adjusting the initial data processing rate according to the health characteristic value to obtain a target data processing efficiency;

[0030] Based on a preset sliding window, the actual data processing efficiency of processing the multiple standard user tag data is monitored, and the actual data processing efficiency is controlled to be within the target data processing efficiency, wherein the actual data processing efficiency is less than a preset data processing efficiency threshold.

[0031] In a second aspect, the present application further provides a user tag data processing method and apparatus, which is applied to a user tag data processing system, comprising:

[0032] A conversion module is used to convert the initial user tag data collected under different data access channels to obtain multiple standard user tag data under a preset data structure;

[0033] a detection module, configured to detect a health characteristic value of the user tag data processing system based on the plurality of standard user tag data and an operating status indicator of the user tag data processing system, wherein the health characteristic value represents a data processing capability of the user tag data processing system;

[0034] A processing module is used to process the multiple pieces of standard user tag data according to the health characteristic value.

[0035] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Initial user tag data collected under different data access channels are converted to obtain multiple pieces of standard user tag data under a preset data structure; based on the multiple pieces of standard user tag data and the operating status indicators of the user tag data processing system, health characteristic values ​​of the user tag data processing system are detected, wherein the health characteristic values ​​represent the data processing capability of the user tag data processing system; and based on the health characteristic values, the multiple pieces of standard user tag data are processed.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Initial user tag data collected under different data access channels are converted to obtain multiple pieces of standard user tag data under a preset data structure; based on the multiple pieces of standard user tag data and the operating status indicators of the user tag data processing system, health characteristic values ​​of the user tag data processing system are detected, wherein the health characteristic values ​​represent the data processing capability of the user tag data processing system; and based on the health characteristic values, the multiple pieces of standard user tag data are processed.

[0039] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0040] Initial user tag data collected under different data access channels are converted to obtain multiple pieces of standard user tag data under a preset data structure; based on the multiple pieces of standard user tag data and the operating status indicators of the user tag data processing system, health characteristic values ​​of the user tag data processing system are detected, wherein the health characteristic values ​​represent the data processing capability of the user tag data processing system; and based on the health characteristic values, the multiple pieces of standard user tag data are processed.

[0041] The above-mentioned user tag data processing method, device, computer equipment and storage medium first convert the initial user tag data collected under different data access channels into multiple standard user tag data under a preset data structure, and then detect the health characteristic value of the user tag data processing system based on the multiple standard user tag data and the operating status indicators of the user tag data processing system, wherein the health characteristic value represents the data processing ability of the user tag data processing system, and finally processes the multiple standard user tag data through the health characteristic value; since the initial user tag data collected under different data access channels can be converted into standard user tag data, the source channel of the user tag data is broadened, and at the same time, relying on By using the health characteristic value that characterizes the data processing capability of the user tag data processing system, multiple standard user tag data are processed, so that the processing capability of the user tag data processing system and the processing volume of the user tag data can be matched, thereby achieving the purpose of efficiently processing rich user tag data, rather than only being able to periodically trigger the collection and processing of user tag data in the form of fixed interface reporting. Therefore, the problem of insufficient richness of user tag data due to the variety of source channels of user tag data is overcome. At the same time, the technical defects of insufficient timeliness and accuracy of tag calculation due to the differences in data traffic and data type distribution in different processing scenarios are overcome. Therefore, the processing effect of user tag data is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 1 is a flow chart of a method for processing user tag data in one embodiment;

[0044] Figure 2 Schematic diagram of conversion of initial user tag data in a method for processing user tag data in one embodiment;

[0045] Figure 3 A schematic diagram of the structure of user tag data processing of a method for processing user tag data in one embodiment;

[0046] Figure 4 is a flowchart of a method for processing user tag data in another embodiment;

[0047] Figure 51 is a flow chart of filtering candidate user tag data in a method for processing user tag data in another embodiment;

[0048] Figure 6 1. A schematic diagram of a buffering process of user tag data to be processed in a method for processing user tag data in another embodiment;

[0049] Figure 7 is a structural block diagram of a user tag data processing device in one embodiment;

[0050] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] First of all, it should be understood that with the rapid development of the consumer finance industry, the number of users served by enterprises is also increasing. In the face of abundant user resources, it is becoming increasingly important to conduct in-depth exploration of user migration. Usually, systematic labels are constructed for multiple user resources. Among them, customer labels are often integrated through multi-dimensional data to build a full-dimensional user portrait, realize differentiated product recommendations and precise reach, and have become an important means to deeply explore user potential. For example, it can realize user churn warning, build prediction models through labels such as repayment delay frequency or APP login interval, and identify potential churned customers in advance; however, the current process of processing user label data has the following technical pain points: 1) Difficulty in collecting user label data: Currently, enterprises usually have about 30% of purchased systems and about 70% of self-developed applications. There are huge differences in the interaction methods between the same systems, and the calculation of user tags depends on rich user tag data. Therefore, how to collect or provide rich user tag data collection methods is a technical difficulty; 2) Difficulty in processing user tag data: User tag data often needs to be processed efficiently after collection. However, due to the uneven distribution of business traffic in daily life and in high-concurrency scenarios, it is difficult to rely on machine load as much as possible to regulate traffic. For example, most business scenarios currently directly adopt a fixed threshold circuit breaker mechanism. Directly rejecting requests will result in data loss. In addition, there is currently a lack of priority management for backlog data processing, and the backlog release rate does not match the current system processing capacity, which can easily cause secondary overload. In summary, there is an urgent need for a user tag data processing method to improve the processing effect of user tag data.

[0053] In one embodiment, Figure 1 As shown, a user tag data processing method is provided. This embodiment uses the method applied to a terminal as an example. The terminal includes but is not limited to a personal computer, a laptop computer, a smart phone, and a tablet computer. The terminal is deployed with a user tag data processing system. The user tag processing system includes a conversion module, a detection module, and a processing module. The conversion module is used to convert initial user tag data collected under different data access channels to obtain multiple standard user tag data under a preset data structure. The detection module is used to detect health characteristic values ​​of the user tag data processing system based on the multiple standard user tag data and the operating status indicators of the user tag data processing system. The health characteristic values ​​represent the data processing capacity of the user tag data processing system. The processing module is used to process the multiple standard user tag data based on the health characteristic values. Through information exchange between the conversion module, the detection module, and the processing module, during the process of processing the user tag data, initial user tag data from different data access channels can be collected, and the user tag data processing capacity can be matched with the processing volume of the user tag data, thereby achieving the purpose of efficiently processing a rich amount of user tag data. Therefore, the processing effect of the user tag data can be improved. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 202 to 206. Among them:

[0054] Step 202 : converting the initial user tag data collected through different data access channels to obtain a plurality of standard user tag data under a preset data structure.

[0055] It should be noted that data access channels refer to different paths and channels for transmission from the generation source to the data processing system. Initial user label data refers to user label data collected through various data access channels without unified processing and standardization. User label data represents user data used to calculate labels, which may specifically include event data, attribute data and statistical data. Among them, event data represents user behavior data, such as borrowing, repayment or credit application data, etc. Attribute data is used to represent the user's own attributes, such as gender, age and credit limit, etc. Statistical data is used to represent user-related statistical data, such as the number of borrowing days, repayment days and average loan amount, etc. It can be understood that different types of initial user label data have different reporting forms. For example, for event data, it supports two reporting forms: Kafka and interface. Among them, the interface can be specifically divided into RPC internal interface and Http interface. For user label data uploaded by internal self-built applications, it can be reported through Kafka and RPC internal interface, while for purchased systems User tag data uploaded by the system can be reported through Kafka and Http interfaces; for attribute data, it only supports interface reporting, which can be specifically divided into RPC internal interface and Http interface. Among them, user tag data uploaded by internal self-built applications can be reported through RPC internal interface, while purchased systems can report data through Http interface; for statistical data, it depends on scheduled data analysis and relies on the big data system to extract business data on a regular basis every day, and generate statistical data in batches on a regular basis. After the statistical data is generated, it can support Kafka and file upload for data reporting. In addition, considering the invasiveness of the business, when the corresponding user tag data is stored in the database, it can also support binlog monitoring to obtain attribute data and event data; that is, the initial user tag data can support five data access channels such as Kafka, RPC internal interface, Http interface, file upload and binlog monitoring to obtain the initial user tag data.

[0056] It should be noted that data conversion specifically refers to the process of converting the initial user tag data from the original format to a format that meets the preset requirements, which may specifically involve operations such as data cleaning, format unification and rule mapping; the preset data structure represents a pre-set standardized data organization form for standardizing the storage and management of user tag data; the standard user tag data represents the user tag data that meets the requirements of the preset data structure after data conversion, and has a unified format, standardized fields and clear meanings; for example, in an implementable manner, a set of preset data structures can be specifically defined, specifically, the "type" field represents the data type of the initial user tag data, which can be "attribute", "statistics" or "event", and the "code" field represents the coded value of the code. Specifically, when type is an attribute, there is only one code, that is, attribute. When type is an event, there can be multiple codes, and each time type is an independent code. For example, "loan" represents a loan event, "repayment" represents a repayment event, and when type is statistics, there is only one code, that is, " "statistics"; the "name" field represents the name, specifically the name of attribute data, statistical data, or event data; the "map" field represents the attribute map. Specifically, for event data, each event code has an attribute map unique to the current event. For example, a loan event includes attributes such as loan amount and loan product. For attribute data, the map stores the user attribute values ​​reported this time. For peer data, the map stores statistical data of this trademark, such as the average loan amount or average repayment period. "user_identify" represents the user identifier, specifically the user's unique identity, such as mobile phone number, ID card number, customer number, and user number; "identity_type" represents the identifier type, which is used to distinguish the type of user identifier, such as mobile phone number, ID card number, customer number, and user number; "time" represents the occurrence time, specifically the time when the data occurred. It should be emphasized that the calculation of many user tags depends on the data occurrence time of the user tag data, and the data reporting time cannot be used here because in some cases there may be data backlogs, resulting in a certain time difference between the actual data occurrence time and the reporting time.

[0057] As an example, step 202 includes: obtaining initial user tag data collected under different data access channels, and converting multiple initial user tag rules into standard user tag data under a preset data structure through preset conversion rules.

[0058] In one practicable manner, referring to Figure 2 , Figure 2 To represent the conversion diagram of the initial user tag data, the user tag data processing system can be connected with the Kafka data receiver, RPC data receiver, HTTP data receiver, file data receiver and binlog data receiver. Among them, the Kafka data receiver uploads event data, the RPC data receiver uploads event data and attribute data, the HTTP data receiver uploads event data and attribute data, the file data receiver uploads statistical data, and the binlog data receiver uploads event data and attribute data. The user tag data processing system is deployed with data converters and data filters. Through the processing of data converters and data filters, multiple standard user tag data under the preset data structure can be obtained.

[0059] Step 204 : detecting a health characteristic value of the user tag data processing system based on the plurality of standard user tag data and the operating status indicators of the user tag data processing system.

[0060] It should be noted that after converting multiple standard user tag data, the label calculation will rely on the standard user tag data. However, the label calculation process will affect the operation of the user tag data processing system. Therefore, after the operation of the user tag data processing system changes, the label calculation rate will be adjusted. After the calculation rate changes, the values ​​of various indicators will be promoted. The overall closed loop can be formed and in a dynamic balance; the operating status indicator represents the quantitative parameters reflecting the system operation status obtained by statistics, analysis and calculation of the standard user tag data, which can be specifically request processing time, active threads, queue backlog, CPU utilization and database connection pool usage, etc.; the health characteristic value represents the data processing capability of the user tag data processing system, which can be specifically 0.7, 0.8 or 0.9, etc.

[0061] As an example, step 204 includes: collecting operating status indicators of the user tag data processing system, inputting multiple standard user tag numbers and operating status indicators into a preset health detection model, and detecting and obtaining health characteristic values ​​of the user tag data processing system.

[0062] Step 206: Process the plurality of standard user tag data according to the health characteristic values.

[0063] It should be noted that after obtaining the health characteristic value, a corresponding health grade response strategy can be set based on the health characteristic value. For example, in one feasible method, when the health characteristic value is greater than 0.8, it indicates that the resource utilization of the user tag data processing system is low, and the processing rate of the standard user tag data can be accelerated; when the health characteristic value is greater than 0.6 and less than 0.8, it indicates that the resource utilization of the user tag data processing system is medium, and the data processing speed can be steadily accelerated; when the health characteristic value is less than 0.6 and greater than 0.4, it indicates that the resource utilization of the user tag processing system is high, and the processing rate of the standard user tag data needs to be steadily adjusted; when the health characteristic value is less than 0.4, it indicates that the resource utilization of the user tag processing system is tight, and the processing rate of the standard user tag data needs to be quickly reduced; it can be understood that after obtaining the health characteristic value, the preset data processing strategy for processing multiple standard user tag data can be queried based on the health characteristic value, and the multiple standard user tag data can be processed according to the corresponding preset data processing strategy.

[0064] As an example, step 206 includes: using the health feature value as an index, querying to obtain a preset data processing strategy, and processing multiple pieces of standard user tag data based on the preset data processing strategy.

[0065] In one practicable manner, referring to Figure 3 , Figure 3 The structural diagram of user tag data processing is shown in FIG. 4 , which may include a perception layer, a decision layer, and an execution layer. The perception layer is mainly responsible for collecting indicators, including business performance indicators and basic resource indicators. Business performance indicators mainly include request processing time, active threads, and queue backlogs. Basic resource indicators include CPU utilization and database connection pool utilization. The relevant indicators of the perception layer can be generated through standard user tag data. The execution layer is mainly responsible for adjusting the specific data processing solution, which can be dynamically adjusting the data processing rate to keep the system dynamically in a healthy state. That is, it will adjust the processing method of multiple standard user tag data depending on the health feature value.

[0066] The above-mentioned user tag data processing method first converts the initial user tag data collected under different data access channels into multiple standard user tag data under a preset data structure, and then detects the health characteristic value of the user tag data processing system through the operating status indicators generated by the multiple standard user tag data, wherein the health characteristic value represents the data processing ability of the user tag data processing system, and finally processes the multiple standard user tag data through the health characteristic value; since the initial user tag data collected under different data access channels can be converted into standard user tag data, the source channel of the user tag data is broadened, and at the same time, it depends on the characteristics of the user tag data processing system. The health characteristic value of the system's data processing capability is used to process multiple standard user tag data, so that the processing capability of the user tag data processing system and the processing volume of the user tag data can be matched, thereby achieving the purpose of efficiently processing rich user tag data, rather than only being able to periodically trigger the collection and processing of user tag data in the form of fixed interface reporting. Therefore, it overcomes the lack of richness of user tag data due to the variety of source channels of user tag data. At the same time, it overcomes the technical defects of insufficient timeliness and accuracy of tag calculation due to the differences in data traffic and data type distribution in different processing scenarios. Therefore, it improves the processing effect of user tag data.

[0067] In one embodiment, Figure 4 As shown, the initial user tag data collected under different data access channels is converted to obtain multiple standard user tag data under a preset data structure, including:

[0068] Step 302 : converting the initial user tag data collected through different data access channels to obtain a plurality of user tag structure data under a preset data structure.

[0069] It should be noted that a unified structure is used to receive the initial user tag data. Due to certain differences between systems, different data receivers are required to have the ability to convert data and process the data into the unified data structure above for reporting.

[0070] As an example, step 302 includes: converting initial user tag data collected under different data access channels through preset data conversion rules to obtain multiple user tag structure data under a preset data structure.

[0071] Step 304 : Mapping the plurality of user tag structure data into corresponding candidate user tag data respectively according to the user identity identifier of each user tag structure data.

[0072] It should be noted that, considering that the initial user tag data uploaded by different data access channels are usually collected with different user identifiers, in order to facilitate subsequent user tag calculations, data conversion and mapping are required after the different initial user tag data are reported. Specifically, different user identifiers are converted into internal unified user identity identifiers. For example, in one feasible method, system A reports the initial user tag data a through the mobile phone number, and system B reports the initial user tag data b through the customer number. The initial user tag data a and the initial user tag data b belong to the same user. The initial user tag data a and the initial user tag data b can be data mapped, so that the initial user tag data of the same user can be integrated; it can be understood that the user tag processing system is provided with a user identifier mapping table, wherein the user identifier mapping table is used to uniformly map the ID card, mobile phone number, customer number and user number into the internally used identityId, thereby ensuring that the initial user tag data uploaded with different identity identifiers can be uniformly processed as user tag data under the same user identity identifier.

[0073] As an example, step 304 includes: extracting the user identity identifier from each user tag structure data, using the user identity information as an index, searching for the corresponding preset identity identifier in the user identifier mapping table, mapping each user tag structure data to the corresponding preset identity identifier, and obtaining multiple candidate user tag data.

[0074] Step 306 : Filter out at least one non-repeating candidate user tag data from the plurality of candidate user tag data as standard user tag data.

[0075] It should be noted that after performing the data mapping and conversion operations, multiple candidate user data need to be filtered to ensure that the standard user label data are independent data without duplication. Specifically, multiple candidate user label data can be compared one by one. If two duplicate candidate user label data are compared, only one candidate user label data is retained until all candidate user label data are screened.

[0076] As an example, step 306 includes: comparing multiple candidate user tag data one by one, and using multiple non-repeating candidate user tag data as standard user tag data.

[0077] In one feasible manner, specific filtering rules can be set for different types of standard user tag data to filter multiple candidate user tag data. For example, assuming that multiple standard user tag data are multiple event data, due to the diversity of data reporting channels, the same event may be uploaded through different data access channels, so that the user tag data detection system receives event data c and event data d for the same event. Furthermore, even if the standard user tag data is uploaded through the same data access channel, data duplication may occur. In order to ensure the accuracy of subsequent calculation of user tags, reference is made to Figure 5 , Figure 5 To represent the process diagram of candidate user tag data filtering, the data occurrence time of each candidate user tag data can be extracted. Before formally processing the standard user tag data, each candidate user tag data is first written into the redis identification data. If the data is written successfully, it means that no relevant data has been entered before, then the candidate user data can be used as the standard user tag data. Otherwise, it is necessary to query redis to obtain the time when the last data occurred. If it is greater than 1 second, start the data processing process, otherwise end it directly.

[0078] In this embodiment, in the process of converting the initial user tag data to obtain standard user tag data, the initial user tag data is first converted according to a unified data conversion method to obtain multiple user tag structure data, and then by extracting the user identity identifier of each user tag structure data, the mapping of all user tag structure data is completed to obtain multiple candidate user tag data, and finally by filtering and processing the multiple candidate user tag data, multiple standard user tag data are obtained, thereby ensuring the integrity and independence of the user tag data, and thus laying the foundation for improving the processing effect of user tag data.

[0079] In one embodiment, detecting health characteristic values ​​of a user tag data processing system based on operating status indicators generated from a plurality of standard user tag data includes:

[0080] Determine the maximum time consumption index for processing a piece of standard user tag data and the average time consumption index for processing each piece of standard user tag data, and obtain the data backlog of multiple pieces of standard user tag data in the preset message queue; determine the health characteristic value of the user tag data processing system based on the maximum time consumption index, average time consumption index, data backlog and operating status index.

[0081] It should be noted that the health characteristic value can be calculated based on a pre-set system health calculation model, taking into account the operating status indicators and standard user tag data. Among them, the average time consumption indicator can be specifically the time consumed by the user tag data processing system to process any standard user tag data, and the maximum time consumption indicator can be specifically the maximum time consumption indicator for processing a standard user tag data, which can be obtained by comparing the data processing time corresponding to multiple standard user tag data. The operating status indicators can specifically include CPU utilization and database connection pool utilization, etc.

[0082] As an example, by comparing the data processing times corresponding to multiple pieces of standard user tag data, a maximum time consumption index is obtained, and the data processing times corresponding to the multiple pieces of standard user tag data are averaged to obtain an average time consumption index for processing each piece of standard user tag data, and the data backlog of the multiple pieces of standard user tag data in the preset message queue and the maximum data processing capacity of the preset message queue are obtained; by inputting the maximum time consumption index, the average time consumption index, the data backlog, the maximum data processing capacity and the operating status index into the system health calculation model, the health characteristic value of the user tag processing system is calculated, wherein the expression of the system health calculation model is as follows:

[0083]

[0084] Among them, H is the health characteristic value of the user tag data processing system, is the current number of active threads, is the maximum number of thread pools, The average time it takes to process a piece of standard user tag data. The maximum time required to process a standard user tag data. is the data backlog, The maximum data processing capacity of the preset message queue, is the CPU utilization, Database connection pool usage is the dynamic weight coefficient for different parameter values.

[0085] In this embodiment, the maximum time consumption index for processing a standard user tag data and the average time consumption index for processing each standard user tag data are first determined, and the data backlog of multiple standard user tag data in the preset message queue is obtained. Finally, the health characteristic value is calculated through the predefined system health calculation model. In the process of detecting the health characteristic value of the user tag data processing system, the matching relationship between the system operation status and the data processing volume is fully considered, thereby laying the foundation for further improving the processing effect of user tag data.

[0086] In one embodiment, before processing the plurality of standard user tag data according to the health characteristic values, the method further includes:

[0087] Determine the data backlog of the user tag data to be processed among the multiple user tag data; and cache the user tag data to be processed according to the data backlog:

[0088] According to the business data to be inspected, the user tag data processing system is provided with a data receiving module, which will put the standard user tag data to be processed into Kafka for subsequent processing of multiple standard user tag data based on health feature values; it is understandable that due to the limitations of Kafka for data storage, data backlog will occur before processing. In order to ensure the normal processing of data, the data tag data to be processed will be cached.

[0089] As an example, the data backlog of pending user tag data among multiple pieces of user tag user data is counted; if the data backlog exceeds a preset data backlog threshold, the pending user tag data is cached, where the pending user tag data represents standard user tag data awaiting processing. In this way, before processing multiple pieces of standard user tag data based on health feature values, the data backlog can be relied upon to cache the pending user tag data, thereby circumventing data squeeze issues in the message queue to a certain extent. This, in turn, lays the foundation for improving the efficiency of user tag data processing.

[0090] In one embodiment, caching the user tag data to be processed based on the amount of data backlog includes:

[0091] When it is detected that the data backlog is less than or equal to the first preset backlog threshold, the user tag data to be processed is cached in the local memory queue; when it is detected that the data backlog is greater than the first preset backlog threshold and less than the second preset backlog threshold, the user tag data to be processed is cached in the preset intermediate database; when it is detected that the data backlog is greater than or equal to the second preset backlog threshold, the user tag data to be processed is cached to the local disk.

[0092] It should be noted that the user tag data processing system can start a thread to monitor the squeeze of the user tag data to be processed in real time. When data squeeze occurs, different caching methods are adopted to cache the data based on different squeeze degrees.

[0093] As an example, when it is detected that the data backlog is less than or equal to the first preset backlog threshold, the user tag data to be processed is cached in the local memory queue; when it is detected that the data backlog is greater than the first preset backlog threshold and less than the second preset backlog threshold, the user tag data to be processed is cached in the preset intermediate database; when it is detected that the data backlog is greater than or equal to the second preset backlog threshold, the user tag data to be processed is cached to the local disk.

[0094] In one practicable manner, referring to Figure 6 , Figure 6 This is a schematic diagram of the cache process for user tag data to be processed. When message squeeze occurs, the user tag data to be processed will not be processed immediately, but will be written to the three-level buffer queue. The three-level buffer queue is the memory queue, the distributed queue and the local disk. Among them, when the data backlog is less than 1,000, it will be written directly to the local memory queue, and a backup copy of the data will also be placed in Redis and the disk to avoid data loss caused by machine restart; when the data backlog exceeds 1,000 and is less than 10,000, it will be written to Redis, and a copy of the data will be placed on the disk; after the data backlog exceeds 10,000, it will be written to the local disk; the subsequent user data processing module will directly obtain user data for processing from the three-level cache, and will not directly consume data from Kafka. The data will be cleared after data processing is completed.

[0095] In this embodiment, the size relationship between the data backlog and the first preset backlog threshold and the second preset backlog threshold is first detected, and a backlog classification processing scheme is adopted based on the size relationship, and a three-level backlog buffer pool and priority-sensitive consumption strategy are proposed to ensure zero loss and priority processing of key business requests, thereby laying the foundation for further improving the processing effect of user tag data.

[0096] In one embodiment, multiple pieces of standard user tag data are processed based on health feature values, including:

[0097] According to the health characteristic value, the target data processing volume of the user tag data processing system is determined; according to the cache priority of the user tag data to be processed, the user tag data to be processed corresponding to the target data processing volume is selected from all the user tag data to be processed as the standard user tag data; and multiple standard user tag data are processed.

[0098] It should be noted that the user tag data processing system may also be provided with a data transfer module. After the user tag data to be processed is written into the data receiving module, the user tag data to be processed will be obtained from the local queue for processing during the calculation process. In the process of processing, different user tag data to be processed will be processed according to the cache priority. For example, in one practicable manner, referring to Figure 7 , Figure 7 To illustrate the process of transferring the user tag data to be processed, first, some Redis data is written to the local queue. The specific amount written cannot exceed the maximum queue length of the local queue (1000); then some disk data will be written to Redis, which also cannot exceed the threshold of the number of data items stored in Redis to ensure that sufficient standard user tag data is obtained.

[0099] As an example, the target data processing volume of the user tag data processing system is queried using the health feature value as an index; based on the cache priority of the user tag data to be processed, the user tag data to be processed corresponding to the target data processing volume is selected from the preset intermediate database and the local disk as the standard user tag data, wherein the preset intermediate database can specifically be Redis; and multiple standard user tag data are processed.

[0100] In one embodiment, the method further comprises:

[0101] Obtain an initial data processing rate for processing multiple pieces of standard user tag data; adjust the initial data processing rate based on the health characteristic value to obtain a target data processing efficiency; monitor the actual data processing efficiency for processing multiple pieces of standard user tag data based on a preset sliding window, and control the actual data processing efficiency to be within the target data processing efficiency, wherein the actual data processing efficiency is less than a preset data processing efficiency threshold.

[0102] It should be noted that in order to effectively control the processing flow of standard user tag data, a sliding window method can be used to count the current machine data processing rate. Through the sliding window method, the number of data processed per second can be smoothly calculated, which is recorded as R. To control the efficiency of data processing, a token bucket method can be used to control the data processing rate in real time. A specified number of tokens are issued per unit time. Each token is obtained to process a piece of data. When performing flow control, it is only necessary to adjust the number of tokens issued per unit time in real time. Specifically, the target data processing efficiency is the expected data processing efficiency for processing standard user tag data. It can be adjusted based on the health characteristic value. For example, when the health characteristic value is greater than or equal to 0.8, it is necessary to significantly increase the current data processing speed. At this time, the current data processing rate is directly doubled. Assuming that the initial data processing rate is 100 pieces / second, the current speed is directly increased to 200 pieces / second, that is, the token issuance rate is controlled to 200 pieces / second. Here, an upper limit value needs to be set for the consumption rate. The upper limit value is related to the configuration of the machine. Based on the configuration of the production machine, the system stress test can be performed in the test environment to obtain the maximum processing speed. Here, the fastest processing speed is recorded as When the health characteristic value is less than or equal to 0.8 and greater than 0.6, it is necessary to steadily increase the data processing speed, first increase the processing of 10 data per second. For example, if the current data processing rate is 100 / second, the adjusted rate will be 110 / second. At the same time, it is necessary to ensure that the adjusted processing speed is less than When the health characteristic value is less than or equal to 0.6 and greater than 0.4, the data processing speed needs to be gradually reduced by 10 data items per second. For example, if the current data processing rate is 200 data items per second, the adjusted rate will be 190 data items per second. When the health characteristic value is less than 0.4, the data processing speed needs to be quickly reduced by halving the current processing rate. For example, if the current data processing rate is 200 data items per second, the adjusted rate will be 100 data items per second. The detailed data adjustment formula is shown below.

[0103]

[0104] Among them, R is the number of standard user label data processed per second, and H is the health feature value.

[0105] As an example, an initial data processing rate for processing multiple pieces of standard user tag data is obtained; the initial data processing rate is adjusted to a target data processing efficiency based on a health characteristic value; the actual data processing efficiency for processing multiple pieces of standard user tag data is monitored based on a preset sliding window, and the actual data processing efficiency is controlled to be within the target data processing efficiency, wherein the actual data processing efficiency is less than a preset data processing efficiency threshold, which can specifically be the fastest processing rate. In this way, by establishing a set of adaptive flow control methods, the data processing rate is gradually increased and the data processing rate is steadily reduced. At the same time, an elastic scaling strategy based on a response time-processing rate feedback loop is established to achieve dynamic adaptation of the processing rate and response time, thereby further improving the processing effect of user tag data.

[0106] Understandably, over 90% of tag calculations currently rely on user data from within the past month, over 60% on data from the past three days, and over 50% on data from within the past 24 hours. Therefore, data from within the past 24 hours is stored in memory to ensure that data for the same user is only routed to a single machine for execution. To ensure that the user's data remains in memory, Kafka uses identityId as the partitioning rule to ensure that all data is routed to the same machine. Data from within the past 24 hours is stored in both Redis and MySQL. This allows for data access from both Redis and MySQL to prevent data loss if the local machine restarts. Data from within the past three days is stored in Redis, with a separate copy in MySQL. Data from within one month is stored in MySQL. A thread is used to regularly clean up expired data from local memory, Redis, and MySQL, and to write data older than one month to Hive. Since query performance decreases with each level of the four-level cache, adopting this four-level storage solution maximizes machine processing speed.

[0107] By adopting the above processing method, since the initial user tag data collected by different data access channels can be converted into standard user tag data, the source channels of user tag data are broadened. At the same time, relying on the health characteristic value that characterizes the data processing capability of the user tag data processing system, multiple standard user tag data are processed, so that the processing capability of the user tag data processing system and the processing volume of the user tag data can be matched, and the purpose of efficiently processing rich user tag data can be achieved, rather than only being able to trigger the collection and processing of user tag data in a fixed interface reporting manner. Therefore, the lack of richness of user tag data due to the variety of source channels of user tag data is overcome. At the same time, the technical defects of insufficient timeliness and accuracy of tag calculation due to the differences in data traffic and data type distribution in different processing scenarios are overcome. Therefore, the processing effect of user tag data is improved.

[0108] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0109] Based on the same inventive concept, the present application also provides a user tag data processing device for implementing the user tag data processing method mentioned above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more user tag data processing device embodiments provided below can be found in the above-mentioned limitations of the user tag data processing method and will not be repeated here.

[0110] In an exemplary embodiment, Figure 7 As shown, a user tag data processing device is provided, which is applied to a user tag data processing system, including: a conversion module 401, a detection module 402 and a processing module 403, wherein:

[0111] The conversion module 401 is used to convert the initial user tag data collected under different data access channels to obtain multiple standard user tag data under a preset data structure;

[0112] A detection module 402 is configured to detect a health characteristic value of the user tag data processing system based on a plurality of standard user tag data and an operating status indicator of the user tag data processing system, wherein the health characteristic value represents the data processing capability of the user tag data processing system;

[0113] The processing module 403 is used to process multiple pieces of standard user tag data according to the health characteristic values.

[0114] In one embodiment, the conversion module 401 is further configured to:

[0115] The initial user tag data collected under different data access channels are converted to obtain multiple user tag structure data under a preset data structure; according to the user identity identifier of each user tag structure data, the multiple user tag structure data are respectively mapped to corresponding candidate user tag data; and at least one non-repeating candidate user tag data is screened out from the multiple candidate user tag data as standard user tag data.

[0116] In one embodiment, the detection module 402 is further configured to:

[0117] Determine the maximum time consumption index for processing a piece of standard user tag data and the average time consumption index for processing each piece of standard user tag data, and obtain the data backlog of multiple pieces of standard user tag data in the preset message queue; determine the health characteristic value of the user tag data processing system based on the maximum time consumption index, average time consumption index, data backlog and operating status index.

[0118] In one embodiment, the user tag data processing device is further configured to:

[0119] Determine a data backlog of user tag data to be processed among multiple pieces of user tag data; and cache the user tag data to be processed according to the data backlog.

[0120] In one embodiment, the user tag data processing device is further configured to:

[0121] When it is detected that the data backlog is less than or equal to the first preset backlog threshold, the user tag data to be processed is cached in the local memory queue; when it is detected that the data backlog is greater than the first preset backlog threshold and less than the second preset backlog threshold, the user tag data to be processed is cached in the preset intermediate database; when it is detected that the data backlog is greater than or equal to the second preset backlog threshold, the user tag data to be processed is cached to the local disk.

[0122] In one embodiment, the user tag data processing device is further configured to:

[0123] According to the health characteristic value, the target data processing volume of the user tag data processing system is determined; according to the cache priority of the user tag data to be processed, the user tag data to be processed corresponding to the target data processing volume is selected from all the user tag data to be processed as the standard user tag data; and multiple standard user tag data are processed.

[0124] In one embodiment, the user tag data processing device is further configured to:

[0125] Obtain an initial data processing rate for processing multiple pieces of standard user tag data; adjust the initial data processing rate based on the health characteristic value to obtain a target data processing efficiency; monitor the actual data processing efficiency for processing multiple pieces of standard user tag data based on a preset sliding window, and control the actual data processing efficiency to be within the target data processing efficiency, wherein the actual data processing efficiency is less than a preset data processing efficiency threshold.

[0126] Each module in the user tag data processing device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0127] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, it implements a user tag data processing method. Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0128] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0130] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0131] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0132] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for processing user tag data, characterized in that: Applied to a user tag data processing system, the method includes: Convert the initial user tag data collected under different data access channels to obtain multiple standard user tag data under a preset data structure; detecting a health characteristic value of the user tag data processing system based on the plurality of standard user tag data and an operating status indicator of the user tag data processing system, wherein the health characteristic value represents a data processing capability of the user tag data processing system; The plurality of pieces of standard user tag data are processed according to the health characteristic values.

2. The method according to claim 1, characterized in that The initial user tag data collected under different data access channels is converted to obtain multiple pieces of standard user tag data under a preset data structure, including: Convert the initial user tag data collected under different data access channels to obtain multiple user tag structure data under a preset data structure; According to the user identity identifier of each user tag structure data, the plurality of user tag structure data are respectively mapped to corresponding candidate user tag data; At least one non-repeating candidate user tag data is selected from the plurality of candidate user tag data as the standard user tag data.

3. The method according to claim 1, characterized in that The detecting the health characteristic value of the user tag data processing system according to the plurality of standard user tag data and the operating status indicator of the user tag data processing system includes: Determine a maximum time consumption index for processing a piece of standard user tag data and an average time consumption index for processing each piece of standard user tag data, and obtain a data backlog of the plurality of pieces of standard user tag data in a preset message queue; A health characteristic value of the user tag data processing system is determined according to the maximum time consumption index, the average time consumption index, the data backlog and the operating status index.

4. The method according to claim 1, wherein Before processing the plurality of pieces of standard user tag data according to the health characteristic values, the method further includes: Determining a data backlog of to-be-processed user tag data among the plurality of pieces of user tag data; The user tag data to be processed is cached according to the data backlog.

5. The method according to claim 4, characterized in that The caching of the to-be-processed user tag data according to the data backlog includes: When it is detected that the data backlog is less than or equal to a first preset backlog threshold, caching the to-be-processed user tag data in a local memory queue; When it is detected that the data backlog is greater than the first preset backlog threshold and less than the second preset backlog threshold, caching the to-be-processed user tag data in a preset intermediate database; When it is detected that the data backlog is greater than or equal to the second preset backlog threshold, the to-be-processed user tag data is cached to the current disk.

6. The method according to claim 5, characterized in that The processing of the plurality of pieces of standard user tag data according to the health characteristic value includes: determining a target data processing capacity of the user tag data processing system according to the health characteristic value; According to the cache priority of the user tag data to be processed, selecting the user tag data to be processed corresponding to the target data processing amount from all the user tag data to be processed as the standard user tag data; The plurality of pieces of standard user tag data are processed.

7. The method according to claim 6, characterized in that The method further comprises: Acquire an initial data processing rate for processing the plurality of pieces of standard user tag data; adjusting the initial data processing rate according to the health characteristic value to obtain a target data processing efficiency; Based on a preset sliding window, the actual data processing efficiency of processing the multiple standard user tag data is monitored, and the actual data processing efficiency is controlled to be within the target data processing efficiency, wherein the actual data processing efficiency is less than a preset data processing efficiency threshold.

8. A user tag data processing device, characterized in that: Applied to a user tag data processing system, the device comprises: A conversion module is used to convert the initial user tag data collected under different data access channels to obtain multiple standard user tag data under a preset data structure; a detection module, configured to detect a health characteristic value of the user tag data processing system based on the plurality of standard user tag data and an operating status indicator of the user tag data processing system, wherein the health characteristic value represents a data processing capability of the user tag data processing system; A processing module is used to process the multiple pieces of standard user tag data according to the health characteristic value.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images