Data quality detection method and data quality detection device
By acquiring and converting business data in real time, conducting quality inspection and scoring, the problem of delayed data quality assessment is solved, and timely inspection of data quality and accuracy of risk control assessment are achieved.
Patent Information
- Application Number
- CN202510567814.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-30
- Publication Date
- 2025-09-23
Smart Images
Figure CN120688907A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data quality detection method and a data quality detection device. Background Art
[0002] Currently, in some business scenarios, business data reported from various business lines is tested for business indicators according to pre-defined detection rules to determine whether any abnormal business data exists. For example, in risk control scenarios, user behavior data generated by users on business lines needs to be tested for abnormal risk control indicators.
[0003] Current risk control assessments rely on "black box" data, making it impossible to assess the reliability of the data itself. Furthermore, declines in data quality are often only discovered after they impact business operations, resulting in a lag. Furthermore, data collection standards are inconsistent across different business lines, resulting in inconsistent risk control decision-making, which in turn impacts risk control assessment results.
[0004] Therefore, there is still a problem in the relevant technology that cannot timely evaluate the quality of the data itself, which needs to be solved. Summary of the Invention
[0005] In this embodiment, a data quality detection method and a data quality detection device are provided to solve the problem in related technologies that the quality of the data itself cannot be evaluated in a timely manner.
[0006] First, in this embodiment, a data quality detection method is provided, including:
[0007] Real-time acquisition of target business data associated with each business line; the target business data is generated based on user behavior events of different users in each business line;
[0008] Converting the target business data into standard behavior data according to attribute sets preset for different user behavior events;
[0009] According to different quality detection dimensions, quality detection is performed on the standard behavior data respectively, and dimension scores corresponding to each quality detection dimension are obtained;
[0010] Perform weighted processing on the scores of each dimension to obtain a real-time data quality score for each business line;
[0011] Obtaining statistical results of quality scores corresponding to the previous degradation detection cycle for each business line, comparing the current data quality score of each business line with the statistical results of quality scores corresponding to the previous degradation detection cycle, and determining a target degradation degree for each business line; the target degradation degree is one of a plurality of preset data quality degradation degrees;
[0012] According to the target degradation degree of each business line, a quality warning mechanism corresponding to the target degradation degree is determined and executed; wherein different data quality degradation degrees correspond to different quality warning mechanisms.
[0013] In some embodiments, the attribute set includes a standard attribute set and an extended attribute set; wherein, the standard attribute sets corresponding to different user behavior events are the same, and the extended attribute sets corresponding to different user behavior events are different.
[0014] In some embodiments, the quality detection dimensions include: completeness detection, accuracy detection, rationality detection and logical consistency detection.
[0015] In some embodiments, converting the target business data into standard behavior data according to attribute sets preset for different user behavior events includes:
[0016] Converting the target business data into initial behavior data according to attribute sets preset for different user behavior events;
[0017] Perform field mapping detection on the initial behavior data to obtain standard behavior data.
[0018] In some embodiments, comparing the current data quality score of each business line with the quality score statistics corresponding to the previous degradation detection cycle to determine the target degradation level of each business line includes:
[0019] Calculating the difference between the current data quality score of each business line and the quality score statistical result corresponding to the previous degradation detection cycle;
[0020] The target degradation degree of each business line is determined based on the difference and a preset difference threshold.
[0021] In some embodiments, based on the target degradation degree of each business line, a quality warning mechanism corresponding to the target degradation degree is determined and executed; wherein different quality warning mechanisms correspond to different data quality degradation degrees, including:
[0022] Determining the cumulative number of degradations achieved by each business line at the target degradation level;
[0023] According to the cumulative number of degradations, it is determined whether each business line meets the warning condition corresponding to the target degradation degree; if so, a quality warning mechanism corresponding to the target degradation degree is determined and executed.
[0024] In some embodiments, the method further comprises:
[0025] According to the preset monthly scoring dimensions, the monthly data quality score of the standard behavior data of each business line is scored on a monthly basis to obtain the monthly data quality score results of each business line; the monthly scoring dimensions include: the missing rate of core weight fields, the logical consistency ratio, the expectation-unsatisfied data ratio, and the data processing delay time rate;
[0026] Adjust the data quality detection strategy for each business line in the next month based on the data quality scoring results of the current month.
[0027] In some embodiments, the method further comprises:
[0028] Integrate the standard behavior data of each business line to obtain user global behavior data constructed according to the user's unique identifier;
[0029] According to the preset statistical granularity, the user's global behavior data is statistically analyzed for active periods to obtain a user active period indicator;
[0030] Based on the user's global behavior data, determine the user's activity intensity in each business line and the coupling jump pattern between the business lines.
[0031] In some embodiments, the method further comprises:
[0032] Based on the real-time data quality score of each business line, a risk control strategy corresponding to the data quality score is executed for each business line.
[0033] In a second aspect, this embodiment provides a data quality detection device, comprising: an acquisition module, a conversion module, a quality detection module, a degradation detection module, and an early warning module; wherein:
[0034] The acquisition module is used to acquire target business data associated with each business line in real time; the target business data is generated based on user behavior events of different users in each business line;
[0035] The conversion module is configured to convert the target business data into standard behavior data according to attribute sets preset for different user behavior events;
[0036] The quality detection module is used to perform quality detection on the standard behavior data according to different quality detection dimensions, and obtain dimension scores corresponding to each quality detection dimension; and perform weighted processing on each dimension score to obtain a real-time data quality score for each business line;
[0037] The degradation detection module is configured to obtain statistical results of quality scores corresponding to the previous degradation detection cycle for each business line, compare the current data quality score of each business line with the statistical results of quality scores corresponding to the previous degradation detection cycle, and determine a target degradation degree for each business line; the target degradation degree is one of a plurality of preset data quality degradation degrees;
[0038] The early warning module is used to determine and execute a quality early warning mechanism corresponding to the target degradation degree of each business line according to the target degradation degree of the business line; wherein different data quality degradation degrees correspond to different quality early warning mechanisms.
[0039] Compared with related technologies, this embodiment provides a data quality detection method and a data quality detection device. The data quality detection method acquires target business data associated with each business line in real time; the target business data is generated based on user behavior events of different users in each business line; the target business data is converted into standard behavior data according to a pre-set attribute set for different user behavior events; the standard behavior data is quality-checked according to different quality detection dimensions, and dimension scores corresponding to each quality detection dimension are obtained; the dimension scores are weighted to obtain real-time data quality scores for each business line; the quality score statistics corresponding to the previous degradation detection cycle of each business line are obtained, the current data quality score of each business line is compared with the quality score statistics corresponding to the previous degradation detection cycle, and the target degradation degree of each business line is determined; the target degradation degree belongs to one of multiple preset data quality degradation degrees; according to the target degradation degree of each business line, a quality warning mechanism corresponding to the target degradation degree is determined and executed; wherein different data quality degradation degrees correspond to different quality warning mechanisms. It can achieve timely detection of data quality before risk control assessment, accurately evaluate the data quality of business lines, and promptly detect data quality degradation, thereby achieving timely assessment of the quality of the data itself.
[0040] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0042] Figure 1 This is a hardware structure block diagram of a terminal of the data quality detection method according to an embodiment of the present application;
[0043] Figure 2is a flow chart of the data quality detection method according to an embodiment of the present application;
[0044] Figure 3 This is a risk control timing diagram based on data quality assessment in an embodiment of the present application;
[0045] Figure 4 is a timing diagram of data quality detection in some embodiments of the present application;
[0046] Figure 5 It is a structural block diagram of the data quality detection device according to an embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.
[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "an", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Generally, the character " / " indicates that the related objects are in an "or" relationship. The terms "first," "second," "third," etc. used in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0049] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG is a block diagram of the hardware structure of the terminal of the data quality detection method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1Only one is shown) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.
[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data quality detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0051] The transmission device 106 is used to receive or send data via a network. The network may include a wireless network provided by the terminal's telecommunications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0052] In this embodiment, a data quality detection method is provided. Figure 2 is a flow chart of the data quality detection method of this embodiment, such as Figure 2 As shown, the process includes the following steps:
[0053] Step S210 , obtaining target business data associated with each business line in real time; the target business data is generated based on user behavior events of different users in each business line.
[0054] Each business line can be a business segment involved in a specific business scenario. For example, in a cross-border transaction risk management scenario, this could include a transaction business line, a supply chain business line, and a user security business line. Each business line can generate target business data based on user behavior. User behavior events can include logins, modifications to sensitive fields, and deposits and withdrawals. Each business line can report the generated target business data to the server in real time, which then performs data quality checks.
[0055] Step S220 : converting the target business data into standard behavior data according to the attribute sets preset for different user behavior events.
[0056] Among them, different attribute sets can be defined in advance according to different user behavior events to ensure that the data of different business lines can be comparable and establish a unified risk control standard. Among them, the attribute set can include both standard attribute sets that are common to different user behavior events and extended attribute sets for different user behavior events. Therefore, the attribute set can provide a set of conversion standards for data collected by different business lines. After the attribute set is established, a mapping relationship can be established between each field of the target business data reported by each business line and each field in the attribute set. Afterwards, after obtaining the target business data reported by the business line, each target business data can be stored in a preset database (for example, in a risk control scenario, it can be the original database of the risk control center), and the business source is marked. In addition, the target business data is converted according to the fields in the attribute set according to the above mapping relationship to form unified standard behavior data, and the original data source is marked, and the standard behavior data is also stored in the database.
[0057] Step S230 , performing quality inspection on the standard behavior data according to different quality inspection dimensions, and obtaining dimension scores corresponding to each quality inspection dimension.
[0058] Based on the data quality inspection center of the server, the standard behavior data can be inspected in multiple dimensions to obtain dimension scores corresponding to each quality inspection dimension. Taking the standard behavior data corresponding to the user login behavior event as an example, the data can be inspected from the integrity dimension to detect whether the data is complete; it can be inspected from the accuracy dimension to determine whether the field value meets expectations; it can also be inspected from the rationality dimension to determine whether the field value reaches the preset numerical range and whether the relationship between the numerical values of different fields meets the preset conditions; and the field logic between different fields can be inspected from the perspective of logical consistency. For example, if the user triggers the login verification and successfully logs in, there should also be a verification method and a corresponding verification record. According to each dimension, the quality inspection results of the standard behavior data are scored to obtain dimension scores corresponding to each quality inspection dimension.
[0059] Step S240: weight the scores of each dimension to obtain a real-time data quality score for each business line.
[0060] Based on the pre-assigned weights for different quality detection dimensions, the scores for each dimension can be weighted and summed to obtain a complete score for each target business data item. The complete scores for all target business data for a business line can be aggregated to obtain the current data quality score for each business line. This data quality score can be retained as one of the bases for subsequent business decisions (such as risk control decisions) for that business scenario.
[0061] Step S250, obtain the quality score statistics corresponding to each business line in the previous degradation detection cycle, compare the current data quality score of each business line with the quality score statistics corresponding to the previous degradation detection cycle, and determine the target degradation degree of each business line; the target degradation degree belongs to one of the preset multiple data quality degradation degrees.
[0062] For example, a week can be used as a degradation detection cycle. The average data quality score of different user behavior events in the business line over the past week can be calculated regularly every week as the latest quality score statistical result, and the quality score statistical result can be updated. The quality score statistical result is used as a baseline score. After obtaining the current real-time data quality score of each business line, the difference a between the current real-time data quality score and the quality score statistical result can be calculated:
[0063] a=real-time data quality score - quality score statistics of last week;
[0064] Based on the difference a, we determine whether the data quality of each business line has degraded, and the specific degree of degradation. Thus, in this step, by comparing the real-time data quality score with the statistical values of the historical quality score, we can promptly detect the current data quality degradation.
[0065] Step S260 , based on the target degradation degree of each business line, determine and execute a quality warning mechanism corresponding to the target degradation degree; wherein different data quality degradation degrees correspond to different quality warning mechanisms.
[0066] After determining the target degradation level of each business line, a quality warning mechanism corresponding to the target degradation level can be executed. In some embodiments, the target degradation level can include slight degradation, moderate degradation, and severe degradation. The corresponding quality warning mechanism can include a log-level warning mechanism, a notification-level warning mechanism, and an alarm-level warning mechanism. For example, for slight degradation, after the preset trigger conditions are met, the record of slight degradation can be entered into the monitoring log based on the log-level warning and displayed on the monitoring platform without immediate processing; for moderate degradation, after the trigger conditions are met, the log can be recorded based on the notification-level warning, and an email can be sent to notify the relevant responsible personnel; for severe degradation, after the trigger conditions are met, the log can be recorded based on the alarm-level warning, and an instant messaging message can be sent to notify the responsible personnel so that the responsible personnel can immediately pay attention and respond.
[0067] This embodiment obtains target business data associated with each business line in real time through the above steps S210 to S260; the target business data is generated based on user behavior events of different users in each business line; the target business data is converted into standard behavior data according to the attribute sets pre-set for different user behavior events; quality inspection is performed on the standard behavior data according to different quality inspection dimensions, and dimension scores corresponding to each quality inspection dimension are obtained; weighted processing is performed on the scores of each dimension to obtain a real-time data quality score for each business line; the quality score statistics corresponding to the previous degradation detection cycle of each business line are obtained, and the current data quality score of each business line is compared with the quality score statistics corresponding to the previous degradation detection cycle to determine the target degradation degree of each business line; the target degradation degree belongs to one of multiple preset data quality degradation degrees; according to the target degradation degree of each business line, a quality warning mechanism corresponding to the target degradation degree is determined and executed; wherein different data quality degradation degrees correspond to different quality warning mechanisms. It can achieve timely detection of data quality before risk control assessment, accurately evaluate the data quality of business lines, and promptly detect data quality degradation, thereby achieving timely assessment of the quality of the data itself.
[0068] In one embodiment, the attribute set includes a standard attribute set and an extended attribute set; wherein, the standard attribute sets corresponding to different user behavior events are the same, and the extended attribute sets corresponding to different user behavior events are different.
[0069] Among them, the attribute set includes a standard attribute set set for the common attributes of different user behavior events, and an extended attribute set set for the characteristics of different user behavior events. For example, the standard attribute set may include fields such as operation time (op_time), operation type (op_type), operation business unit (op_bu), operation channel (op_channel), success (op_result), operator identification (user_id), and quality calculation score (score). The extended attribute set can be a field set based on the unique information of different user behavior events. Taking the user login behavior event as an example, the extended attribute set may include fields such as login type (login_type), whether a password is entered (need_password), and whether it has been verified (risk_verify). Each field in the attribute set can also be set with a corresponding weight ratio for quality scoring.
[0070] In this embodiment, by dividing the attribute set into a standard attribute set and an extended attribute set, appropriate attribute sets can be constructed based on the commonalities and individualities of different user behavior events, thereby providing a basis and standard for subsequent more comprehensive and objective data quality testing.
[0071] Additionally, in one embodiment, the quality detection dimensions include: completeness detection, accuracy detection, rationality detection, and logical consistency detection.
[0072] Among them, the quality detection dimension can be set according to the needs of the actual application scenario. In this embodiment, for the standard behavior data corresponding to the user login behavior event in the risk control scenario, the quality detection dimension is set to integrity detection, accuracy detection, rationality detection and logical consistency detection. Specifically, integrity detection is used to detect whether the data in the attribute set is complete. For example, in the standard behavior data related to login, the login device field and the login IP (Internet Protocol, referred to as Internet Protocol) address field are used as core weight fields. If the total proportion of the two fields is greater than 0.5, and both fields are not empty, the integrity test is confirmed to pass. If there are more other fields, a quality score representing excellent quality can be obtained in the dimension of integrity detection.
[0073] Accuracy testing can be used to check whether the values of fields in an attribute set meet expectations. For example, in the behavioral standard data corresponding to a user login event, the public IP field of the IPv4 protocol is required to be transparently transmitted. If the business line reports the value of the LAN IP or the IPv6 protocol address, and the device ID does not meet the generation rules and is suspected of being forged, based on the preset weights of the two fields, if the quality score corresponding to the accuracy test dimension is at most 0.3, and therefore less than 0.5, it indicates that the data accuracy is poor.
[0074] Reasonability testing checks whether the corresponding values of fields in an attribute set are reasonable within the corresponding business scenario. For example, the value of the login time field should not correspond to a future time, but should be within a reasonable time range; the difference between the time values corresponding to the collection time field and the login time field should be around one minute.
[0075] Logical consistency testing is used to check whether the field logic in the attribute set is consistent. For example, if a user triggers login verification and successfully logs in, the corresponding standard behavior data should include the user's verification method and the corresponding verification record.
[0076] This embodiment scores data quality based on multiple quality detection dimensions such as completeness detection, accuracy detection, rationality detection, and logical consistency detection, thereby improving the accuracy and comprehensiveness of data quality detection.
[0077] It is understandable that corresponding scoring criteria can be set for different data quality dimensions, and finally the weights of each data quality dimension can be combined to obtain a final complete score for a piece of standard behavior data.
[0078] Additionally, in one embodiment, the target business data is converted into standard behavior data according to the attribute sets preset for different user behavior events, which may specifically include:
[0079] Based on the pre-set attribute sets for different user behavior events, the target business data is converted into initial behavior data. Field mapping is then checked on the initial behavior data to obtain standard behavior data. Based on the mapping rules, the standard behavior data can be compared with the original target business data to determine whether there are any field mapping errors or whether any fields in the target business data are not mapped to standard behavior data.
[0080] Additionally, in one embodiment, the current data quality score of each business line is compared with the quality score statistics corresponding to the previous degradation detection cycle to determine the target degradation level of each business line, which may specifically include:
[0081] Calculate the difference between the current data quality score of each business line and the statistical result of the quality score corresponding to the previous degradation detection cycle; determine the target degradation degree of each business line based on the difference and the preset difference threshold.
[0082] Specifically, referring to the above embodiment, the difference a between the real-time data quality score and the statistical quality score corresponding to the previous degradation detection cycle can be calculated. The corresponding target degradation level is determined based on the numerical range within which the difference a falls, constructed from the difference thresholds. For example, the numerical range corresponding to mild degradation is [-10, 0); the numerical range corresponding to moderate degradation is [-20, -5); and the numerical range corresponding to severe degradation is below -20. Thus, based on the numerical range within which the difference a falls, the corresponding degradation level is determined from mild degradation, moderate degradation, and severe degradation as the target degradation level.
[0083] This embodiment can classify different degrees of data quality degradation and match them with numerical ranges, thereby achieving objective quantification of the degree of data quality degradation.
[0084] Additionally, in one embodiment, based on the target degradation level of each business line, a quality warning mechanism corresponding to the target degradation level is determined and executed. Different quality warning mechanisms may correspond to different data quality degradation levels, and may include:
[0085] Determine the cumulative number of degradations achieved by each business line at the target degradation level; based on the cumulative number of degradations, determine whether each business line meets the warning conditions corresponding to the target degradation level; if so, determine and implement the quality warning mechanism corresponding to the target degradation level.
[0086] Among them, after determining the target degradation degree of each business line, the cumulative number of times each business line reaches the corresponding target degradation degree within a specified time window can be used as the cumulative degradation number. For example, the cumulative number of times a business line reaches a slight degradation degree within 5 minutes is counted as the cumulative number of degradations of the business line under the slight degradation degree. When the cumulative number of degradations reaches a certain number threshold, it can be determined that the corresponding business line meets the warning conditions corresponding to the target degradation degree. For example, if slight degradation is reached 5 times within 5 minutes, a log-level warning corresponding to slight degradation is triggered. It can be understood that the number thresholds corresponding to different data quality degradation degrees may be different, that is, the above-mentioned warning conditions may be different.
[0087] This embodiment is based on the calculation of the cumulative number of degradations and combines it with warning conditions to determine and execute the corresponding quality warning mechanism. On the one hand, it can avoid the interference caused by frequent and unnecessary warnings to processing personnel. On the other hand, it can also issue warnings in time for situations where quality warnings are really needed.
[0088] In addition, in one embodiment, the data quality detection method may further include:
[0089] Based on the preset monthly scoring dimensions, the monthly data quality score of the standard behavior data of each business line is evaluated on a monthly basis to obtain the monthly data quality score results of each business line. The monthly scoring dimensions include: the missing rate of core weight fields, the logical consistency ratio, the ratio of data that does not meet expectations, and the data processing delay time rate. Based on the monthly data quality score results, the data quality detection strategy of each business line for the next month is adjusted.
[0090] In this embodiment, real-time data quality testing of all data reported by each business line consumes considerable computing resources. Furthermore, indiscriminate, comprehensive quality testing may also impact real-time risk control performance. Furthermore, business lines with different quality levels require data quality testing strategies of varying intensity. To address this, in this embodiment, a monthly scoring cycle is adopted, with the monthly data quality score for each business line's standard behavioral data calculated on the last day of each month. Based on the monthly data quality score, the data quality testing strategy for each business line for the following month is determined.
[0091] Data integrity can be assessed based on the missing rate of pre-determined core weighted fields in the standard behavioral data corresponding to different user behavior events. The logical consistency of data over the monthly scoring cycle can be determined by calculating the ratio of data that violates logical consistency. The accuracy of data over the monthly scoring cycle can be determined by calculating the ratio of data that does not match expected values. Furthermore, the actual data processing delay rate can be calculated.
[0092] The scores for each monthly scoring dimension can range from 0 to 100. Based on the preset conditions, the above ratios are mapped to the scores under each monthly scoring dimension. Afterwards, the scores corresponding to different monthly scoring dimensions can be weighted and summed. For example, the weight corresponding to each monthly scoring dimension can be configured as 0.25 (it can also be adjusted to other weights based on the actual application scenario requirements), and finally the data quality score result for that month is obtained.
[0093] Furthermore, based on the scoring range that the monthly data quality score results fall into, the monthly data quality of the business line is divided into the following levels:
[0094] If the monthly data quality score falls between 80 and 100, the corresponding monthly data quality rating is considered excellent. This indicates that the data quality for this business line is stable and reliable, with few historical issues. A lightweight data quality testing strategy (only validating required fields) can be adopted for the next month.
[0095] If the monthly data quality score falls between 70 and 79, the corresponding monthly data quality level can be considered good. This indicates that the overall data quality for the business line is good, but occasional issues still exist. A moderate-intensity test (compulsory field testing combined with key business rule verification) can be used as the data quality testing strategy for the next month.
[0096] If the monthly data quality score falls between 60 and 69, the corresponding monthly data quality grade is considered passing. This indicates that the data quality for the business line for the month is generally acceptable, but some issues still exist. Standard strength testing (mandatory field testing combined with value expectation management verification) can be used as the data quality testing strategy for the next month. Notifications will be sent to the corresponding business line to schedule data quality control.
[0097] If the monthly data quality score falls between 0 and 59, the corresponding data quality level for that month can be considered low. This indicates that the data quality for that business line was poor and had many issues. A comprehensive strength test (full-field, full-rule verification) can be implemented as the data quality testing strategy for the next month. Notifications will be sent to the corresponding business line to prioritize data quality management.
[0098] This embodiment can detect the data quality of the current month based on a monthly cycle, and then adaptively adjust the data quality detection strategy for the next month based on the data quality score results of the current month, thereby reducing the amount of data for data quality detection and improving risk control performance; in addition, it can also match detection strategies of different intensities for business lines with different quality levels.
[0099] Additionally, in one embodiment, the data quality detection method may further include:
[0100] Integrate the standard behavior data of each business line to obtain the user's global behavior data constructed according to the user's unique identifier; according to the preset statistical granularity, perform active period statistics on the user's global behavior data to obtain the user's active period indicator; based on the user's global behavior data, determine the user's activity intensity in each business line and the coupling jump pattern between business lines.
[0101] Time series analysis can be used to monitor user behavior in real time and determine whether there are any abnormalities. Based on global user behavior data, user activity periods can be counted, with hourly granularity used to determine user activity period indicators. This can also mark user activity patterns, such as weekend or weekday activity, to create a user time activity profile.
[0102] Additionally, the user's activity intensity in each business line can be counted and marked to determine the user's business line activity profile in the business line, and to determine the user's coupling jump pattern between different business lines. The coupling jump pattern is the interaction method of calling, jumping, or relying on each other between different business lines, which can include deep link jump mode, single sign-on integration mode, unified navigation framework mode, etc.
[0103] In this embodiment, by collecting and integrating standard behavioral data from each business line, it is possible to build a global user behavior portrait using the user's unique identifier as a link, and detect user behavior anomalies in real time through time series analysis technology, providing accurate data support for subsequent risk control decisions.
[0104] Additionally, in one embodiment, the data quality detection method may further include:
[0105] Based on the real-time data quality scores of each business line, risk control strategies corresponding to the data quality scores are implemented for each business line.
[0106] For example, for low-quality business lines, data from multiple business lines can be used to supplement risk control decision-making. Decision-making data derived from risk control testing based on standard behavioral data can be subject to comprehensive verification, extending decision-making time. This comprehensive verification process can include comprehensive parameter verification, more in-depth risk assessment, and review of the entire risk control rule set, supplementation of data from multiple business lines, and rigorous risk control decision-making.
[0107] For medium-quality business lines, specific risk control steps can be strengthened and some risk control verification measures can be added. For high-quality business lines, risk control processes can be appropriately simplified, verification steps can be reduced, and decision-making speed can be improved. For example, this can include rapid verification of risk control parameters, lightweight risk assessments, testing based on core risk control rules, and rapid decision-making.
[0108] In one embodiment, the real-time risk control process may include the following steps:
[0109] Accept risk control assessment requests, obtain the unique identifier of user behavior events, and request to obtain target business data of the corresponding business line;
[0110] Based on the unique event and business line identifiers, the target business data corresponding to different user behavior events for each business line is converted into standard behavior data according to the corresponding attribute set. Data quality is then tested to obtain real-time data quality scores for each business line. User time series information is obtained to determine user behavior profiles.
[0111] Based on the mandatory parameters in the attribute set, the standard behavior data is verified. If it fails, the risk control decision is returned as failed. The risk control strategy is determined based on the data quality score of the business line. The risk control decision result is returned and recorded.
[0112] Figure 3 This is a risk control timing diagram based on data quality assessment in this embodiment. Figure 3 As shown, first, the client initiates a service request to the business end, which then requests a risk control decision from the risk control system. The request includes the business line ID, event type (type of user behavior event), and user information. The risk control system obtains the data quality score of the business line (the specific scoring method can be referred to in the above embodiment and will not be repeated here), as well as the user behavior time series pattern (specifically, it can be derived based on the above-mentioned user active period indicators, the activity intensity of different business lines, and other data). The risk control system then performs mandatory parameter verification. If the parameter verification fails, the risk control system returns a verification failure result to the business end. The business end sends a prompt to the client, indicating that manual review or additional information is required. If the parameter verification passes, the risk control system determines the data quality score of the business line obtained above. If the business line is high-quality, a lightweight risk assessment (core rule verification) is performed to detect any abnormalities in the user behavior time series (for example, whether the operation speed, sequence, and frequency are abnormal). If the core rules pass and the time series pattern is normal, the risk control system returns a decision result to the business end. The business end continues the business process with the customer. If the core rules fail or the timing pattern is abnormal, the risk control system returns a decision rejection to the business side, which notifies the customer of a risk control interception. If the business line is low-quality, comprehensive parameter verification is performed. If comprehensive parameter verification passes, the full rule set is verified, and in-depth time series analysis (focusing on the sequence, time intervals, and pattern changes of user behavior) is performed. If the full rule set verification passes and the timing analysis is normal, cross-business line data is obtained, and a cross-business line timing consistency analysis is performed. A comprehensive decision is then made, including a comprehensive decision based on the rule results and timing characteristics. If the comprehensive decision passes, the risk control system returns a decision approval to the business side, which continues the business process with the customer. If the comprehensive decision fails, the risk control system returns a decision rejection to the business side, which notifies the customer of a risk control interception. If full rule verification fails or the timing analysis is abnormal, the risk control system returns a decision rejection to the business side, which notifies the customer of a risk control interception. If comprehensive parameter verification fails, the risk control system returns a verification failure to the business side, which notifies the customer that manual review or additional information is required.
[0113] Figure 4 is a timing diagram of data quality detection in some embodiments, such as Figure 4As shown, the data collection layer on the business line uploads business data (i.e., the target business data mentioned above), the data collection layer performs data cleaning and processing, and transmits the cleaned data (standard behavior data) to the risk control center. The risk control center sequentially performs integrity dimension detection, accuracy dimension detection, rationality dimension detection, and logical consistency dimension detection. Based on the data quality detection of each dimension, a comprehensive score for a single piece of data is obtained, and then the business line score is summarized. The difference between the summarized business line score and the baseline score is calculated, and the score difference data is transmitted to the alarm monitoring system. The alarm monitoring system performs quality degradation detection to determine the degree of data quality degradation. When quality degradation is detected, the corresponding level of data quality warning is generated according to the degradation degree classification and the corresponding degradation count accumulation (accumulated number of degradations). The risk control center saves the business line quality scoring results.
[0114] For example, if the login time of a user in a certain department is too long, the cause is located and it is found that there is a long historical risk control process and a business bottleneck. In order to reduce the risk control time while stabilizing the risk control failure rate, the data quality detection method of the above embodiment can be used to construct an attribute set, establish a mapping relationship between the business field and the attribute set field, and convert the target business data reported by the business line into standard behavior data. Perform a data quality assessment on the standard behavior data. If the business data quality is high after the assessment, the risk control process can be simplified. For subsequent user logins, the behavioral risk control only needs to be verified based on the core rules and based on the user's normal active period, and the risk control results can be returned.
[0115] This embodiment also provides a data quality detection device, which is used to implement the above-mentioned embodiments and preferred embodiments. Details that have already been described will not be repeated. The terms "module," "unit," "subunit," etc. used below refer to a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.
[0116] Figure 5 This is a structural block diagram of the data quality detection device of this embodiment. Figure 5 As shown, the data quality detection device 50 includes: an acquisition module 51, a conversion module 52, a quality detection module 53, a degradation detection module 54 and an early warning module 55; wherein:
[0117] An acquisition module 51 is used to acquire target business data associated with each business line in real time; the target business data is generated based on user behavior events of different users in each business line;
[0118] a conversion module 52 for converting target business data into standard behavior data according to attribute sets preset for different user behavior events;
[0119] The quality detection module 53 is used to perform quality detection on the standard behavior data according to different quality detection dimensions, and obtain dimension scores corresponding to each quality detection dimension; and perform weighted processing on the scores of each dimension to obtain real-time data quality scores for each business line;
[0120] The degradation detection module 54 is configured to obtain the quality score statistics corresponding to the previous degradation detection cycle for each business line, compare the current data quality score of each business line with the quality score statistics corresponding to the previous degradation detection cycle, and determine the target degradation level for each business line; the target degradation level is one of multiple preset data quality degradation levels;
[0121] The early warning module 55 is used to determine and execute a quality early warning mechanism corresponding to the target degradation degree of each business line according to the target degradation degree of each business line; wherein different data quality degradation degrees correspond to different quality early warning mechanisms.
[0122] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0123] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.
[0124] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0126] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.
[0127] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.
[0128] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.
Claims
1. A data quality detection method, characterized in that: include: Obtain target business data associated with each business line in real time; The target business data is generated based on user behavior events of different users in each business line; Converting the target business data into standard behavior data according to attribute sets preset for different user behavior events; According to different quality detection dimensions, quality detection is performed on the standard behavior data respectively, and dimension scores corresponding to each quality detection dimension are obtained; Perform weighted processing on the scores of each dimension to obtain a real-time data quality score for each business line; Obtaining statistical results of quality scores corresponding to the previous degradation detection cycle for each business line, comparing the current data quality score of each business line with the statistical results of quality scores corresponding to the previous degradation detection cycle, and determining a target degradation degree for each business line; The target degradation degree belongs to one of a plurality of preset data quality degradation degrees; According to the target degradation degree of each business line, a quality warning mechanism corresponding to the target degradation degree is determined and executed; wherein different data quality degradation degrees correspond to different quality warning mechanisms.
2. The data quality detection method according to claim 1, characterized in that: The attribute set includes a standard attribute set and an extended attribute set; wherein, the standard attribute sets corresponding to different user behavior events are the same, and the extended attribute sets corresponding to different user behavior events are different.
3. The data quality detection method according to claim 1, characterized in that: The quality detection dimensions include: completeness detection, accuracy detection, rationality detection and logical consistency detection.
4. The data quality detection method according to claim 1, characterized in that: According to the attribute sets preset for different user behavior events, the target business data is converted into standard behavior data, including: Converting the target business data into initial behavior data according to attribute sets preset for different user behavior events; Perform field mapping detection on the initial behavior data to obtain standard behavior data.
5. The data quality detection method according to claim 1, characterized in that: Comparing the current data quality score of each business line with the quality score statistics corresponding to the previous degradation detection cycle to determine the target degradation degree of each business line, including: Calculating the difference between the current data quality score of each business line and the quality score statistical result corresponding to the previous degradation detection cycle; The target degradation degree of each business line is determined based on the difference and a preset difference threshold.
6. The data quality detection method according to claim 1, characterized in that: Determine and execute a quality warning mechanism corresponding to the target degradation degree of each business line according to the target degradation degree; wherein different data quality degradation degrees correspond to different quality warning mechanisms, including: Determining the cumulative number of degradations achieved by each business line at the target degradation level; According to the cumulative number of degradations, it is determined whether each business line meets the warning condition corresponding to the target degradation degree; if so, a quality warning mechanism corresponding to the target degradation degree is determined and executed.
7. The data quality detection method according to claim 1, characterized in that: The method further comprises: According to the preset monthly scoring dimensions, the monthly data quality score of the standard behavior data of each business line is scored on a monthly basis to obtain the monthly data quality score results of each business line; the monthly scoring dimensions include: the missing rate of core weight fields, the logical consistency ratio, the expectation-unsatisfied data ratio, and the data processing delay time rate; Adjust the data quality detection strategy for each business line in the next month based on the data quality scoring results of the current month.
8. The data quality detection method according to claim 1, characterized in that: The method further comprises: Integrate the standard behavior data of each business line to obtain user global behavior data constructed according to the user's unique identifier; According to the preset statistical granularity, the user's global behavior data is statistically analyzed for active periods to obtain a user active period indicator; Based on the user's global behavior data, determine the user's activity intensity in each business line and the coupling jump pattern between the business lines.
9. The data quality detection method according to any one of claims 1 to 8, characterized in that: The method further comprises: Based on the real-time data quality score of each business line, a risk control strategy corresponding to the data quality score is executed for each business line.
10. A data quality detection device, characterized in that: include: Acquisition module, conversion module, quality detection module, degradation detection module and early warning module; among which: The acquisition module is used to acquire target business data associated with each business line in real time; the target business data is generated based on user behavior events of different users in each business line; The conversion module is configured to convert the target business data into standard behavior data according to attribute sets preset for different user behavior events; The quality detection module is used to perform quality detection on the standard behavior data according to different quality detection dimensions, and obtain dimension scores corresponding to each quality detection dimension; and perform weighted processing on each dimension score to obtain a real-time data quality score for each business line; The degradation detection module is configured to obtain statistical results of quality scores corresponding to the previous degradation detection cycle for each business line, compare the current data quality score of each business line with the statistical results of quality scores corresponding to the previous degradation detection cycle, and determine a target degradation degree for each business line; the target degradation degree is one of a plurality of preset data quality degradation degrees; The early warning module is used to determine and execute a quality early warning mechanism corresponding to the target degradation degree of each business line according to the target degradation degree of the business line; wherein different data quality degradation degrees correspond to different quality early warning mechanisms.