Quality evaluation method and device for network resource data, medium and product
By quantifying the hierarchical attributes and maintenance methods of network resource data, and using cluster fitting models to calculate the credibility evaluation value, the problem of inaccurate evaluation of network resource data in the existing technology is solved, and a higher Data assessment accuracy and credibility.
Patent Information
- Application Number
- CN202410967015.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-18
AI Technical Summary
The existing network resource data quality evaluation methods are not accurate enough to effectively represent the credibility of the data, resulting in data being unusable in some scenarios.
By quantifying the importance and automation of the data based on the hierarchical attributes and maintenance methods of the resource objects to be evaluated, combining with pre-trained cluster fitting models, the credibility evaluation value of the data is calculated, and getting rid of human experience and data Constraints of isolated assumptions.
It improves the accuracy and reliability of data evaluation, provides more accurate data services, and enhances the credibility and reliability of network resource data in business scenarios.
Smart Images

Figure CN118964340A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data quality assessment, and in particular to a method, device, medium and product for assessing the quality of network resource data. Background Art
[0002] In the operation support system, network resource data is mainly managed in a unified manner in the resource management system, involving a wide range of basic network types and data types, with a huge data volume, providing data services for home customer service activation, network planning and design, fault monitoring, asset inventory, maintenance fee settlement, etc. Therefore, the accuracy of resource data is one of the key guarantees for major operators.
[0003] The existing network resource data quality assessment methods are still not accurate enough. It often happens that although the data verification indicators are high, they do not mean that the data is accurate and reliable. In some scenarios, the data is still unusable. Summary of the invention
[0004] Compared with the prior art, the present invention proposes a method, device, medium and product for evaluating the quality of network resource data, which provide more accurate and reliable data services for business management.
[0005] An embodiment of the present invention provides a method for evaluating the quality of network resource data, the method comprising:
[0006] Quantifying the importance of the data to be evaluated according to the hierarchical attributes of the resource objects of the data to be evaluated, and obtaining a quantified importance value of the data to be evaluated;
[0007] According to the maintenance mode of each attribute of the resource object of the data to be evaluated, the degree of automation of data maintenance is quantified to obtain a quantified value of the degree of automation of the data to be evaluated;
[0008] Quantifying the data quality of the data to be evaluated and determining the quality quantification values of different indicators;
[0009] The credibility evaluation value of the data to be evaluated is calculated according to the quantitative value of the importance, the quantitative value of the automation degree and the quantitative values of the quality of different indicators.
[0010] Preferably, calculating the credibility evaluation value of the data to be evaluated according to the importance quantified value, the automation degree quantified value and the quality quantified values of different indicators includes:
[0011] The importance quantified value, the automation degree quantified value and the quality quantified values of different indicators are used as evaluation items of the data to be evaluated;
[0012] Fitting the evaluation items according to the pre-trained cluster fitting model, and calculating the weights of different evaluation items;
[0013] The credibility evaluation value is obtained by performing weighted summation on the weights of different evaluation items and the quantized values of the corresponding evaluation items.
[0014] Preferably, the importance of the data to be evaluated is quantified according to the hierarchical attributes of the resource objects of the data to be evaluated to obtain the quantified importance value of the data to be evaluated, including:
[0015] Obtaining hierarchical attributes of the resource object of the data to be evaluated;
[0016] Matching corresponding importance quantization values from a preset importance matching library according to the hierarchical attributes of the resource objects of the data to be evaluated;
[0017] The hierarchical attribute includes at least one of level, type or hierarchy.
[0018] Preferably, according to the maintenance mode of each attribute of the resource object of the data to be evaluated, the degree of automation of data maintenance is quantified to obtain the quantified value of the degree of automation of the data to be evaluated, including:
[0019] Obtaining the maintenance method of each attribute of the resource object of the data to be evaluated;
[0020] Calculate the proportion of the attributes whose preset necessary attributes centralized maintenance mode is automatic collection or automatic system maintenance for each attribute;
[0021] The automation degree quantitative value is determined according to the calculated ratio.
[0022] Preferably, the data quality of the data to be evaluated is quantified to determine the quality quantification values of different indicators, including:
[0023] Extract a preset number of fields from the data to be evaluated for verification, and detect whether the extracted fields meet the requirements of the verification rules preset for indicators of different dimensions;
[0024] The verification pass rates of indicators in different dimensions are counted separately as the direct data quality quantification values of the corresponding indicators.
[0025] Preferably, the method further comprises:
[0026] Determine the parent resource and child resource corresponding to the indicator of the data to be evaluated according to the dependency relationship of the network resource data;
[0027] Calculate the quality quantification values of different indicators of the sub-resources of the data to be evaluated;
[0028] The direct data quality quantification values of different indicators of the data to be evaluated, the number of sub-resources and the direct data quality quantification values of the indicators corresponding to the sub-resources are input into a preset comprehensive evaluation model to calculate a comprehensive evaluation value as the indirect data quality quantification value of the indicator.
[0029] As a preferred solution, the process of constructing the cluster fitting model includes:
[0030] Annotate the credibility of the acquired training data set;
[0031] According to the importance quantified value and the automation quantified value in the training data set, a K-means algorithm is introduced to perform cluster analysis on the data, and the training data set is divided into multiple intervals;
[0032] The labeled training data set is fitted according to the divided multiple intervals using a preset linear regression algorithm to obtain the cluster fitting model.
[0033] As a preferred embodiment, the method further comprises:
[0034] Calculate the credibility evaluation values of different data to be evaluated in the acquired network resource data;
[0035] The network resource data is screened according to the credibility evaluation values of different data to be evaluated and the preset credibility degree intervals.
[0036] An embodiment of the present invention further provides a quality assessment device for network resource data, the device comprising:
[0037] An importance quantification module, used to quantify the importance of the data to be evaluated according to the hierarchical attributes of the resource objects of the data to be evaluated, and obtain a quantified value of the importance of the data to be evaluated;
[0038] An automation quantification module, used to quantify the degree of automation of data maintenance according to the maintenance mode of each attribute of the resource object of the data to be evaluated, so as to obtain a quantified value of the degree of automation of the data to be evaluated;
[0039] A quality quantification module is used to quantify the data quality of the data to be evaluated and determine the quality quantification values of different indicators;
[0040] The credibility evaluation module is used to calculate the credibility evaluation value of the data to be evaluated according to the quantitative value of the importance, the quantitative value of the automation degree and the quantitative values of the quality of different indicators.
[0041] An embodiment of the present invention also provides a quality assessment device for network resource data, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a quality assessment method for network resource data as described in any of the above embodiments.
[0042] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a quality assessment method for network resource data as described in any of the above embodiments.
[0043] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.
[0044] Compared with the prior art, the present invention provides a quality assessment method, device, medium and product for network resource data, which quantifies the importance of the data to be assessed according to the hierarchical attributes of the resource objects of the data to be assessed, and obtains the quantified value of the importance of the data to be assessed; quantifies the degree of automation of data maintenance according to the maintenance method of each attribute of the resource objects of the data to be assessed, and obtains the quantified value of the degree of automation of the data to be assessed; quantifies the data quality of the data to be assessed, and determines the quality quantified values of different indicators; calculates the credibility assessment value of the data to be assessed according to the quantified value of importance, the quantified value of automation and the quality quantified values of different indicators. The present application scheme quantifies the data from the perspectives of importance and automation respectively through the hierarchical attributes of network resource data and the maintenance method of each attribute of the data to be assessed, gets rid of the constraints of human experience and data isolation assumptions, and provides more accurate and reliable data services for business management. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a flow chart of a method for evaluating the quality of network resource data provided by an embodiment of the present invention;
[0046] Figure 2 Schematic diagram of the process of constructing a cluster fitting model provided by an embodiment of the present invention;
[0047] Figure 3 is a schematic diagram of the principle of a clustering analysis algorithm provided by an embodiment of the present invention;
[0048] Figure 4 It is a structural schematic diagram of a quality assessment device for network resource data provided by an embodiment of the present invention;
[0049] Figure 5It is another structural schematic diagram of a network resource data quality assessment device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] In view of the above technical problems, this application proposes a quality assessment method for network resource data, see Figure 1 , is a flow chart of a method for evaluating the quality of network resource data provided by an embodiment of the present invention, the method comprising the following steps:
[0052] Step S1, quantifying the importance of the data to be evaluated according to the hierarchical attributes of the network resource data, and obtaining a quantified importance value of the data to be evaluated;
[0053] Step S2, quantifying the degree of automation of data maintenance according to the maintenance mode of each attribute of the data to be evaluated, and obtaining a quantified value of the degree of automation of the data to be evaluated;
[0054] Step S3, quantifying the data quality of the data to be evaluated, and determining the quality quantification values of different indicators;
[0055] Step S4, calculating the credibility evaluation value of the data to be evaluated according to the importance quantified value, the automation degree quantified value and the quality quantified values of different indicators.
[0056] In the specific implementation of this embodiment, when evaluating the quality of network resource data, the verification rules are formulated, the verification results are quantified, and different weights are assigned to different verification dimensions to evaluate the data quality. However, it has been proven that this evaluation method is still not accurate enough. It often happens that although the data verification indicators are high, the data are still unusable in certain scenarios. High data verification indicators do not mean that the data is credible.
[0057] Therefore, in response to the above technical problems, this case provides a quality assessment method for network resource data and improves the data quantification in various dimensions, including:
[0058] The hierarchical attributes of the network provide a basis for quantifying the importance of data, that is, the importance of the data to be evaluated is quantified according to the hierarchical attributes of the network resource data, so that the closer the resources are to the core layer and the backbone layer, the higher the credibility assessment requirements of the data, and the quantitative value of the importance of the data to be evaluated is obtained.
[0059] When quantifying the degree of automation of data maintenance, the degree of automation of data maintenance is quantified according to the maintenance mode of each attribute of each resource object to obtain the quantified value of the degree of automation of the data to be evaluated.
[0060] The data quality of the data to be evaluated is quantified according to a preset data quality quantification method to determine the quality quantification values of different indicators.
[0061] The importance quantified value, the automation degree quantified value and the quality quantified values of different indicators are input into the credibility evaluation model to calculate the credibility evaluation value of the data to be evaluated.
[0062] According to the above quantitative results, the overall calculation model of the credibility evaluation model of network resource data can be expressed as: (S, F) = f(x, y, z 1.1 ,z 1.2 ,...,z m.n )
[0063] Input: x is the quantitative value of the importance of the data to be evaluated, y is the quantitative value of the degree of automation of the data to be evaluated, z n.m is the data quality metric of the indicators in each dimension, the output S of the model is whether it is credible, and F is the comprehensive evaluation metric of credibility.
[0064] This solution quantifies the data from the perspective of importance and automation through the hierarchical attributes of network resource data and the maintenance method of each attribute of the data to be evaluated. The degree of automation indicates whether the data is automatically acquired. Automatically acquired data is better than manually input or manually set data, and its data credibility is higher. The indicator evaluation weights are all based on real data, which eliminates human subjective factors to a certain extent. Avoid the influence weights of various indicators in the existing technology, which are derived based on basic probability data relying on human experience, and there are many subjective factors that affect accuracy.
[0065] It should be noted that the above embodiment only discloses a credibility evaluation method for each resource to be evaluated in the network resource data. In specific implementation, each data in the network resource data can be used as a resource to be evaluated for credibility evaluation.
[0066] quantifying the importance of each data according to the hierarchical attribute of the resource object of each data in the network resource data to obtain a quantified value of the importance of each data;
[0067] According to the maintenance mode of each attribute of each data resource object, the automation degree of data maintenance is quantified to obtain the automation degree quantified value of each data;
[0068] Quantify the data quality of each data and determine the quality quantitative values of different indicators;
[0069] The credibility evaluation value of each data is calculated according to the importance quantified value, the automation degree quantified value and the quality quantified values of different indicators.
[0070] The network resource data is screened according to the credibility evaluation value of each data, and the screened data is used to provide data services for management business.
[0071] This solution improves the data quantification of each dimension, intelligently calculates the impact of each dimension on the data, gets rid of the constraints of human experience and data isolation assumptions, and builds a more accurate credibility assessment model for a single network resource data. The credibility assessment value obtained is highly credible. After determining the credibility of the network resource data, the data of the object to be evaluated in the network resource data is screened according to the credibility assessment value, thereby improving the reliability of the network resource data in providing data services in various business scenarios.
[0072] In another embodiment provided by the present invention, the step S4 specifically includes:
[0073] The quantitative values of the importance of the data to be evaluated, the quantitative values of the degree of automation, and the quantitative values of the quality of different indicators are used as evaluation items of the overall calculation model of the credibility assessment model.
[0074] After fitting all the labeled data, the weights of each evaluation item in different intervals are determined based on the cluster fitting model.
[0075] In specific implementation, based on DS evidence theory, AHP and other algorithms, the credibility weight of each dimension is calculated by adopting a single algorithm or combining it with the EWM algorithm. When citing DS evidence theory, it is necessary to evaluate the basic probability of each data distribution in different credibility intervals in advance based on expert experience; when citing the AHP algorithm, it is necessary to evaluate the relative importance between any two dimensions based on expert experience.
[0076] The weights of different evaluation items and the corresponding quantitative values of the evaluation items are weighted and summed, that is, the comprehensive credibility is calculated by weighted summation of the quantitative results and credibility weights of each dimension. The calculation formula is expressed as where s i Represents the quantitative value of each verification dimension evaluation item, w i Indicates the weight of each verification dimension evaluation item.
[0077] By comprehensively calculating each evaluation item and introducing a regression algorithm to intelligently calculate the weight, the credibility evaluation value is obtained, which avoids the subjective limitations of human experience, builds a more accurate credibility evaluation model for a single network resource data, and improves the accuracy of credibility evaluation.
[0078] In another embodiment provided by the present invention, the step S1 specifically includes:
[0079] Obtaining the hierarchical attributes of the resource objects of the data to be evaluated provides the main basis for quantifying the importance of the data according to the hierarchical attributes of the network. The closer the resources are to the core layer and backbone layer, the higher the credibility evaluation requirements of the data.
[0080] For the network's hierarchical attributes of the provincial core layer and cross-provincial resources, the importance of its data is extremely important, and its importance is quantified as 4.
[0081] For the network's hierarchical attributes of provincial aggregation layer and secondary trunk resources, the importance of its data is very important, and its importance is quantified as 3.
[0082] For the hierarchical attributes of the network, which are the metropolitan access layer and between the central office access point and the client-side access point, the importance of the data is relatively important, and its importance is quantified as 2.
[0083] The hierarchical attribute of the network is the terminal access layer resources, the importance of its data is generally important, and its importance is quantified as 1.
[0084] Most resource data have maintained similar attributes such as level / category / hierarchy directly or indirectly through association / inheritance relationships. This proposal quantifies these attributes and builds an importance matching library based on the hierarchical attributes of the network in advance. The importance matching library for resources in the space and transmission professions is shown in Table 1:
[0085] Table 1, Importance matching library
[0086]
[0087]
[0088] The corresponding importance quantization value is matched from a preset importance matching library according to the hierarchical attribute of the resource object of the data to be evaluated, wherein the hierarchical attribute includes at least one of level, type or hierarchy.
[0089] When quantifying data in this case, the dependency of network resource data was taken into consideration. The data was no longer evaluated based on the assumption of mutual independence. The fact that network resource data is correlated was respected. The impact of different levels on data accuracy was also taken into consideration. To a certain extent, the contradiction of high-quality but unusable data was eliminated, making the evaluation results more accurate.
[0090] In another embodiment provided by the present invention, the step S2 specifically includes:
[0091] Obtaining the maintenance method of each attribute of the resource object of the data to be evaluated;
[0092] The attribute maintenance method, that is, the data maintenance method of different attributes, does not at least include automatic collection or system automation methods, but may also include other methods, such as manual collection methods and manual input methods.
[0093] According to the maintenance mode of each attribute of each resource object, the proportion of automatically collected and automatically maintained attributes in the necessary attribute set is counted, and the quantitative value of the automation degree is determined according to the calculated proportion.
[0094] The proportion of automatically collected data in the necessary attribute set is used as a quantitative basis for the degree of automation of data maintenance.
[0095] Taking IP bearer-network element as a resource object as an example, the required and conditionally required attributes are defined as necessary attributes. There are 39 necessary attributes in total, and 21 attributes can be automatically maintained. The proportion of IP network elements is 21 / 39*100=53.8. The quantitative value of the degree of automation is determined according to the proportion of the attributes.
[0096] It should be noted that the automation degree quantification value is determined according to the calculated ratio. Specifically, the ratio may be directly used as the automation degree quantification value, or the automation degree quantification value may be matched according to the ratio according to a preset matching relationship.
[0097] Taking into account the impact of different maintenance methods on data accuracy in network resource data, the contradiction of high data quality but unavailability is eliminated to a certain extent, making the evaluation results more accurate.
[0098] In another embodiment provided by the present invention, the step S3 specifically includes a direct quality score quantization process, specifically including:
[0099] Extract a preset number of fields from the data to be evaluated for verification, and detect whether the extracted fields meet the requirements of the verification rules preset for indicators of different dimensions;
[0100] The verification pass rates of indicators in different dimensions are counted separately as the direct data quality quantitative values of the corresponding indicators.
[0101] The obtained direct data quality quantitative index is used as the quality quantitative value of the index.
[0102] In specific implementation, the direct data quality score will be quantified and composed of 11 indicators in six dimensions: completeness, compliance, standardization, uniqueness, consistency, and activity.
[0103] For the completeness indicator of the completeness dimension, when quantifying the indicator, it is quantified based on the number of required and conditionally required fields that pass the integrity check of a single data record, so the completeness rate = the number of fields that pass the integrity check / the total number of required & conditionally required fields*100.
[0104] For the cross-professional correlation rate indicator in the compliance dimension, when quantifying the indicator, it is quantified based on the number of cross-professional correlation fields that pass the verification for a single piece of data. Cross-professional correlation rate = cross-professional correlation verification passed fields / total number of cross-professional correlation verification fields*100.
[0105] For the business compliance rate indicator in the compliance dimension, when quantifying the indicator, it is quantified based on the number of business compliance verification rules passed for a single piece of data. Business compliance rate = number of rules passed / total number of business compliance rules * 100.
[0106] For the end-to-end completeness rate indicator in the compliance dimension, when quantifying the indicator, it is quantified based on the number of end-to-end verification rules passed for a single piece of data. The quantification result of the end-to-end completeness = the number of end-to-end rules that passed the verification / the total number of end-to-end rules*100.
[0107] For the intra-professional correlation rate indicator of the normative dimension, when quantifying the indicator, it is quantified according to the number of related fields of a single data that pass the intra-professional correlation check. Intra-professional correlation rate = number of fields that pass the intra-professional correlation check / total number of related verification fields*100.
[0108] For the enumeration standardization rate indicator of the standardization dimension, when quantifying the indicator, it is quantified based on the number of enumeration fields that pass the verification for a single piece of data. Enumeration standardization rate = number of fields that pass the enumeration verification / total number of enumeration fields*100.
[0109] For the naming compliance rate indicator of the normative dimension, when quantifying the indicator, it is quantified according to the number of naming compliance verification rules passed for a single piece of data. Naming compliance rate = number of rules passed / total number of naming compliance verification rules * 100.
[0110] For the uniqueness indicator of the uniqueness dimension, when quantifying the indicator, quantification is performed based on whether the unique primary key of a single piece of data is repeated, not limited to UID, IP, and name. If any primary key is repeated, the quantification is 0; otherwise, the quantification is 100.
[0111] For the consistency rate indicator of the consistency dimension, when quantifying the indicator, consistency refers to whether the data is consistent with the external system data such as faults, signaling, and professional workbenches. The consistency quantification result of each system is the percentage of the number of fields that are consistent in the comparison to the total number of fields involved in the comparison, and the consistency quantification result of all systems is the average of each system. That is, consistency rate = avg (number of fields that are consistent in each system / number of fields involved in the comparison * 100).
[0112] For the silent time indicator of the activity dimension, when quantifying the indicator, the silent time refers to the time from the last data change to the current time. The longer the silent time, the lower the credibility. The quantified result of silent time = (current time - last update time) / (current time - network access time) * 100.
[0113] For the external usage time indicator of the activity dimension, when quantifying the indicator, we judge whether the data is used externally and the usage time based on the feedback from work orders such as business activation, fault dispatch, and inspection supporting the external system. The quantification result of external usage time = (current time - the most recent usage time of the external system) / (current time - network access time) * 100.
[0114] When quantifying indicators, the quantification of data quality is accurate enough. The quantification is based on the verification of atomic-level evaluation items in each dimension through fields and the proportion of the number of rules. Intermediate results are allowed, which can objectively reflect the overall situation of each dimension and avoid information loss caused by one-size-fits-all quantification.
[0115] In another embodiment provided by the present invention, on the basis of the direct quality score quantization in step S3, the specific implementation of this case also includes an indirect quality score quantization process, specifically including:
[0116] The dependency relationship of network resource data includes the correspondence between parent resources and child resources.
[0117] For example, the sub-resource of a network element is a port, the sub-resource of a computer room is a network element, and the sub-resource of an optical cable is a fiber core.
[0118] Furthermore, the parent resource with child resources in the indicator of the data to be evaluated and the child resources corresponding to the parent resource can be determined according to the dependency relationship of the network resource data.
[0119] During specific implementation, the resource object indicator quality quantification method may adopt the solution in the above embodiment to calculate the quality quantification values of different indicators of the resource object as a sub-resource.
[0120] The direct data quality quantification values of different indicators of the data to be evaluated, the number of sub-resources and the direct data quality quantification values of the indicators corresponding to the sub-resources are input into a preset comprehensive evaluation model, and the comprehensive evaluation value is calculated to obtain the indirect data quality quantification value of the indicator.
[0121] The obtained indirect data quality quantitative index is used as the quality quantitative value of the index.
[0122] For example, if the data quality of a parent resource is directly evaluated as Z1, and there are N child resources, each with an evaluation value of S n , then the final evaluation value of the parent resource
[0123] When N is 0, that is, there is no child resource under the parent resource, Z2 is directly 0.
[0124] That is, for two or more resources with dependencies, if the quality of the child resource is not high, the quality of the parent resource will be reduced accordingly, and the quality quantification value of the indicator will be updated.
[0125] Based on the fact that network resource data are correlated, independent data without sub-resources indicates that there are problems with its data quality, and the quantitative value of its data network quality is low. Taking into account the dependency of network resource data, the data is no longer evaluated based on the assumption of mutual independence, thereby improving the reliability of data quality assessment.
[0126] In another embodiment provided by the present invention, the process of constructing a cluster fitting model specifically includes a labeling phase, a training phase, and a service publishing phase.
[0127] During the annotation, the obtained training data set is annotated with a credibility level. See Table 2, which is a data annotation format table provided in an embodiment of the present invention:
[0128] Table 2, Data annotation format table
[0129]
[0130] The obtained training data set is labeled with a degree of credibility according to the format of Table 2.
[0131] In the training phase, this proposal no longer refers to expert experience for the weight of each evaluation indicator, but introduces a linear regression algorithm, intelligent training and tuning. The entire process is mainly divided into three parts: labeling, training, and service release. Figure 2 , is a schematic diagram of the construction process of the cluster fitting model provided by an embodiment of the present invention.
[0132] According to the formula derivation of the subjective weighting algorithm, the data credibility is ultimately calculated by the weighted sum of the evaluation results of each dimension and its credibility measurement weight, which is the main basis for this proposal to select the linear regression algorithm. On this basis, the ridge regression or LASSO regression algorithm is simultaneously introduced to correct the deviation. The purpose of this stage is to calculate the weight of each indicator based on the labeled data fitting.
[0133] Before fitting calculation, we first introduce K-means algorithm to perform cluster analysis on the data according to the importance of the data and the degree of maintenance automation, see Figure 3 , is a schematic diagram of the principle of the clustering analysis algorithm provided in an embodiment of the present invention, wherein the labeled training data set is divided into multiple intervals, and then the weights in each interval are determined to ensure the consistency of the data evaluation standard.
[0134] That is, after fitting all the labeled data, the weights of each evaluation item in different intervals and the thresholds of different credibility levels are determined, the data credibility modeling is completed, and it is encapsulated as a service to obtain the cluster fitting model for subsequent calls.
[0135] Based on the existing data quality assessment method, this scheme improves the data quantification of each dimension, proposes a hierarchical assessment method based on linear regression algorithm, intelligently calculates the weight of each influencing factor, gets rid of the constraints of human experience and data isolation assumptions, and improves the accuracy of credibility assessment.
[0136] In another embodiment provided by the present invention, the method further includes:
[0137] Calculate the credibility evaluation values of different data to be evaluated in the acquired network resource data;
[0138] The network resource data is screened according to the credibility evaluation values of different data to be evaluated and the preset credibility degree intervals.
[0139] This solution calculates the credibility evaluation value of each data according to the importance quantification value, the automation degree quantification value and the quality quantification values of different indicators on the basis of credibility evaluation calculation of the data to be evaluated.
[0140] The network resource data is screened according to the credibility evaluation value of each data and the preset credibility level interval, and the data in the credibility level interval is screened out.
[0141] Use the filtered data to provide data services for business management and provide reliable data services for business management.
[0142] See also Figure 4 , is a schematic diagram of a network resource data quality assessment device provided by an embodiment of the present invention, the device comprising:
[0143] An importance quantification module, used to quantify the importance of the data to be evaluated according to the hierarchical attributes of the resource objects of the data to be evaluated, and obtain a quantified value of the importance of the data to be evaluated;
[0144] An automation quantification module, used to quantify the degree of automation of data maintenance according to the maintenance mode of each attribute of the resource object of the data to be evaluated, so as to obtain a quantified value of the degree of automation of the data to be evaluated;
[0145] A quality quantification module is used to quantify the data quality of the data to be evaluated and determine the quality quantification values of different indicators;
[0146] The credibility evaluation module is used to calculate the credibility evaluation value of the data to be evaluated according to the quantitative value of the importance, the quantitative value of the automation degree and the quantitative values of the quality of different indicators.
[0147] It should be noted that the network resource data quality assessment device provided in this embodiment can execute all the steps and functions of the network resource data quality assessment method provided in any of the above embodiments, and the specific functions of the device are not described in detail here.
[0148] See also Figure 5 , is another structural diagram of a network resource data quality assessment device provided by an embodiment of the present invention. The network resource data quality assessment device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a network resource data quality assessment program. When the processor executes the computer program, the steps in each of the above-mentioned network resource data quality assessment method embodiments are implemented, such as Figure 1 Alternatively, the processor implements the functions of each module in the above-mentioned device embodiments when executing the computer program.
[0149] Exemplarily, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the quality assessment device for network resource data. For example, the computer program may be divided into several modules, and the specific functions of each module have been described in detail in a quality assessment method for network resource data provided in any of the above embodiments, and the specific functions of the device will not be repeated here.
[0150] The network resource data quality assessment device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The network resource data quality assessment device may include, but is not limited to, a processor and a memory. A person skilled in the art may understand that the schematic diagram is only an example of a network resource data quality assessment device and does not constitute a limitation on a network resource data quality assessment device. The device may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the network resource data quality assessment device may also include input and output devices, network access devices, buses, etc.
[0151] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the network resource data quality assessment device, and uses various interfaces and lines to connect various parts of the entire network resource data quality assessment device.
[0152] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the network resource data quality assessment device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0153] Wherein, if the module integrated in the quality assessment device of the network resource data is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0154] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.
[0155] It should be noted that the computer program product provided in this embodiment can execute all the steps and functions of a network resource data quality assessment method provided in any of the above embodiments, and the specific functions of the device are not described in detail here.
[0156] It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for evaluating the quality of network resource data, characterized in that: The method comprises: Quantifying the importance of the data to be evaluated according to the hierarchical attributes of the resource objects of the data to be evaluated, and obtaining a quantified importance value of the data to be evaluated; According to the maintenance mode of each attribute of the resource object of the data to be evaluated, the degree of automation of data maintenance is quantified to obtain a quantified value of the degree of automation of the data to be evaluated; Quantifying the data quality of the data to be evaluated and determining the quality quantification values of different indicators; The credibility evaluation value of the data to be evaluated is calculated according to the quantified value of the importance, the quantified value of the automation degree and the quantified values of the quality of different indicators.
2. A method for evaluating the quality of network resource data according to claim 1, characterized in that: Calculating the credibility evaluation value of the data to be evaluated according to the importance quantified value, the automation degree quantified value and the quality quantified values of different indicators includes: The importance quantified value, the automation degree quantified value and the quality quantified values of different indicators are used as evaluation items of the data to be evaluated; Fitting the evaluation items according to the pre-trained cluster fitting model, and calculating the weights of different evaluation items; The credibility evaluation value is obtained by performing weighted summation on the weights of different evaluation items and the quantized values of the corresponding evaluation items.
3. A method for evaluating the quality of network resource data according to claim 1, characterized in that: The importance of the data to be evaluated is quantified according to the hierarchical attributes of the resource objects of the data to be evaluated, and a quantified importance value of the data to be evaluated is obtained, including: Obtaining hierarchical attributes of the resource object of the data to be evaluated; Matching corresponding importance quantization values from a preset importance matching library according to the hierarchical attributes of the resource objects of the data to be evaluated; The hierarchical attribute includes at least one of level, type or hierarchy.
4. A method for evaluating the quality of network resource data according to claim 1, characterized in that: According to the maintenance mode of each attribute of the resource object of the data to be evaluated, the degree of automation of data maintenance is quantified to obtain the quantified value of the degree of automation of the data to be evaluated, including: Obtaining the maintenance method of each attribute of the resource object of the data to be evaluated; Calculate the proportion of the attributes whose preset necessary attributes centralized maintenance mode is automatic collection or automatic system maintenance for each attribute; The automation degree quantitative value is determined according to the calculated ratio.
5. A method for evaluating the quality of network resource data according to claim 1, characterized in that: Quantify the data quality of the data to be evaluated and determine the quality quantification values of different indicators, including: Extract a preset number of fields from the data to be evaluated for verification, and detect whether the extracted fields meet the requirements of the verification rules preset for indicators of different dimensions; The verification pass rates of indicators in different dimensions are counted separately as the direct data quality quantification values of the corresponding indicators.
6. A method for evaluating the quality of network resource data according to claim 5, characterized in that: The method further comprises: Determine the parent resource and child resource corresponding to the indicator of the data to be evaluated according to the dependency relationship of the network resource data; The direct data quality quantification values of different indicators of the data to be evaluated, the number of sub-resources and the direct data quality quantification values of the indicators corresponding to the sub-resources are input into a preset comprehensive evaluation model to calculate a comprehensive evaluation value as the indirect data quality quantification value of the indicator.
7. A method for evaluating the quality of network resource data according to claim 2, characterized in that: The construction process of the cluster fitting model includes: Annotate the credibility of the acquired training data set; According to the importance quantified value and the automation quantified value in the training data set, a K-means algorithm is introduced to perform cluster analysis on the data, and the training data set is divided into multiple intervals; The labeled training data set is fitted according to the divided multiple intervals using a preset linear regression algorithm to obtain the cluster fitting model.
8. A method for evaluating the quality of network resource data according to claim 1, characterized in that: The method further comprises: Calculate the credibility evaluation values of different data to be evaluated in the acquired network resource data; The network resource data is screened according to the credibility evaluation values of different data to be evaluated and the preset credibility degree intervals.
9. A quality assessment device for network resource data, characterized in that: The device comprises: An importance quantification module, used to quantify the importance of the data to be evaluated according to the hierarchical attributes of the resource objects of the data to be evaluated, and obtain a quantified value of the importance of the data to be evaluated; An automation quantification module, used to quantify the degree of automation of data maintenance according to the maintenance mode of each attribute of the resource object of the data to be evaluated, so as to obtain a quantified value of the degree of automation of the data to be evaluated; A quality quantification module is used to quantify the data quality of the data to be evaluated and determine the quality quantification values of different indicators; The credibility evaluation module is used to calculate the credibility evaluation value of the data to be evaluated according to the quantitative value of the importance, the quantitative value of the automation degree and the quantitative values of the quality of different indicators.
10. A quality assessment device for network resource data, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a quality assessment method for network resource data as described in any one of claims 1 to 8 is implemented.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a quality assessment method for network resource data as described in any one of claims 1 to 8.
12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Method for evaluating reliability of data assets
CN105023119A
Service data quality checking method and system
CN112686527A
Network asset management method and device, equipment and medium
CN113408948A
Data asset value evaluation method and device and computer equipment
CN117273953A
Data Quality Tracking
US20090319566A1
Cited By
Credibility evaluation method and device, equipment, medium and program product
CN119721843A
