Data resource quality assessment method and device

Abstract

The invention discloses a data resource quality evaluation method and device, and the method comprises the steps: collecting data resources from different sources, extracting data, and carrying out the preprocessing of the data resources and the extracted data; the data resources comprise standard data and non-standard data; performing difference matching on the non-standard data to obtain difference degree data, and performing comparative analysis on the difference degree data and the standard data to obtain an abnormal coefficient; performing satisfaction comprehensive analysis on the extracted data to obtain a satisfaction comprehensive score, and constructing a data resource quality evaluation model according to the abnormal coefficient and the satisfaction comprehensive score; and optimizing the data resource quality evaluation model according to the evaluation error, inputting to-be-evaluated data into the data resource quality evaluation model, and outputting an evaluation result. The method not only can improve the accuracy of data resource quality evaluation, but also has good interpretability, and can be directly applied to a data resource quality evaluation system.

Description

Technical Field

[0001] The present invention relates to the field of quality assessment, and in particular to a data resource quality assessment method and device. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, data has become an important basis for corporate decision-making, scientific research and social governance. However, the sources of data resources are extensive and complex, including various standard data and non-standard data. Standard data usually follows specific formats, specifications and standards, and is easy to process and analyze; while non-standard data may come from different systems, platforms or devices, with different formats, structures and qualities, which brings great challenges to data applications.

[0003] In the application process of data resources, the quality of data directly affects the accuracy of data analysis and decision-making. Therefore, it is particularly important to evaluate the quality of data resources. Most of the existing data quality evaluation methods focus on the evaluation of single dimensions such as data integrity, accuracy, consistency and timeliness, and lack a comprehensive and systematic evaluation system. Especially for data resources containing a large amount of non-standard data, how to effectively identify and process the differences between non-standard data and standard data, and then accurately evaluate the quality of data resources, has become a problem that needs to be solved urgently.

[0004] Therefore, Kou Dai invented a new data resource quality assessment method to improve the comprehensiveness and accuracy of data resource quality assessment. Summary of the invention

[0005] The purpose of the present invention is to provide a data resource quality assessment method.

[0006] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0007] The present invention comprises the following steps:

[0008] Collecting data resources and extracted data from different sources, and preprocessing the data resources and the extracted data; the data resources include standard data and non-standard data;

[0009] Performing difference matching on the non-standard data to obtain difference data, and performing comparative analysis on the difference data and the standard data to obtain anomaly coefficients; including:

[0010] Input non-standard data into the matching model, randomly segment the non-standard data, and calculate the similarity between the non-standard data:

[0011]

[0012] The similarity between non-standard data w and non-standard data y is γ(w,y), and the i-th non-standard data vector is w i , non-standard data vector w i The neighborhood of Q(w i ), the non-standard data vector at the i-th position is y i , non-standard data vector y i The neighborhood of Q(y i ), the i-th influence multiplier is ζ i , the control constant is x, and the number of segments of non-standard data is

[0013] The non-standard data with the largest similarity are grouped into one group to obtain a regular group, and the non-standard data with a similarity less than 0.352 are grouped as a special group to obtain a grouping;

[0014] Compute the degree of match between non-standard data and conventional groups:

[0015]

[0016] The non-standard data w of the regular group z is z w , the non-standard data y of the conventional group z is z y , non-standard data z w and non-standard data y The matching degree is The number of non-standard data in the regular group z is n w , non-standard data z w The true probability distribution of w ), non-standard data z w The estimated probability distribution of w );

[0017] Calculate the difference between the non-standard data and the regular group:

[0018]

[0019] The non-standard data z w and non-standard data y The difference is β(z w ,z y ), the associated difference degree takes the difference part of the conventional group non-standard data as the first difference degree data;

[0020] Matching the special group with the standard data, calculating the difference between the standard data and the non-standard data, and associating the difference with the difference part of the special group and the non-standard data as the second difference data;

[0021] Combining the first difference data and the second difference data to output as difference data;

[0022] Performing a comprehensive satisfaction analysis on the extracted data to obtain a comprehensive satisfaction score, and constructing a data resource quality assessment model based on the abnormality coefficient and the comprehensive satisfaction score;

[0023] The data resource quality assessment model is optimized according to the assessment error, the data to be assessed is input into the data resource quality assessment model, and the assessment result is output.

[0024] Furthermore, the method of performing comparative analysis on the difference data and the standard data to obtain an abnormality coefficient includes:

[0025] The difference data and the standard data are input into the comparative analysis model, the standard data is used as the comparison library, and the similarity between the first difference data and the standard data is calculated:

[0026]

[0027] The kernel width is φ, the control factor is σ, and the jth first difference data is f j , the jth standard data is a j , the first difference data f j and standard data a j The Euclidean distance of j ,f j ), the first difference data f j and standard data a j The similarity is

[0028] The standard data with the largest similarity is used as the first control group of the first difference data, and the data with a similarity less than 0.312 is selected as the ambiguous data;

[0029] Calculate the matching degree between ambiguous data and standard data:

[0030]

[0031] The first difference data f j and ambiguous data q j The matching degree is The weight coefficient of Euclidean distance is χ, and the weight coefficient of similarity is χ o , the cosine function is cos(·), and the optimization parameter is υ;

[0032] The standard data with the highest matching degree is used as the second reference data for the ambiguous data.

[0033] The first control group and the second control data are segmented, and the segmented first control group and the second control data are sorted according to the number of abnormal digits to obtain sorted data, and the abnormal coefficient of the sorted data is calculated:

[0034]

[0035] The number of segments of sorted data is n1, and the number of abnormal bits of sorted data is n am , the quality correlation of the jth sorted data is ε j , the proportional weight is The outlier weight is The importance of sorted data is θ j , the anomaly coefficient of the sorted data f is

[0036] Furthermore, the method of performing a comprehensive satisfaction analysis on the extracted data to obtain a comprehensive satisfaction score includes:

[0037] Calculate the overall satisfaction score for the extracted data:

[0038]

[0039] The comprehensive satisfaction score of the extracted data u is D u , the jth extracted data is u j , the jth standard data is a j , extract data u j and standard data a j The similarity is The upload time of standard data is T o , the upload time of the extracted data is T u , the difficulty of extracting data u is ψ u , extract data u j The completeness is The first weight is η1, the second weight is η2, and the third weight is η3;

[0040] Taking the satisfaction comprehensive score as the analysis purpose, the decision tree algorithm is used for comprehensive analysis to obtain the predicted satisfaction comprehensive score, and the first weight, the second weight, and the third weight are adjusted according to the error between the predicted satisfaction comprehensive score and the actual satisfaction comprehensive score;

[0041] The comprehensive satisfaction score of the extracted data is recalculated based on the adjusted first weight, second weight, and third weight.

[0042] Furthermore, the method of constructing a data resource quality assessment model according to the abnormality coefficient and the comprehensive satisfaction score includes:

[0043] The quality evaluation function is constructed based on the abnormal coefficient and the comprehensive score of satisfaction. The expression is:

[0044]

[0045] The importance of the sth data is μ s , the anomaly coefficient of the sorted data f is The sth extracted data is u s , extract data u s The overall satisfaction score is The number of research data is N, and the quality assessment function is

[0046] The minimum value of the loss function between the actual value of the quality assessment function and the predicted value is used as the objective function of the data resource quality assessment model. The data resource quality assessment model includes a local outlier factor detection algorithm, a hash algorithm, and a machine learning algorithm.

[0047] The local outlier factor detection algorithm evaluates the density difference between the input data and the neighboring data, identifies outliers based on the density difference, and obtains abnormal data based on the outliers;

[0048] The hash algorithm calculates the hash values ​​of the pattern string and the text string substring of the abnormal data, compares the hash values ​​for potential matches, and compares the potential matches to obtain a matching group;

[0049] The machine learning algorithm automatically identifies and marks potential data quality issues by learning the data patterns of the matching group, and automatically evaluates the data quality issues according to the quality assessment function until the objective function reaches the minimum value, and then outputs the evaluation results.

[0050] Furthermore, the method for optimizing the data resource quality assessment model according to the assessment error includes:

[0051] The particle population is introduced, and the evaluation error minimization is used as the search strategy to perform chaotic mapping on the particle population. The expression is:

[0052]

[0053] The chaotic mapping sequence of the i+1th particle is U i+1 , the initial position sequence of the i-th particle is U i , the random number from 0 to 1 is r1, the chaotic variation factor is ω, and the control parameter is L;

[0054] The evaluation error is used as the fitness, the particle fitness is calculated, the particle with the smallest fitness is taken as the optimal solution, and the particle position is calculated:

[0055]

[0056] The exploration parameter is p, the random number is r, and the random numbers from 0 to 1 are r2, r3, r4, and r5 respectively. The current position of the zth particle in the tth iteration is The position of the zth particle at the t+1th iteration is The upper limit of the optimization variable is Ub, the lower limit of the optimization variable is Lb, the random number between -1 and 1 is α, the current number of iterations is t, and the maximum number of iterations is t max , the position of the random particle at the tth iteration is The adaptive parameter is e, the chaotic mapping sequence of the tth iteration is U(t), and the competition parameter is

[0057] According to the competition mechanism, the particle position is updated to obtain the competition position. The expression is:

[0058]

[0059] The competitive position of the zth particle in the t+1th iteration is The optimal solution is The competition operator is λ and the number of sequences is The initial position sequence of the i-th particle in the t-th iteration is U i (t);

[0060] Use the optimal solution to update particles, the expression is:

[0061]

[0062] The updated i+1th sequence is The chaotic mapping sequence of the tth iteration is U(t), and the random books from 0 to 1 are r6 and r7 respectively;

[0063] Update the particle's position according to the proportional factor to obtain the proportional position. The expression is:

[0064]

[0065] The proportional position of the zth particle in the t+1th iteration is The scaling factor is τ, and the competitive position of the zth particle in the tth iteration is

[0066] Continue iterating until the maximum number of carrying times is reached, otherwise update the particles.

[0067] In a second aspect, an embodiment of the present application further provides an electronic device, including:

[0068] A processor; and a memory arranged to store computer executable instructions, which when executed cause the processor to perform the method steps described in the first aspect.

[0069] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the method steps described in the first aspect.

[0070] The beneficial effects of the present invention are:

[0071] The present invention is a data resource quality assessment method and device. Compared with the prior art, the present invention has the following technical effects:

[0072] The present invention can improve the accuracy of data resource quality assessment through preprocessing, difference matching, comparative analysis, comprehensive analysis of satisfaction, model building and model optimization steps, thereby improving the precision of data resource quality assessment, optimizing data resource quality assessment, greatly saving resources, and improving work efficiency. It can realize automatic assessment of data resource quality, and perform comprehensive assessment and estimation optimization of data resource quality assessment in real time. It is of great significance to the quality assessment of multiple data resources, can adapt to data resource quality assessment of different standards and different data resource quality assessment needs, and has a certain universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A flowchart of the steps of a data resource quality assessment method of the present invention;

[0074] Figure 2 It is a schematic diagram of the structure of an electronic device in an embodiment of this specification. DETAILED DESCRIPTION

[0075] The present invention is further described below by means of specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0076] A data resource quality assessment method of the present invention comprises the following steps:

[0077] like Figure 1 As shown, in this embodiment, the following steps are included:

[0078] Collecting data resources and extracted data from different sources, and preprocessing the data resources and the extracted data; the data resources include standard data and non-standard data;

[0079] In the actual evaluation, the historical sales data of product B on the XXX e-commerce platform in the spring of 2021 was taken as the research object, where the historical sales data included monthly sales, product reviews, order returns, sales prices, returns and exchanges, sales cycles, inventory data, logistics data, and product quality;

[0080] The extracted data were obtained from the W website respectively;

[0081] Performing difference matching on the non-standard data to obtain difference data, and performing comparative analysis on the difference data and the standard data to obtain anomaly coefficients; including:

[0082] Input non-standard data into the matching model, randomly segment the non-standard data, and calculate the similarity between the non-standard data:

[0083]

[0084] The similarity between non-standard data w and non-standard data y is γ(w,y), and the i-th non-standard data vector is w i , non-standard data vector w i The neighborhood of Q(w i ), the non-standard data vector at the i-th position is y i , non-standard data vector y i The neighborhood of Q(y i ), the i-th influence multiplier is ζ i , the control constant is x, and the number of segments of non-standard data is n;

[0085] The non-standard data with the largest similarity are grouped into one group to obtain a regular group, and the non-standard data with a similarity less than 0.352 are grouped as a special group to obtain a grouping;

[0086] Compute the degree of match between non-standard data and conventional groups:

[0087]

[0088] The non-standard data w of the regular group z is z w , the non-standard data y of the conventional group z is z y , non-standard data z w and non-standard data y The matching degree is The number of non-standard data in the regular group z is n w , non-standard data z w The true probability distribution of w ), non-standard data z w The estimated probability distribution of w );

[0089] Calculate the difference between the non-standard data and the regular group:

[0090] β(z w ,z y )=1-l(z w ,z y )

[0091] The non-standard data z w and non-standard data y The difference is β(z w ,z y ), the associated difference degree takes the difference part of the conventional group non-standard data as the first difference degree data;

[0092] Matching the special group with the standard data, calculating the difference between the standard data and the non-standard data, and associating the difference with the difference part of the special group and the non-standard data as the second difference data;

[0093] Combining the first difference data and the second difference data to output as difference data;

[0094] In the actual evaluation, the difference data appeared in the first and second months of spring. The first month was a product evaluation problem, and the second month was a logistics problem. The anomaly coefficients in the first and second months were -0.21 and 0.053 respectively.

[0095] Performing a comprehensive satisfaction analysis on the extracted data to obtain a comprehensive satisfaction score, and constructing a data resource quality assessment model based on the abnormality coefficient and the comprehensive satisfaction score;

[0096] In the actual evaluation, the overall satisfaction score of Product B was 0.63;

[0097] Optimizing the data resource quality assessment model according to the assessment error, inputting the data to be assessed into the data resource quality assessment model, and outputting the assessment result;

[0098] In the actual evaluation, the data quality evaluation result of the W website was 69.3%.

[0099] In this embodiment, the method of performing comparative analysis on the difference data and the standard data to obtain an abnormality coefficient includes:

[0100] The difference data and the standard data are input into the comparative analysis model, the standard data is used as the comparison library, and the similarity between the first difference data and the standard data is calculated:

[0101]

[0102] The kernel width is φ, the control factor is σ, and the jth first difference data is f j , the jth standard data is a j , the first difference data f j and standard data a j The Euclidean distance of j ,f j ), the first difference data f j and standard data aj The similarity is

[0103] The standard data with the largest similarity is used as the first control group of the first difference data, and the data with a similarity less than 0.312 is selected as the ambiguous data;

[0104] Calculate the matching degree between ambiguous data and standard data:

[0105]

[0106] The first difference data f j and ambiguous data q j The matching degree is The weight coefficient of Euclidean distance is χ, and the weight coefficient of similarity is χ o , the cosine function is cos(·), and the optimization parameter is υ;

[0107] The standard data with the highest matching degree is used as the second reference data for the ambiguous data.

[0108] The first control group and the second control data are segmented, and the segmented first control group and the second control data are sorted according to the number of abnormal digits to obtain sorted data, and the abnormal coefficient of the sorted data is calculated:

[0109]

[0110] The number of segments of sorted data is n1, and the number of abnormal bits of sorted data is n am , the quality correlation of the jth sorted data is ε j , the proportional weight is The outlier weight is The importance of sorted data is θ j , the anomaly coefficient of the sorted data f is

[0111] In this embodiment, the method of performing comprehensive satisfaction analysis on the extracted data to obtain a comprehensive satisfaction score includes:

[0112] Calculate the overall satisfaction score for the extracted data:

[0113]

[0114] The comprehensive satisfaction score of the extracted data u is D u , the jth extracted data is u j , the jth standard data is a j , extract data u j and standard data a j The similarity is The upload time of standard data is To , the upload time of the extracted data is T u , the difficulty of extracting data u is ψ u , extract data u j The completeness is The first weight is η1, the second weight is η2, and the third weight is η3;

[0115] Taking the satisfaction comprehensive score as the analysis purpose, the decision tree algorithm is used for comprehensive analysis to obtain the predicted satisfaction comprehensive score, and the first weight, the second weight, and the third weight are adjusted according to the error between the predicted satisfaction comprehensive score and the actual satisfaction comprehensive score;

[0116] The comprehensive satisfaction score of the extracted data is recalculated based on the adjusted first weight, second weight, and third weight.

[0117] In this embodiment, the method for constructing a data resource quality assessment model according to the abnormality coefficient and the comprehensive satisfaction score includes:

[0118] The quality evaluation function is constructed based on the abnormal coefficient and the comprehensive score of satisfaction. The expression is:

[0119]

[0120] The importance of the sth data is μ s , the anomaly coefficient of the sorted data f is The sth extracted data is u s , extract data u s The overall satisfaction score is The number of research data is N, and the quality assessment function is

[0121] The minimum value of the loss function between the actual value of the quality assessment function and the predicted value is used as the objective function of the data resource quality assessment model. The data resource quality assessment model includes a local outlier factor detection algorithm, a hash algorithm, and a machine learning algorithm.

[0122] The local outlier factor detection algorithm evaluates the density difference between the input data and the neighboring data, identifies outliers based on the density difference, and obtains abnormal data based on the outliers;

[0123] The hash algorithm calculates the hash values ​​of the pattern string and the text string substring of the abnormal data, compares the hash values ​​for potential matches, and compares the potential matches to obtain a matching group;

[0124] The machine learning algorithm automatically identifies and marks potential data quality issues by learning the data patterns of the matching group, and automatically evaluates the data quality issues according to the quality assessment function until the objective function reaches the minimum value, and then outputs the evaluation results.

[0125] In this embodiment, the method for optimizing the data resource quality assessment model according to the assessment error includes:

[0126] The particle population is introduced, and the evaluation error minimization is used as the search strategy to perform chaotic mapping on the particle population. The expression is:

[0127]

[0128] The chaotic mapping sequence of the i+1th particle is U i+1 , the initial position sequence of the i-th particle is U i , the random number from 0 to 1 is r1, the chaotic variation factor is ω, and the control parameter is L;

[0129] The evaluation error is used as the fitness, the particle fitness is calculated, the particle with the smallest fitness is taken as the optimal solution, and the particle position is calculated:

[0130]

[0131] The exploration parameter is p, the random number is r, and the random numbers from 0 to 1 are r2, r3, r4, and r5 respectively. The current position of the zth particle in the tth iteration is The position of the zth particle at the t+1th iteration is The upper limit of the optimization variable is Ub, the lower limit of the optimization variable is Lb, the random number between -1 and 1 is α, the current number of iterations is t, and the maximum number of iterations is t max , the position of the random particle at the tth iteration is The adaptive parameter is l, the chaotic mapping sequence of the tth iteration is U(t), and the competition parameter is h;

[0132] According to the competition mechanism, the particle position is updated to obtain the competition position. The expression is:

[0133]

[0134] The competitive position of the zth particle in the t+1th iteration is The optimal solution is The competition operator is λ and the number of sequences is The initial position sequence of the i-th particle in the t-th iteration is U i (t);

[0135] Use the optimal solution to update particles, the expression is:

[0136]

[0137] The updated i+1th sequence is The chaotic mapping sequence of the tth iteration is U(t), and the random books from 0 to 1 are r6 and r7 respectively;

[0138] Update the particle's position according to the proportional factor to obtain the proportional position. The expression is:

[0139]

[0140] The proportional position of the zth particle in the t+1th iteration is The scaling factor is τ, and the competitive position of the zth particle in the tth iteration is

[0141] Continue iterating until the maximum number of carrying times is reached, otherwise update the particles.

[0142] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 2 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0143] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0144] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0145] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data resource quality assessment device at the logical level. The processor executes the program stored in the memory and is specifically used to execute any of the above-mentioned data resource quality assessment methods.

[0146] The above application Figure 1 A data resource quality assessment method disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0147] The electronic device may also perform Figure 1 A data resource quality assessment method is proposed and implemented Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0148] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple applications, any one of the aforementioned data resource quality assessment methods is executed.

[0149] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0154] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0155] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0156] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data resource quality assessment method, characterized in that: The following steps are involved: Collecting data resources and extracted data from different sources, and preprocessing the data resources and the extracted data; the data resources include standard data and non-standard data; Performing difference matching on the non-standard data to obtain difference data, and performing comparative analysis on the difference data and the standard data to obtain anomaly coefficients; including: Input non-standard data into the matching model, randomly segment the non-standard data, and calculate the similarity between the non-standard data: The similarity between non-standard data w and non-standard data y is γ(w,y), and the i-th non-standard data vector is w i , non-standard data vector w i The neighborhood of Q(w i ), the non-standard data vector at the i-th position is y i , non-standard data vector y i The neighborhood of Q(y i ), the i-th influence multiplier is ζ i , the control constant is x, and the number of segments of non-standard data is n; The non-standard data with the largest similarity are grouped into one group to obtain a regular group, and the non-standard data with a similarity less than 0.352 are grouped as a special group to obtain a grouping; Compute the degree of match between non-standard data according to the regular group: The non-standard data w of the regular group z is z w , the non-standard data y of the conventional group z is z y , non-standard data z w and non-standard data y The matching degree is The number of non-standard data in the regular group z is n w , non-standard data z w The true probability distribution of w ), non-standard data z w The estimated probability distribution of w ); Calculate the difference between the non-standard data and the regular group: β(z w ,With y )=1-l(z w ,With y ) The non-standard data z w and non-standard data y The difference is β(z w ,z y ), the associated difference degree takes the difference part of the conventional group non-standard data as the first difference degree data; Matching the special group with the standard data, calculating the difference between the standard data and the non-standard data, and associating the difference with the difference part of the special group and the non-standard data as the second difference data; Combining the first difference data and the second difference data to output as difference data; Performing a comprehensive satisfaction analysis on the extracted data to obtain a comprehensive satisfaction score, and constructing a data resource quality assessment model based on the abnormality coefficient and the comprehensive satisfaction score; The data resource quality assessment model is optimized according to the assessment error, the data to be assessed is input into the data resource quality assessment model, and the assessment result is output.

2. A data resource quality assessment method according to claim 1, characterized in that: The method of performing comparative analysis on the difference data and the standard data to obtain an abnormality coefficient includes: The difference data and the standard data are input into the comparative analysis model, the standard data is used as the comparison library, and the similarity between the first difference data and the standard data is calculated: The kernel width is φ, the control factor is σ, and the jth first difference data is f j , the jth standard data is a j , the first difference data f j and standard data a j The Euclidean distance of j ,f j ), the first difference data f j and standard data a j The similarity is The standard data with the largest similarity is used as the first control group of the first difference data, and the data with a similarity less than 0.312 is selected as the ambiguous data; Calculate the matching degree between ambiguous data and standard data: The first difference data f j and ambiguous data q j The matching degree is The weight coefficient of Euclidean distance is χ, and the weight coefficient of similarity is χ o , the cosine function is cos(·), and the optimization parameter is υ; The standard data with the highest matching degree is used as the second reference data for the ambiguous data. The first control group and the second control data are segmented, and the segmented first control group and the second control data are sorted according to the number of abnormal digits to obtain sorted data, and the abnormal coefficient of the sorted data is calculated: The number of segments of sorted data is n1, and the number of abnormal bits of sorted data is n am , the quality correlation of the jth sorted data is ε j , the proportional weight is The outlier weight is The importance of sorted data is θ j , the anomaly coefficient of the sorted data f is 3. A data resource quality assessment method according to claim 1, characterized in that: The method of performing a comprehensive satisfaction analysis on the extracted data to obtain a comprehensive satisfaction score includes: Calculate the overall satisfaction score for the extracted data: The comprehensive satisfaction score of the extracted data u is D u , the jth extracted data is u j , the jth standard data is a j , extract data u j and standard data a j The similarity is The upload time of standard data is T o , the upload time of the extracted data is T u , the difficulty of extracting data u is ψ u , extract data u j The completeness is The first weight is η1, the second weight is η2, and the third weight is η3; Taking the comprehensive satisfaction score as the analysis purpose, the decision tree algorithm is used for comprehensive analysis to obtain the predicted comprehensive satisfaction score, and the first weight, the second weight, and the third weight are adjusted according to the error between the predicted comprehensive satisfaction score and the actual comprehensive satisfaction score; The comprehensive satisfaction score of the extracted data is recalculated based on the adjusted first weight, second weight, and third weight.

4. A data resource quality assessment method according to claim 1, characterized in that: The method for constructing a data resource quality assessment model according to the abnormality coefficient and the comprehensive satisfaction score includes: The quality evaluation function is constructed based on the abnormal coefficient and the comprehensive score of satisfaction. The expression is: The importance of the sth data is μ s , the anomaly coefficient of the sorted data f is The sth extracted data is u s , extract data u s The overall satisfaction score is The number of research data is N, and the quality assessment function is The minimum value of the loss function between the actual value of the quality assessment function and the predicted value is used as the objective function of the data resource quality assessment model. The data resource quality assessment model includes a local outlier factor detection algorithm, a hash algorithm, and a machine learning algorithm. The local outlier factor detection algorithm evaluates the density difference between the input data and the neighboring data, identifies outliers based on the density difference, and obtains abnormal data based on the outliers; The hash algorithm calculates the hash values ​​of the pattern string and the text string substring of the abnormal data, compares the hash values ​​for potential matches, and compares the potential matches to obtain a matching group; The machine learning algorithm automatically identifies and marks potential data quality issues by learning the data patterns of the matching group, and automatically evaluates the data quality issues according to the quality assessment function until the objective function reaches the minimum value, and then outputs the evaluation results.

5. A data resource quality assessment method according to claim 1, characterized in that: The method for optimizing the data resource quality assessment model according to the assessment error includes: The particle population is introduced, and the evaluation error minimization is used as the search strategy to perform chaotic mapping on the particle population. The expression is: The chaotic mapping sequence of the i+1th particle is U i+1 , the initial position sequence of the i-th particle is U i , the random number from 0 to 1 is r1, the chaotic variation factor is ω, and the control parameter is L; The evaluation error is used as the fitness, the particle fitness is calculated, the particle with the smallest fitness is taken as the optimal solution, and the particle position is calculated: The exploration parameter is p, the random number is r, and the random numbers from 0 to 1 are r2, r3, r4, and r5 respectively. The current position of the zth particle in the tth iteration is The position of the zth particle at the t+1th iteration is The upper limit of the optimization variable is Ub, the lower limit of the optimization variable is Lb, the random number between -1 and 1 is α, the current number of iterations is t, and the maximum number of iterations is t max , the position of the random particle at the tth iteration is The adaptive parameter is l, the chaotic mapping sequence of the tth iteration is U(t), and the competition parameter is According to the competition mechanism, the particle position is updated to obtain the competition position. The expression is: The competitive position of the zth particle in the t+1th iteration is The optimal solution is The competition operator is λ, the number of sequences is m, and the initial position sequence of the i-th particle in the t-th iteration is U i (t); Use the optimal solution to update particles, the expression is: The updated i+1th sequence is The chaotic mapping sequence of the tth iteration is U)t), and the random books from 0 to 1 are r6 and r7 respectively; Update the particle's position according to the proportional factor to obtain the proportional position. The expression is: The proportional position of the zth particle in the t+1th iteration is The scaling factor is τ, and the competitive position of the zth particle in the tth iteration is Continue iterating until the maximum number of carrying times is reached, otherwise update the particles.

6. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method according to any one of claims 1 to 5.

7. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute the method according to any one of claims 1 to 5.

Images