Data quality scoring method, apparatus, storage medium, and electronic device

Abstract

The application discloses a data quality scoring method and device, a storage medium and an electronic device, and relates to the technical field of smart homes. The data quality scoring method comprises the following steps: acquiring a text list associated with a data warehouse; determining the weight relationship between different texts in the text list, wherein the text list comprises a plurality of target texts; establishing an association relationship between the text list and a data table stored in the data warehouse according to a preset corresponding relationship between the data table and the text; and in the case that a health degree score corresponding to the data table is calculated using a preset target evaluation process, constructing a target graph network based on the text list, the weight relationship, the data table and the health degree score corresponding to the data table, so as to score the data quality of the to-be-evaluated text selected by a target object through the target graph network. The above technical solution solves the technical problem that the data quality health degree of part of the data in a big data set under different dimensions cannot be quickly determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart home, in particular to a data quality scoring method and device, a storage medium and an electronic device. BACKGROUND

[0002] At present, data as an important resource has important value for different individuals. In the data warehouse construction process, data quality determines the value of data, and only high-quality data can provide reliable and accurate information. In order to measure data quality, the index of "data quality health degree" is proposed. Related objects evaluate data quality by direct observation, social investigation, theoretical derivation and other ways. Generally speaking, it mainly covers data accuracy, data integrity, data consistency, data timeliness, entity unity, data reliability, data credibility and other key indicators. At present, research institutions, enterprises and other organizations have proposed their own measurement methods, which are not unified at present. In addition, in actual application, the technical scheme can only evaluate the data quality of the determined data set in a preset manner, and cannot evaluate the data quality health degree of part of the data in the data lake under different dimensions.

[0003] Therefore, in view of the technical problem in the related art that the data quality health degree of part of the data in the big data set under different dimensions cannot be quickly determined, an effective solution has not been proposed. SUMMARY

[0004] The embodiments of the present application provide a data quality scoring method, device, storage medium and electronic device to at least solve the technical problem in the related art that the data quality health degree of part of the data in the big data set under different dimensions cannot be quickly determined.

[0005] According to one of the embodiments of the present application, a data quality scoring method is provided, including: obtaining a text list associated with a data warehouse, and determining the weight relationship between different texts in the text list, wherein the text list includes a plurality of target texts; establishing an association relationship between the text list and a data table stored in the data warehouse according to a preset correspondence relationship between the data table and the text; in the case that a health degree score corresponding to the data table is calculated using a preset target evaluation process, constructing a target graph network based on the text list, the weight relationship, the data table and the health degree score corresponding to the data table, so as to score the data quality of the to-be-evaluated text selected by the target object through the target graph network.

[0006] In an example embodiment, determining the weight relationship between different texts in the text list comprises: scoring each target text included in the text list using a target scoring method to obtain a plurality of scoring results; and determining the weight relationship between different texts in the text list based on the plurality of scoring results.

[0007] In an example embodiment, the health score corresponding to the data table is calculated using a preset target evaluation process, comprising: obtaining N data tables to be evaluated in the data warehouse, and identifying M data columns in each of the N data tables; scoring the fields in the M data columns using a preset field evaluation standard to obtain a first score corresponding to each data column; and adding the M first scores corresponding to the M data columns in the data table to obtain a health score corresponding to the data table, wherein M and N are positive integers.

[0008] In an example embodiment, a target graph network is constructed based on the text list, the weight relationship, the data table, and the health score corresponding to the data table, comprising: determining a plurality of nodes corresponding to the graph text list in a preset graph structure; wherein each node corresponds to a data table set; determining edges between the plurality of nodes according to the weight relationship; taking the data table and the health score corresponding to the data table as a node attribute corresponding to each node in the plurality of nodes; and combining the plurality of nodes, the edges, and the node attributes in the preset graph structure to construct a target graph network.

[0009] In an example embodiment, a data quality score of a target object selected to be evaluated text is determined through the target graph network, comprising: determining data content included in the to-be-evaluated text; finding a neighborhood including the data content in the graph network based on the data content; determining a plurality of target nodes included in the neighborhood and obtaining a health score corresponding to each target node to obtain a plurality of health scores; and performing a summation operation on the plurality of health scores to determine a result of a data quality score corresponding to the to-be-evaluated text.

[0010] In an example embodiment, before determining the text list associated with the data warehouse, the above method further comprises: determining an application scenario corresponding to the data stored in the data warehouse; determining an information type corresponding to the text list, wherein the information type comprises at least one of the following: text information corresponding to a business entity, text information corresponding to a process description, and text information corresponding to an attribute description; determining a configuration relationship between the information type and the application scenario according to a preset corresponding rule to obtain matching information between data and target text.

[0011] In an example embodiment, after the target object selected to-be-evaluated text is scored for data quality by the target graph network, the method further comprises: obtaining a historical quality score corresponding to the to-be-evaluated text; in a case where the historical quality score is greater than a target quality score corresponding to the data quality score, determining that the availability of the target data corresponding to the to-be-evaluated text is reduced; and in a case where the historical quality score is less than or equal to the target quality score corresponding to the data quality score, determining that the availability of the target data corresponding to the to-be-evaluated text is improved.

[0012] According to another embodiment of the embodiments of the present application, a data quality scoring device is also provided, comprising: an obtaining module configured to obtain a text list associated with a data warehouse and determine a weight relationship between different texts in the text list, wherein the text list comprises a plurality of target texts; an establishing module configured to establish an association relationship between the text list and a data table stored in the data warehouse according to a preset correspondence relationship between the data table and the text; and a constructing module configured to, in a case where a health degree score corresponding to the data table is calculated using a preset target evaluation process, construct a target graph network based on the text list, the weight relationship, the data table, and the health degree score corresponding to the data table, so as to score data quality of a to-be-evaluated text selected by a target object through the target graph network.

[0013] According to still another aspect of the embodiments of the present application, a computer readable storage medium having a computer program stored therein is also provided, wherein the computer program is configured to execute the above data quality scoring method when running.

[0014] According to still another aspect of the embodiments of the present application, an electronic device is also provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the above data quality scoring method through the computer program.

[0015] In this embodiment, a list of texts related to data already stored in the data warehouse is identified, and a correspondence between the data table and the text list is established. The weight relationships between different texts in the text list are then evaluated. Based on this information, a graph network is constructed using the text list as nodes and the weights between texts as edges. When a target object needs to filter target texts in the text list, the constructed graph network can quickly determine the data quality score of the data corresponding to different texts. This allows for rapid determination of the data quality of some data corresponding to some texts even with an uncertain dataset. This solves the technical problem in related technologies where it is impossible to quickly determine the data quality health of some data in large datasets across different dimensions. It enables the measurement of data quality health within a non-deterministic data range, better meeting the data quality assessment needs of different usage scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the hardware environment for a data quality scoring method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a data quality scoring method according to an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating the data quality scoring method according to an embodiment of this application;

[0021] Figure 4 This is a structural block diagram (a) of a data quality scoring device according to an embodiment of this application;

[0022] Figure 5 This is a structural block diagram (II) of a data quality scoring device according to an embodiment of this application. Detailed Implementation

[0023] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an aspect of the embodiments of the present application, a data quality scoring method is provided. The data quality scoring method is widely used in smart home, smart home, smart home device ecosystem, intelligence house ecosystem and other whole-house intelligent digital control application scenarios. Optionally, in the present embodiment, the above-mentioned data quality scoring method can be applied to the hardware environment composed of terminal device 102 and server 104 as shown in Figure 1 As shown in Figure 1 The server 104 is connected with the terminal device 102 through the network, which can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal, a database can be set on the server or independently of the server, which can be used to provide data storage services for the server 104, cloud computing and / or edge computing services can be configured on the server or independently of the server, which can be used to provide data operation services for the server 104.

[0026] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0027] This embodiment provides a data quality scoring method, which is applied to the aforementioned terminal device or server. Figure 2 This is a flowchart of a data quality scoring method according to an embodiment of this application, which includes the following steps:

[0028] Step S202: Obtain a list of texts associated with the data warehouse and determine the weight relationship between different texts in the list of texts, wherein the list of texts includes: multiple target texts;

[0029] It is understandable that the above text list is a collection of texts formed by the business object based on the data type of the data existing in the data warehouse, assuming that the corresponding data resources have already been stored in the data warehouse. This text list covers business entities, process descriptions, attribute descriptions, etc.

[0030] Step S204: Establish the association between the text list and the data tables stored in the data warehouse according to the preset correspondence between data tables and text;

[0031] Understandably, once the predefined correspondence between text and data tables is obtained, the predefined correspondence between data tables and text can be determined based on this correspondence, thereby establishing the association between the text list and the data tables. It should be noted that a data table can be associated with multiple texts, and a text can also be associated with multiple data tables.

[0032] Step S206, in the case where the health score corresponding to the data table is calculated using a preset target evaluation process, a target graph network is constructed based on the text list, the weight relationship, the data table, and the health score corresponding to the data table, so as to perform data quality scoring on the to-be-evaluated text selected by the target object through the target graph network.

[0033] Through the above steps, by determining the text list having a relationship with the data stored in the data warehouse, and then establishing the correspondence between the data table and the text list, the weight relationship between different texts in the text list is re-evaluated, and the graph network is constructed based on the determined information, taking the text list as the node of the graph network and taking the weight between the texts as the edge. When the target object needs to screen the target text in the text list, the data quality score of the data corresponding to different texts can be quickly determined through the constructed graph network, and then the data quality of part of the data corresponding to part of the texts can be quickly determined without determining the data set, thereby solving the technical problem that the data quality health degree of part of the data in the large data set under different dimensions cannot be quickly determined in the related art, measuring the data quality health degree of the non-deterministic data range, and better meeting the data quality evaluation needs in different use scenarios.

[0034] In one example embodiment, for the above step S202, the weight relationship between different texts in the text list can be determined by the following technical solution, and the specific steps include:

[0035] Step S1: using a target scoring method to score the target texts included in the text list one by one to obtain a plurality of scoring results;

[0036] Optionally, the target scoring method can be an expert scoring comparison method, which specifically includes the following contents:

[0037] 1. Determine the weight of the text, that is, determine the weight of each text according to the relative importance of the data that may exist in the different texts, and the sum of the weights is 1.

[0038] 2. Grade, that is, divide the texts into multiple grades, and assign quantitative values to each grade to determine the grade of each index of the current text in the text list.

[0039] 3. Multiply the weight of each target text by the corresponding grade to obtain the score of the target text.

[0040] Step S2: determining the weight relationship between different texts in the text list based on the plurality of scoring results.

[0041] As an optional implementation, the determining of the weight relationship between different texts in the text list based on the plurality of scoring results comprises:

[0042] Step S2-1: selecting the scoring results between any two texts to multiply, to obtain a target weight relationship between the any two texts;

[0043] Step S2-2: determining the weight relationship between different texts in the text list through the target weight relationship.

[0044] In summary, through the above steps, the plurality of target texts in the text list are associated, and the correlation between different texts is used to determine the correlation level relationship between texts, so that the plurality of target texts in the text list can be associated, and the search efficiency of searching for the same text set according to the content corresponding to the text is improved.

[0045] In an example embodiment, the health score corresponding to the data table is calculated using a preset target evaluation process, comprising: obtaining N data tables to be evaluated in the data warehouse, identifying M data columns in each of the N data tables; scoring the fields in the M data columns using a preset field evaluation standard to obtain a first score corresponding to each data column; adding the M first scores corresponding to the M data columns in the data table to obtain a health score corresponding to the data table, wherein M and N are positive integers.

[0046] That is, according to the hierarchy, on the basis of determining the data warehouse, the data existing in the data warehouse is decomposed and arranged through the relationship from the data warehouse to the database, from the database to the data table, and from the data table to the data column, and then the fields of the data table are evaluated according to the preset single field data quality evaluation standard, and the weighted average method is used to evaluate the data quality of each data table. Thus, on the basis of determining the health score of part of the data columns, the health score of the corresponding data table can be determined by combination, and the health score of the database is determined by combining a plurality of data tables. When the health score of part of the data is determined, the health score of the corresponding part of the data can be quickly determined without calculation through content search.

[0047] In an example embodiment, based on the text list, the weight relationship, the data table and the health score corresponding to the data table, a target graph network is constructed, including: determining a plurality of nodes corresponding to the graph text list in a preset graph structure; wherein each node corresponds to a data table set; determining edges between the plurality of nodes according to the weight relationship; taking the data table and the health score corresponding to the data table as a node attribute corresponding to each node in the plurality of nodes; and combining the plurality of nodes, the edges and the node attribute in the preset graph structure to construct the target graph network.

[0048] It can be known that the graph network is a visual data structure composed of nodes and edges combined with corresponding node information, therefore, the text list formed by a plurality of target texts and the weight relationship between different target texts are needed to construct the graph network, the text list is taken as a node, the weight between texts is taken as an edge, and the data table and the data quality health score of the data table are taken as a node attribute, and the graph network is constructed, so that when a certain text needs to be focused on, the data quality health of the corresponding text and the content of the corresponding data table can be quickly determined by searching for the corresponding node in the graph network.

[0049] In an example embodiment, the data quality score of the target object selected by the target object is evaluated by the target graph network, including: determining the data content included in the text to be evaluated; finding the neighborhood including the data content in the graph network based on the data content; determining a plurality of target nodes included in the neighborhood and obtaining a health score corresponding to each target node to obtain a plurality of health scores; and performing a summation operation on the plurality of health scores to determine the result of the data quality score corresponding to the text to be evaluated.

[0050] For example, in the system, the user can select the text w a ∈W.

[0051] For this text w a , through the graph network, its neighborhood is found, and the sub-nodes of the neighborhood are determined. Including the sub-nodes themselves, a new set is formed. For any one sub-node, there is a related data table, which is represented in the form of an attribute in the graph network. The node w x corresponds to a data table set, which is represented as

[0052] The data health of each node w x of V’ is calculated.

[0053] wherein, corresponding to each node in the set V’, corresponding to each node in the set V’.

[0054] The data quality health degree related to the text w a is calculated using the set V’:

[0055] wherein, represents the association weight between any one node in the set V’ and the text w a .

[0056] It should be noted that in the present embodiment, the relationship between texts is determined based on the graph network, the data table related to a specific text can be screened out, and the data quality score of the data table related to the specific text is calculated using the relationship weight between texts; so that the process of data quality score can effectively evaluate the data quality health degree of part of the data in the summarized data under different dimensions, thereby meeting the data quality evaluation needs of various scenarios.

[0057] In one example embodiment, before determining the list of texts associated with the data warehouse, the above method further comprises: determining an application scenario corresponding to the data stored in the data warehouse; determining an information type corresponding to the list of texts, wherein the information type comprises at least one of the following: text information corresponding to a business entity, text information corresponding to a process description, and text information corresponding to an attribute description; determining a configuration relationship between the information type and the application scenario according to a preset corresponding rule to obtain matching information between the data and the target text.

[0058] In short, in determining the list of texts, it is also necessary to determine the simple contact between the data and the text, i.e. a certain kind of data will appear or be used in those texts, so that the list of texts can be more generalized when determining the use scenario of the data.

[0059] In one example embodiment, after the target graph network performs data quality score on the to-be-evaluated text selected by the target object, the above method further comprises: obtaining a historical quality score corresponding to the to-be-evaluated text; in the case that the historical quality score is greater than a target quality score corresponding to the data quality score, determining that the availability of the target data corresponding to the to-be-evaluated text is reduced; in the case that the historical quality score is less than or equal to the target quality score corresponding to the data quality score, determining that the availability of the target data corresponding to the to-be-evaluated text is improved.

[0060] That is, when multiple evaluations are performed on the text, the change trend of the data corresponding to the to-be-evaluated text can be determined according to the size relationship between the historical quality score and the target quality score corresponding to the to-be-evaluated text, so as to reduce the evaluation of the to-be-evaluated text corresponding to the data with low availability, and save data resources.

[0061] In order to better understand the process of the above data quality scoring method, the implementation method flow of the above data quality scoring will be described in combination with optional embodiments below, but not used to limit the technical solutions of the embodiments of the present application.

[0062] In the present embodiment, a data quality scoring method is provided, Figure 3 is a schematic diagram of the data quality scoring method according to the embodiments of the present application, as Figure 3 shown, mainly includes the following steps:

[0063] Step one, for the fields of the data table, the preset single field data quality evaluation standard is used to evaluate the score, and the weighted average method is used to evaluate the data quality of each data table;

[0064] Step two, based on business analysis, a text list of business entities, process description and attribute description is formed, and the association relationship between the data table and the above text list is established.

[0065] Step three, the expert scoring method is used to form the weight relationship of the above text list;

[0066] Step four, taking the text list as the node of the graph network and the weight between the texts as the edge, the graph network is constructed;

[0067] Step five, for different text screening, the corresponding data quality score is calculated based on the graph network.

[0068] Specifically, the implementation process is as follows:

[0069] For step one: the structured database is divided into layers, including database Schema (S for short), data table Table (T for short) and data column Column (C for short).

[0070] In the data warehouse, there are i database Schemas, which constitute a database set Φ={S1, S2, S3, S4……Si}. i}。

[0071] Each Schema corresponds to a data table Table, and for a database S i with m i data tables, the data table set can be expressed as: For the data warehouse, it contains databases S1, S2, S3, S4……Si iEach database corresponds to m1, m2, …, m i The data table set of the data warehouse is:

[0072] Each data table Table corresponds to a specific data column Column. For a data table with Column data columns The data table set of the data warehouse is: The data table set of the data warehouse is:

[0073]

[0074] For a data warehouse, the data table set is Ψ, and the number of fields corresponding to each data table is The corresponding field set is:

[0075]

[0076] There is a pre-set data quality health assessment method for data columns in the system.

[0077] Users can configure schemes for data columns according to their needs. Schemes include: format specification, data reliability, data accuracy, data accuracy, etc.

[0078] If a data column The scores obtained by selecting a index are The data quality score of the data column is: The average weight corresponding to a index is represented; using the score of the data column, the data quality health score of the data table is calculated The calculation formula is: The average weight is calculated again based on the number of data columns possessed by the data table.

[0079] Optionally, the above score calculation can use average weight or set weight method for weighting.

[0080] For step two: the business department sorts out the text list related to the data warehouse, covering business entities, process description, attribute description, etc. For example: order, order paper, brand, product, sales, supply chain, etc. Form a text set W = {w1, w2, w3 … w x}. The above text has a clear meaning related to the business. For each data table, establish an association with the above list of entities in the form of weights. For example: data table The association weight of text w x A data table can have an association with multiple texts, and a text can also have an association with multiple data tables.

[0081] For step three: use expert scoring method to score each text in the list, and calculate the weight relationship between texts. For example, the weight between w1 and w2

[0082] For step four: construct a graph network with the text list as nodes, the weight between texts as edges, and the data table and data table data quality health score as node attributes. Optionally, construct a graph network G(V, E): where the text set (equivalent to the text list in the above embodiment) V = {w1, w2, w3 … w x}; the weight relationship between texts and a≠b.

[0083] For step five: data quality assessment is performed on part of the data in the data warehouse with certain characteristics. In the system, users can select the text w a ∈W according to their needs.

[0084] For this text w a , find its neighborhood determine its sub-nodes. Including the sub-nodes themselves, form a new set

[0085] For any sub-node, there is a related data table, which is represented in the form of attributes in the graph network. The node w x corresponds to the data table set, which is represented as Calculate the data health degree of each node w x : Use the set V to calculate the data quality health degree related to the text w a :

[0086] ​By the above steps, the association between texts is established, and the association between the texts and the data tables is also established. Based on the association between the texts, the domain range related to a specific text can be screened out, and then the data table related to the specific text is determined in the text range; by comprehensive weighting, the data quality health degree of the data table related to the specific text is obtained, so as to measure the data quality health degree of the non-deterministic data range. The scheme is different from the existing data quality health degree evaluation method of specifying a specific data range, and can better meet the data quality evaluation demand.

[0087] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method of each embodiment of the present application.

[0088] Figure 4 is a structural block diagram of a data quality scoring device according to an embodiment of the present application (one); as shown in Figure 4 , comprising:

[0089] The acquisition module 42 is configured to acquire a text list associated with a data warehouse, and determine a weight relationship between different texts in the text list, wherein the text list includes a plurality of target texts.

[0090] The establishment module 44 is configured to establish an association relationship between the text list and a data table stored in the data warehouse according to a preset correspondence relationship between the data table and the text.

[0091] The construction module 46 is configured to, in a case where a health degree score corresponding to the data table is calculated by using a preset target evaluation process, construct a target graph network based on the text list, the weight relationship, the data table, and the health degree score corresponding to the data table, so as to perform data quality scoring on a to-be-evaluated text selected by a target object through the target graph network.

[0092] By the above device, by determining the text list having a relationship with the data stored in the data warehouse, and then establishing the corresponding relationship between the data table and the text list, and then evaluating the weight relationship between different texts in the text list, the graph network is constructed based on the determined information, taking the text list as the node of the graph network, and taking the weight between texts as the edge. When the target object needs to filter the target text in the text list, the data quality score of the data corresponding to different texts can be quickly determined through the constructed graph network, and then the data quality of part of the data corresponding to part of the texts can be quickly determined without determining the data set, which solves the technical problem that the data quality health degree of part of the data in the big data set under different dimensions cannot be quickly determined in the related art, measures the data quality health degree of the non-deterministic data range, and better meets the data quality evaluation needs in different use scenarios.

[0093] In one example embodiment, the acquisition module 42 is further configured to score each target text in the text list using a target scoring method to obtain a plurality of scoring results, and determine the weight relationship between different texts in the text list based on the plurality of scoring results.

[0094] Optionally, the target scoring method can be an expert scoring comparison method, which specifically includes the following contents:

[0095] 1. Determine the weight of the text, that is, determine the weight of each text according to the relative importance of the data that may exist in different texts, and the sum of the weights is 1.

[0096] 2. Grade, that is, divide the text into multiple grades, and assign a quantitative value to each grade to determine the grade of the current text in the text list.

[0097] 3. Multiply the weight of each target text by the corresponding grade to obtain the score of the target text.

[0098] In summary, by associating multiple target texts in the text list, the correlation between different texts is used to determine the correlation grade relationship between texts, so that multiple target texts in the text list can be linked, and the search efficiency of searching the same text set according to the corresponding content of the text is improved.

[0099] In an example embodiment, the above-mentioned construction module 46 is further configured to obtain N data tables to be evaluated in the data warehouse, identify M data columns in each of the N data tables; score fields in the M data columns using preset field evaluation criteria to obtain a first score corresponding to each data column; and add the M first scores corresponding to the M data columns in the data table to obtain a health degree score corresponding to the data table, wherein M and N are positive integers.

[0100] That is, according to the hierarchy, on the basis of determining the data warehouse, the data in the data warehouse is decomposed and arranged through the relationships from the data warehouse to the database, from the database to the data table, and from the data table to the data column, and then the fields of the data table are scored according to the preset single field data quality evaluation criteria, and the weighted average method is used to score the data quality of each data table, so that on the basis of determining the health degree scores of part of the data columns, the health degree scores of the corresponding data tables can be determined by combination, and the health degree scores of the databases are determined by combination of multiple data tables, so that when the health degree scores of part of the data are determined, the health degree scores of the corresponding part of the data can be quickly determined without calculation through content searching.

[0101] In an example embodiment, the above-mentioned construction module 46 is further configured to determine a plurality of nodes corresponding to the text list in the preset graph structure; each node corresponds to a data table set; determine the edges between the plurality of nodes according to the weight relationship; the data table and the health degree score corresponding to the data table are used as the node attribute corresponding to each node in the plurality of nodes; and the plurality of nodes, the edges, and the node attribute are combined in the preset graph structure to construct a target graph network.

[0102] It can be known that the graph network is a visual data structure composed of nodes and edges combined with corresponding node information, therefore, the text list formed by multiple target texts and the weight relationship between different target texts are required to construct the graph network, the text list is used as the node, the weight between the texts is used as the edge, and the data table and the data quality health degree score of the data table are used as the node attribute, so as to construct the graph network, so that when a certain text needs to be focused on, the data quality health of the corresponding text and the content of the corresponding data table can be quickly determined by searching for the corresponding node in the graph network.

[0103] In an example embodiment, the construction module 46 described above further comprises: a scoring unit configured to determine data content included in the text to be evaluated; find a neighborhood including the data content in the graph network based on the data content; determine a plurality of target nodes included in the neighborhood, and obtain a health score corresponding to each target node to obtain a plurality of health scores; and perform a summation operation on the plurality of health scores to determine the result of the data quality score corresponding to the text to be evaluated.

[0104] For example, in the system, a user can select a text w a ∈W according to needs.

[0105] For this text w a , find its neighborhood in the graph network, and determine the child nodes of the neighborhood. Including the child nodes themselves, form a new set For any child node, there is a related data table, which is represented in the form of an attribute in the graph network. The node w x corresponding data table set is represented as

[0106] Calculate V, the data health of each node w x :

[0107] wherein, is the correlation weight corresponding to each node in the set V, is the health score corresponding to each node in the set V.

[0108] Using the set V, calculate the data quality health related to the text w a :

[0109] wherein, represents the correlation weight between any node in the set V and the text w a .

[0110] It should be noted that in the present embodiment, the relationship between texts is determined based on the graph network, which can filter out the data tables related to a specific text; and the data quality score of the data table related to the specific text is calculated using the relationship weight between the texts; so that the process of data quality score can effectively evaluate the data quality health of part of the data in the summarized data under different dimensions, thereby meeting the data quality evaluation needs of various scenarios.

[0111] In an example embodiment, the data quality scoring device described above, in addition to the above-mentioned acquisition module 42, the above-mentioned establishment module 44 and the above-mentioned construction module 46, further comprises a matching module 40 and a determination module 48. AsFigure 5 As shown, Figure 5 is a structural block diagram of a data quality scoring device according to an embodiment of the present application (two),

[0112] In one example embodiment, the above device further comprises a matching module 40, configured to determine an application scenario corresponding to data stored in the data warehouse before determining the text list associated with the data warehouse; determine an information type corresponding to the text list, wherein the information type comprises at least one of the following: text information corresponding to a business entity, text information corresponding to a process description, and text information corresponding to an attribute description; determine a configuration relationship between the information type and the application scenario according to a preset corresponding rule to obtain matching information between data and target text.

[0113] In short, in addition to determining the relationship between text lists, it is also necessary to determine the simple relationship between data and text, that is, a certain data will appear or be used in those texts, so that the text list can be more generalized when determining the use scenario of the data.

[0114] In one example embodiment, the above device further comprises a determination module 48, configured to obtain a historical quality score corresponding to the to-be-evaluated text after the target graph network performs data quality scoring on the to-be-evaluated text selected by the target object; in a case where the historical quality score is greater than a target quality score corresponding to the data quality score, determine that the availability of the target data corresponding to the to-be-evaluated text is reduced; in a case where the historical quality score is less than or equal to the target quality score corresponding to the data quality score, determine that the availability of the target data corresponding to the to-be-evaluated text is improved.

[0115] That is, when the text is evaluated multiple times, the change trend of the data corresponding to the to-be-evaluated text can be determined according to the size relationship between the historical quality score and the target quality score corresponding to the to-be-evaluated text, so as to reduce the evaluation of the to-be-evaluated text corresponding to data with low availability and save data resources.

[0116] Embodiments of the present application also provide a storage medium comprising a stored program, wherein the above-mentioned program executes any of the above-mentioned methods when running.

[0117] Optionally, in the present embodiment, the above-mentioned storage medium can be configured to store program code for executing the following steps:

[0118] S1, obtaining a text list associated with a data warehouse, and determining a weight relationship between different texts in the text list, wherein the text list comprises: a plurality of target texts;

[0119] S2, establish an association relationship between the text list and data tables stored in the data warehouse according to a preset data table-text correspondence relationship;

[0120] S3, in a case where a health degree score corresponding to the data table is calculated using a preset target evaluation process, construct a target graph network based on the text list, the weight relationship, the data table, and the health degree score corresponding to the data table, to perform data quality scoring on a to-be-evaluated text selected by a target object through the target graph network.

[0121] Embodiments of the present application also provide an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the above method embodiments.

[0122] Optionally, the electronic device can further include a transmission device and an input-output device, wherein the transmission device is connected to the processor, and the input-output device is connected to the processor.

[0123] Optionally, in the present embodiment, the processor can be configured to execute the following steps through the computer program:

[0124] S1, acquire a text list associated with a data warehouse, and determine a weight relationship between different texts in the text list, wherein the text list includes a plurality of target texts;

[0125] S2, establish an association relationship between the text list and data tables stored in the data warehouse according to a preset data table-text correspondence relationship;

[0126] S3, in a case where a health degree score corresponding to the data table is calculated using a preset target evaluation process, construct a target graph network based on the text list, the weight relationship, the data table, and the health degree score corresponding to the data table, to perform data quality scoring on a to-be-evaluated text selected by a target object through the target graph network.

[0127] Optionally, in the present embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0128] Optionally, specific examples in the present embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the present embodiment will not be described here again.

[0129] It is apparent that those skilled in the art can, without departing from the spirit of the present application, make various changes and modifications of the modules or steps of the present application described above, which can be implemented by general computing devices, and can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, can be implemented by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or can be made into individual integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0130] The above description is only the preferred embodiments of the present application, and it should be pointed out that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A data quality scoring method, characterized in that, include: Obtain a list of texts associated with the data warehouse, and determine the weight relationships between different texts in the list of texts, wherein the list of texts includes: multiple target texts; Establish the association between the text list and the data tables stored in the data warehouse based on the preset correspondence between data tables and text; When the health score corresponding to the data table is calculated using a preset target evaluation process, a target graph network is constructed based on the text list, the weight relationship, the data table, and the health score corresponding to the data table, so as to score the data quality of the text to be evaluated selected by the target object through the target graph network; The step of calculating the health score corresponding to the data table using a preset target evaluation process includes: obtaining N data tables to be evaluated in the data warehouse, identifying M data columns in each of the N data tables; scoring the fields in the M data columns using a preset field evaluation standard to obtain a first score corresponding to each data column; and adding the M first scores corresponding to the M data columns in the data table to obtain the health score corresponding to the data table, where M and N are positive integers. The step of constructing a target graph network based on the text list, the weight relationship, the data table, and the health score corresponding to the data table includes: determining multiple nodes corresponding to the text list in a preset graph structure; wherein each node corresponds to a data table set; determining the edges between the multiple nodes according to the weight relationship; using the data table and the health score corresponding to the data table as node attributes for each of the multiple nodes; and combining the multiple nodes, the edges, and the node attributes in the preset graph structure to construct the target graph network.

2. The data quality scoring method according to claim 1, characterized in that, Determining the weight relationships between different texts in the text list includes: The target scoring method is used to score each of the target texts in the text list, resulting in multiple scoring results. The weight relationships between different texts in the text list are determined based on the multiple scoring results.

3. The data quality scoring method according to claim 1, characterized in that, The target graph network is used to score the data quality of the selected text to be evaluated, including: Determine the data content included in the text to be evaluated; Based on the data content, a neighborhood containing the data content is searched in the graph network; Identify multiple target nodes included in the neighborhood and obtain the health score corresponding to each target node to obtain multiple health scores; A summation operation is performed on the multiple health scores to determine the data quality score corresponding to the text to be evaluated.

4. The data quality scoring method according to claim 1, characterized in that, Before determining the list of texts associated with the data warehouse, the method further includes: Determine the application scenarios corresponding to the data stored in the data warehouse; Determine the information type corresponding to the text list, wherein the information type includes at least one of the following: text information corresponding to a business entity, text information corresponding to a process description, and text information corresponding to an attribute description; The configuration relationship between the information type and the application scenario is determined according to the preset corresponding rules, and the matching information between the data and the target text is obtained.

5. The data quality scoring method according to claim 1, characterized in that, After scoring the data quality of the selected text to be evaluated using the target graph network, the method further includes: Obtain the historical quality score corresponding to the text to be evaluated; If the historical quality score is greater than the target quality score corresponding to the data quality score, it is determined that the usability of the target data corresponding to the text to be evaluated has decreased. If the historical quality score is less than or equal to the target quality score corresponding to the data quality score, the usability improvement of the target data corresponding to the text to be evaluated is determined.

6. A data quality scoring device, characterized in that, include: The acquisition module is used to acquire a list of texts associated with the data warehouse and determine the weight relationship between different texts in the list of texts, wherein the list of texts includes: multiple target texts; A module is established to establish the association between the text list and the data tables stored in the data warehouse based on a preset correspondence between data tables and text. The construction module is used to construct a target graph network based on the text list, the weight relationship, the data table, and the health score corresponding to the data table when the health score corresponding to the data table is calculated using a preset target evaluation process, so as to score the data quality of the text to be evaluated selected by the target object through the target graph network; The construction module is further configured to acquire N data tables to be evaluated in the data warehouse, identify M data columns in each of the N data tables; score the fields in the M data columns using a preset field evaluation standard to obtain a first score corresponding to each data column; and add the M first scores corresponding to the M data columns in the data table to obtain the health score corresponding to the data table, where M and N are positive integers. The construction module is further configured to determine multiple nodes corresponding to the text list in a preset graph structure; wherein each node corresponds to a set of data tables; determine the edges between the multiple nodes according to the weight relationship; use the data tables and the health scores corresponding to the data tables as node attributes corresponding to each of the multiple nodes; and combine the multiple nodes, the edges, and the node attributes in the preset graph structure to construct a target graph network.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the data quality scoring method as described in any one of claims 1 to 5.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the data quality scoring method according to any one of claims 1 to 5 through the computer program.

Images