Data acquisition method and device, equipment and storage medium
By establishing a tag tree in multiple databases and calculating the confidence of the data labels, the target data required by the task is automatically obtained, and the problem of inconsistency of multi-source data labels is solved, efficient and accurate data acquisition is achieved, and labor costs are saved.
Patent Information
- Application Number
- CN202411085909.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-08
AI Technical Summary
In the era of big data, multi-source data is difficult to automatically obtain the target data required by the task due to label inconsistency, redundancy or conflict. The existing technology relies on manual acquisition, which consumes manpower and affects efficiency.
By establishing a tag tree in multiple databases, determining the confidence calculation method of the data label, and determining the target data label based on the confidence, the target data required by the task is automatically obtained.
The automated data acquisition process is realized, saving labor costs, improving the efficiency and accuracy of data acquisition, and reducing wrong decisions and additional work caused by data errors or inconsistencies.
Smart Images

Figure CN118916384A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data acquisition method, device, equipment and storage medium. Background Art
[0002] In the era of big data, the diversity, complexity and scale of data have brought unprecedented challenges to data integration and analysis. Multi-source data, that is, data from different sources, is widely present in many fields such as social media, e-commerce, Internet of Things, and healthcare. These data often have their own unique labels to describe the content, attributes or classification of the data. Due to the diversity of data sources, data labels for the same task may be inconsistent, redundant or conflicting, making it difficult to obtain the target data required for the task from multi-source data based on labels.
[0003] In the prior art, technicians are usually required to manually obtain the target data required for the task from multiple source data. Although this method meets business needs to a certain extent, it is very labor-intensive and affects the processing efficiency of the task.
[0004] Therefore, it is urgent to propose a new method to solve the above problems. Summary of the invention
[0005] The present invention provides a data acquisition method, device, equipment and storage medium, which can automatically acquire target data required for a task in multiple databases, thereby saving labor costs.
[0006] In a first aspect, an embodiment of the present invention provides a data acquisition method, comprising:
[0007] Acquire task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data, wherein the task data includes data tags, and the tag tree is composed of multiple tag levels, and each of the tag levels includes at least one data tag;
[0008] Determine a confidence calculation method for each of the data tags, calculate the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each of the data tags in each of the tag levels;
[0009] The target task data is determined in a database corresponding to each of the target data tags according to each of the target data tags.
[0010] The embodiment of the present invention provides a data acquisition method, including: acquiring task data from multiple databases corresponding to the current task, establishing a label tree based on the task data, wherein the task data includes data labels, the label tree is composed of multiple label levels, and each label level includes at least one data label; determining the confidence calculation method of each data label, calculating the confidence of each data label based on the label level to which each data label belongs in the label tree and the confidence calculation method, and determining the target data label according to the confidence of each data label in each label level; determining the target task data in the database corresponding to each target data label according to each target data label. The above technical scheme first acquires task data from multiple databases corresponding to the current task, and establishes a label tree based on the task data, thereby realizing the establishment of a label tree for task data. Then, by determining the confidence calculation method of each data label, the accuracy of the calculated confidence of each data label is improved, and then the target data label is determined by the confidence of each data label, and finally, the target task data is determined in the database corresponding to each target data label according to each target data label, thereby realizing the automatic acquisition of the target data required for the task in multiple databases, thereby saving labor costs. Compared with the prior art in which technicians manually obtain the target data required for the task from multi-source data, the present invention establishes a label tree for the task data, calculates the confidence of each data label based on the label tree of the task data and a determined confidence calculation method, determines the target data label according to the confidence of each data label, and finally determines the target task data according to each target data label. This can automatically obtain the target data required for the task from multiple databases, saving labor costs.
[0011] In a second aspect, an embodiment of the present invention further provides a data acquisition device, the device comprising:
[0012] An acquisition module, used to acquire task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data, wherein the task data includes data tags, and the tag tree is composed of multiple tag levels, each of which includes at least one data tag;
[0013] A first determination module is used to determine a confidence calculation method for each of the data tags, calculate the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each of the data tags in each of the tag levels;
[0014] The second determination module is used to determine the target task data in the database corresponding to each target data tag according to each target data tag.
[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0016] at least one processor; and a memory communicatively coupled to the at least one processor;
[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can implement any data acquisition method described in the first aspect.
[0018] In a fourth aspect, an embodiment of the present invention further provides a storage medium including computer executable instructions,
[0019] The computer executable instructions implement any data acquisition method described in the first aspect when executed by a computer processor.
[0020] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the data acquisition device, or may be packaged separately from the processor of the data acquisition device, which is not limited in this application.
[0021] The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third and fourth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.
[0022] In this application, the name of the above data acquisition device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.
[0023] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1 A flow chart of a data acquisition method provided by an embodiment of the present invention;
[0026] Figure 2 A flowchart of another data acquisition method provided by an embodiment of the present invention;
[0027] Figure 3 A schematic diagram of the structure of a data acquisition device provided by an embodiment of the present invention;
[0028] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0030] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0031] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.
[0032] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0033] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.
[0034] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0035] In the description of the present application, unless otherwise specified, “plurality” means two or more.
[0036] In the current data-driven era, data quality in data-based tasks has become a key factor affecting the performance of the task. Due to the uneven data quality in each data source, only part of the data can meet high-quality standards, which directly affects the performance of the task. In order to solve this problem, it is inevitable to select the best data from each data source as the target data required for the task. However, different data sources often use different labeling systems, classification standards or data models, resulting in inconsistency, redundancy or conflict in the data labels of the same task between different data sources, making it extremely difficult to obtain the target data required for the task from multi-source data based on labels. Although technicians can manually obtain the target data required for the task from multi-source data. However, this method is not only labor-intensive, but also affects the processing efficiency of the task. Therefore, it is urgent to propose a new method to solve the above problems.
[0037] Figure 1 This is a flow chart of a data acquisition method provided by an embodiment of the present invention. This embodiment can be applied to the case where required task data is automatically selected from multiple databases according to task requirements. The method can be executed by a data acquisition device and specifically includes the following steps:
[0038] Step 110: Obtain task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data.
[0039] Among them, the current task refers to a task determined according to actual conditions or business needs, for example: the current task can be a classification task, a prediction task or a recommendation task, etc. Task data refers to all relevant data used when performing the current task, and the task data can be in various forms such as text, numbers, images, etc. In this embodiment, the task data is a data set containing data labels, and the data label refers to a short identifier for classifying or describing data (such as documents, images, etc.). The label tree is a hierarchical data structure composed of multiple label levels, which is used to organize and represent the hierarchical relationship between data labels, and each label level includes at least one data label. In the label tree, each node represents a label, and the connection between nodes represents the hierarchical relationship between labels.
[0040] Specifically, the database related to the current task can be determined first, and then the data related to the current task can be obtained from the database related to the current task, and the obtained data can be determined as the task data. Then, according to the actual situation and application requirements, a suitable hierarchical relationship determination method can be selected to determine the hierarchical relationship between the data labels contained in the task data, and then a label tree can be constructed according to the determined hierarchical relationship. For example, the hierarchical relationship determination method can be a hierarchical clustering method, a k-means clustering algorithm, or a density-based clustering algorithm.
[0041] For example, the current task may be to classify products on an e-commerce platform, and the databases corresponding to the task may be a product catalog database, a user review database, and a supplier information database. Data related to the current task, namely, task data, may then be obtained from the product catalog database, the user review database, and the supplier information database. A hierarchical clustering algorithm may then be used to calculate the similarity between each data label contained in the task data, and the hierarchical relationship between the data labels may be determined based on the similarity between each data label, and finally a label tree may be constructed based on the determined hierarchical relationship.
[0042] In this embodiment, the establishment of a label tree for task data is implemented.
[0043] Step 120, determine the confidence calculation method for each data tag, calculate the confidence of each data tag based on the tag level to which each data tag belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each data tag in each tag level.
[0044] Among them, the confidence calculation method refers to the method used to quantify the reliability of the data label in the current task. For example, the confidence calculation method can be a machine learning method, a graph-based fusion method, or a weighted average method. Confidence refers to a numerical value that indicates the reliability of the data label in the current task. For example, the confidence value range can be 0 to 1. The target data label refers to the data label determined based on the confidence of each data label in each label level.
[0045] Specifically, a suitable confidence calculation method can be selected according to actual conditions and application requirements to determine the confidence calculation method. For example, the confidence calculation method can be a determination method based on the amount of data, a determination method based on the attributes of the data label, or a determination method based on the type of task data. The confidence of each data label is then calculated based on the label level to which each data label belongs in the label tree and the selected confidence calculation method, and finally the target data label is determined based on the confidence of each data label in each label level. Specifically, the selected confidence calculation method can be applied to calculate the confidence of each data label in each label level, and then the data label with the highest confidence in each label level is selected as the target data label in each label level.
[0046] Exemplarily, if the confidence calculation method is a method based on the amount of data, and the optional confidence calculation methods include weighted average method and machine learning method, the specific process of using the method based on the amount of data to determine whether the confidence calculation method is the weighted average method or the machine learning method can be: first calculate the amount of task data to obtain the task data amount; then compare the task data amount with the pre-set data amount threshold. If the task data amount exceeds the data amount threshold, the confidence calculation method is determined to be the machine learning method; otherwise, the confidence calculation method is the weighted average method. Among them, the setting of the data amount threshold can be determined according to the specific application scenario and requirements. For example: the databases corresponding to the current task are database A and database B, the amount of data obtained from database A is a, and the amount of data obtained from database B is b, then the task data amount can be determined to be a+b.
[0047] Exemplarily, when the confidence calculation method determined is the weighted average method, the data volume, database weight and label weight of the task data corresponding to each data label are obtained, and the importance of the data volume, database weight and label weight of the task data corresponding to each data label are scored according to the expert scoring method, and then the weighted sum is performed according to the data volume, database weight, label weight, the importance score of the data volume of the task data corresponding to each data label, the importance score of the database weight and the importance score of the label weight, so as to obtain the confidence of each data label. For example: if the data volume of the task data of a data label is 1, the database weight is 0.5, and the label weight is 0.7, according to the expert scoring method, the importance score of the data volume of the task data of the data label is 0.3, the importance score of the database weight is 0.3, and the importance score of the label weight is 0.4, then the confidence of the current data label can be determined to be 1×0.3+0.5×0.3+0.7×0.4=0.73.
[0048] Exemplarily, when the confidence calculation method is a machine learning method, the obtained data labels can be input into a pre-trained confidence determination model to obtain the confidence of each data label. The pre-trained confidence determination model is trained using historical data labels and the confidence of the corresponding historical data labels.
[0049] In this embodiment, by determining the confidence calculation method of each data label, the accuracy of the calculated confidence of each data label is improved, and then the target data label is determined by the confidence of each data label, providing a data basis for subsequently determining the target task data.
[0050] Step 130: Determine target task data in a database corresponding to each target data tag according to each target data tag.
[0051] Among them, the target task data refers to the data set that is screened from multiple databases based on each target data label and best meets the current task requirements.
[0052] Specifically, the data records corresponding to the target data tag can be searched in the multiple databases corresponding to the current task determined above, and then the data records corresponding to the target data tag found in each database are integrated together to obtain the target task data.
[0053] In addition, in order to improve efficiency, a parallel query method can be used, that is, to query the data records corresponding to the target data tags in multiple databases at the same time. At the same time, in order to facilitate subsequent data processing and analysis, the format can be unified when integrating data to ensure that the field name, data type, etc. remain consistent.
[0054] In this embodiment, target task data is determined in a database corresponding to each target data tag according to each target data tag, so that target data required for the task can be automatically acquired from multiple databases, thereby saving manpower costs.
[0055] The data acquisition method provided by the embodiment of the present invention includes: acquiring task data from multiple databases corresponding to the current task, and establishing a label tree based on the task data; determining the confidence calculation method of each data label, calculating the confidence of each data label based on the label level to which each data label belongs in the label tree and the confidence calculation method, and determining the target data label according to the confidence of each data label in each label level; determining the target task data in the database corresponding to each target data label according to each target data label. The above technical scheme first acquires task data from multiple databases corresponding to the current task, and establishes a label tree based on the task data, thereby realizing the establishment of a label tree for task data. Then, by determining the confidence calculation method of each data label, the accuracy of the calculated confidence of each data label is improved, and then the target data label is determined by the confidence of each data label, and finally, according to each target data label, the target task data is determined in the database corresponding to each target data label, thereby realizing the automatic acquisition of the target data required for the task in multiple databases, thereby saving labor costs. Compared with the prior art in which technicians manually obtain the target data required for the task from multi-source data, the present invention establishes a label tree for the task data, calculates the confidence of each data label based on the label tree of the task data and a determined confidence calculation method, determines the target data label according to the confidence of each data label, and finally determines the target task data according to each target data label. This can automatically obtain the target data required for the task from multiple databases, saving labor costs.
[0056] Figure 2 A flowchart of another data acquisition method provided by an embodiment of the present invention, which is specific based on the above embodiment. In this embodiment, the method may also include:
[0057] Step 210: Obtain task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data.
[0058] Furthermore, a label tree is established based on the task data, including: taking each data label as a cluster and calculating the similarity between each cluster; merging the two clusters corresponding to the minimum similarity into one cluster, returning to execute the similarity calculation between each cluster until the minimum similarity is greater than a preset threshold, and obtaining multiple clusters; for each cluster, constructing a label hierarchy corresponding to the cluster according to each data label corresponding to the cluster; calculating the intra-cluster similarity of each cluster, determining the hierarchical relationship between the label hierarchies corresponding to each cluster according to the intra-cluster similarity, and establishing a label tree according to the hierarchical relationship between the label hierarchies.
[0059] Specifically, after obtaining the task data, each data label in the task data can be regarded as a cluster, and then a suitable similarity calculation method can be selected according to the actual situation and application requirements to calculate the similarity between the clusters. For example, the similarity calculation method can be the cosine similarity method or the Jaccard similarity method. Then the selected similarity calculation method is applied to calculate the similarity between the clusters. Afterwards, if the smallest similarity among the calculated similarities is not greater than the preset similarity threshold, the two clusters corresponding to the smallest similarity are merged into one cluster, and the cluster set is updated at the same time, and the similarity between the clusters is recalculated until the smallest similarity among the calculated similarities is greater than the preset similarity threshold. Among them, the setting of the similarity threshold can be determined according to the specific application scenario and requirements.
[0060] Then, for each cluster in the latest cluster set, a label hierarchy corresponding to the cluster is constructed according to the data labels corresponding to the cluster. Then, according to the actual situation and application requirements, an appropriate similarity calculation method is selected to calculate the similarity within each cluster. Then, the hierarchical relationship between the label hierarchies corresponding to each cluster is determined according to the similarity within each cluster, and a label tree is established according to the hierarchical relationship between the label hierarchies.
[0061] Exemplarily, when the data labels corresponding to the task data are Z, Y, C, D, E, and F, Z, Y, C, D, E, and F are taken as 6 clusters, and the similarities between Z, Y, C, D, E, and F are calculated. If the two clusters corresponding to the minimum similarity are E and F and the minimum similarity is not greater than the preset threshold q, E and F are merged into one cluster, that is, {E, F}. Then, the similarities between Z, Y, C, D, and {E, F} are calculated, and the two clusters corresponding to the minimum similarity are C and D and the minimum similarity is not greater than the preset threshold q. C and D are merged into one cluster, that is, {C, D}, continue to calculate the similarity between Z, Y, {C, D} and {E, F}, and get the two clusters corresponding to the minimum similarity as Z and Y, and the minimum similarity is not greater than the preset threshold q. Merge Z and Y into one cluster, that is, {Z, Y}, and then calculate the similarity between {Z, Y}, {C, D} and {E, F} to get the minimum similarity q1, where q1>q. At this time, the similarity between each cluster is no longer calculated. It can be found that the hierarchical relationship between the data labels Z, Y, C, D, E and F is that Z and Y are at the same level, C and D are at the same level, and E and F are at the same level. Then calculate the intra-cluster similarity of {Z, Y}, {C, D} and {E, F} to determine the hierarchical relationship between the label levels corresponding to each cluster. If the intra-cluster similarity of cluster {Z, Y} is z, the intra-cluster similarity of cluster {C, D} is x, and the intra-cluster similarity of cluster {E, F} is c, z>x>c, then the hierarchical relationship corresponding to the data labels Z, Y, C, D, E, and F is that Z and Y are at the same level, C and D are at the same level, E and F are at the same level, E and F are the previous level of C and D, and C and D are the previous level of Z and Y.
[0062] In this embodiment, through the above steps, the hierarchical relationship between data labels can be automatically discovered, and a label tree can be constructed according to the determined hierarchical relationship. At the same time, the implementation process does not rely on the preset number or shape of categories and can adapt to different application scenarios.
[0063] Step 211: Determine a confidence calculation method for data labels corresponding to task data obtained from each database according to the amount of data in each database.
[0064] Specifically, the confidence calculation method of the data label corresponding to the task data obtained from each database can be determined by the difference between the data volumes of each database and the sum of the data volumes of each database.
[0065] Exemplarily, assuming that the optional confidence calculation methods include weighted average method, graph calculation method and machine learning method. Then, it can be determined whether the sum of the data volume of each database is greater than the preset data volume threshold. If the sum of the data volume of each database is not greater than the preset data volume threshold, it can be determined that the confidence calculation method is the weighted average method; if the sum of the data volume of each database is greater than the preset data volume threshold, the difference between the data volumes of each database can be calculated, and the maximum value of all calculated differences can be found. If this maximum difference is greater than the preset difference threshold, it can be determined that the confidence calculation method is the graph calculation method. If the calculated maximum difference is not greater than the preset difference threshold, it can be determined that the confidence calculation method is the machine learning method.
[0066] In addition, in order to improve the efficiency of determining the confidence calculation method, before determining the confidence calculation method of the data labels corresponding to the task data obtained from each database according to the data volume of each database, it is possible to first determine whether the task data includes the label weights and database weights of the data labels. If included, the confidence calculation method of the data labels can be directly determined to be the weighted average method; if not included, the confidence calculation method of the data labels corresponding to the task data obtained from each database can be determined according to the data volume of each database.
[0067] In this embodiment, the confidence calculation method of the data label of the appropriate task data is selected according to the data volume of each database, which can reduce the computing time and resource consumption, improve the efficiency of data processing, and improve the accuracy and efficiency of data analysis.
[0068] Step 212: For the current data tag, determine the data volume of the task data corresponding to the current data tag.
[0069] The data volume of the task data of the current data label refers to the number of times the current data label appears in the corresponding source database.
[0070] Specifically, for the current data tag, the amount of task data corresponding to the current data tag can be determined by counting the number of times the current data tag appears in the corresponding source database. For example, the data record corresponding to the current data tag can be searched in the source database corresponding to the current data tag, and then the amount of task data corresponding to the current data tag can be determined based on the number of data records found.
[0071] In this embodiment, by determining the data volume of the task data corresponding to the current data label, a data basis is provided for subsequently calculating the confidence of the current data label.
[0072] Step 213, calculate the confidence of the current data label according to the data volume of the task data corresponding to the current data label, the database weight of the database of the task data corresponding to the current data label, the label weight of the current data label, the label level to which the current data label belongs in the label tree, and the confidence calculation method.
[0073] Specifically, the initial confidence of the current data label can be calculated based on the label level to which the current data label belongs in the label tree and the confidence calculation method determined above. For example, if the determined confidence calculation method is a machine learning method, a hierarchical multi-label classification algorithm can be used to calculate the initial confidence of the current data label.
[0074] Then, the initial confidence of the current data label can be adjusted according to the data volume of the task data corresponding to the current data label, the database weight of the database corresponding to the task data with the current data label, and the label weight of the current data label to obtain the confidence of the current data label. Specifically, according to actual needs and circumstances, a suitable importance determination method can be selected to assign corresponding importance scores to the data volume, database weight, label weight and initial confidence of the task data of the current data label. After determining the importance scores, the data volume, database weight, label weight and initial confidence of the task data of the current data label can be weighted and summed according to the corresponding importance scores to obtain the confidence of the current data label. For example, the importance determination method can be an expert scoring method or an objective weighting method.
[0075] Exemplarily, assuming that the data volume of the task data of the current data label is 1, the database weight is 0.5, the label weight is 0.7, and the initial confidence is 0.6, the importance determination method is the expert scoring method, and according to the expert scoring method, the importance score of the data volume of the task data of the current data label is 0.1, the importance score of the database weight is 0.1, the importance score of the label weight is 0.1, and the importance score of the initial confidence is 0.7, then the confidence of the current data label can be determined to be 0.64.
[0076] In this embodiment, the confidence of the current data label is calculated based on the data volume of the task data corresponding to the current data label, the database weight of the database of the task data corresponding to the current data label, the label weight of the current data label, the label level to which the current data label belongs in the label tree, and the confidence calculation method, thereby improving the accuracy of the calculated confidence of the current data label.
[0077] Furthermore, before calculating the confidence of the current data label, the method further includes: adjusting the label weight of the current data label according to the update times and / or evaluation information of the current data label.
[0078] Among them, the update times of the current data label refers to the number of times the current data label is modified, which reflects the frequency of change of the data label and is used to evaluate the stability and reliability of the data label. The evaluation information of the current data label refers to the information that evaluates the quality of the current data label based on the actual business application situation.
[0079] Specifically, after obtaining the label weight of the current data label, the number of changes of the current data label can be calculated by querying the number of changes of the timestamp recorded by the current data label in the database, so as to obtain the number of updates of the current data label, and the evaluation information of the current data label can also be obtained by analyzing the internal business report. After that, it can be determined whether the number of updates of the current data label exceeds the preset update number threshold. If it exceeds, the label weight of the current data label can be reduced, otherwise the label weight of the current data label is not operated. After adjusting the label weight of the current data label according to the number of updates of the current data label, the label weight of the current data label can also be adjusted according to the evaluation information of the current data label. Specifically, if the evaluation information of the current data label is a positive evaluation (such as accurate, like or satisfied, etc.), the label weight of the current data label can be increased; if the evaluation information of the current data label is a negative evaluation (such as inaccurate, dislike or bad, etc.), the label weight of the current data label can be reduced. The label weight of the current data label can also be adjusted directly according to the evaluation information of the current data label.
[0080] In this embodiment, the label weight of the current data label is adjusted according to the update times and / or evaluation information of the current data label, so as to improve the accuracy of the label weight of the current data label.
[0081] Step 214: sort the data labels in each label level according to the confidence level of each data label in each label level to obtain the sorting results corresponding to each label level.
[0082] Specifically, after obtaining the confidence of each data label in each label level, the data labels in each label level can be sorted in descending or ascending order according to the confidence of each data label in each label level to obtain the sorting results corresponding to each label level.
[0083] In this embodiment, obtaining the sorting results corresponding to each label level provides a data basis for subsequently determining the target data label.
[0084] Step 215: Determine the target data label corresponding to each label level according to the sorting results corresponding to each label level.
[0085] If the sorting result corresponding to each label level is a descending sorting result, the data label ranked first in each label level can be determined as the target data label corresponding to each label level. If the sorting result corresponding to each label level is an ascending sorting result, the data label ranked last in each label level can be determined as the target data label corresponding to each label level, thereby determining the data label with the highest confidence in each label level as the target data label corresponding to each label level.
[0086] In this embodiment, by determining the target data label corresponding to each label level, a data basis is provided for subsequently determining the task data corresponding to each target data label.
[0087] Step 216: determine the database corresponding to each target data tag, and obtain the task data corresponding to each target data tag in the database corresponding to each target data tag.
[0088] Specifically, before determining the database corresponding to each target data tag, a correspondence table between the data tag and the database can be established. After determining each target data tag, the aforementioned determined correspondence table can be queried based on each target data tag to determine the database corresponding to each target data tag. Then, the task data corresponding to each target data tag is obtained from the database corresponding to each target data tag.
[0089] In this embodiment, by acquiring the task data corresponding to each target data tag, a data basis is provided for subsequently determining the target task data corresponding to the current task.
[0090] Step 217: determine the task data corresponding to each target data tag as the target task data corresponding to the current task.
[0091] In this embodiment, the target task data is relatively high-quality data that is screened out from the database corresponding to the current task according to the target data tag, which reduces the amount of data that needs to be processed to execute the current task, saves storage space, and uses high-quality data for task processing, which can reduce wrong decisions and extra work caused by data errors or inconsistencies. Therefore, by determining the target task data, not only can the quality of the data and the processing efficiency of the current task be improved, but also the performance of the current task can be improved.
[0092] The data acquisition method provided by the embodiment of the present invention includes: acquiring task data in multiple databases corresponding to the current task, establishing a label tree based on the task data; determining the confidence calculation method of the data label corresponding to the task data acquired in each database according to the data volume of each database; for the current data label, determining the data volume of the task data corresponding to the current data label; calculating the confidence of the current data label according to the data volume of the task data corresponding to the current data label, the database weight of the database of the task data corresponding to the current data label, the label weight of the current data label, and the label level to which the current data label belongs in the label tree and the confidence calculation method; sorting the data labels in each label level according to the confidence of each data label in each label level, and obtaining the sorting results corresponding to each label level; determining the target data label corresponding to each label level according to the sorting results corresponding to each label level; determining the database corresponding to each target data label, and acquiring the task data corresponding to each target data label in the database corresponding to each target data label; determining the task data corresponding to each target data label as the target task data corresponding to the current task. The above technical scheme first acquires task data in multiple databases corresponding to the current task, and establishes a label tree based on the task data, thereby realizing the establishment of a label tree for task data. Then, by determining the data volume of the task data corresponding to the current data label, the confidence of the current data label is calculated according to the data volume of the task data corresponding to the current data label, the database weight of the database of the task data corresponding to the current data label, the label weight of the current data label, the label level to which the current data label belongs in the label tree, and the confidence calculation method, thereby improving the accuracy of the calculated confidence of the current data label. Afterwards, the data labels in each label level are sorted according to the confidence of each data label in each label level, and the sorting results corresponding to each label level are obtained, and the target data labels corresponding to each label level are determined according to the sorting results corresponding to each label level, and then the database corresponding to each target data label is determined, and the task data corresponding to each target data label is obtained in the database corresponding to each target data label, and finally the task data corresponding to each target data label is determined as the target task data corresponding to the current task, thereby improving the quality of the data, reducing the amount of data to be processed, and reducing the wrong decisions and extra work caused by data errors or inconsistencies during task processing, thereby improving the processing efficiency and performance of the current task. Compared with the prior art in which technicians manually obtain the target data required for the task from multi-source data, the present invention establishes a label tree for the task data, calculates the confidence of each data label based on the label tree of the task data and a determined confidence calculation method, determines the target data label according to the confidence of each data label, and finally determines the target task data according to each target data label. This can automatically obtain the target data required for the task from multiple databases, saves labor costs, and improves the processing efficiency and performance of the target task.
[0093] In addition, the confidence calculation method of the data label is determined according to the data volume of each database, which reduces the computing time and resource consumption, improves the efficiency of data processing, and improves the accuracy and efficiency of data analysis.
[0094] Furthermore, the label weight of the current data label is adjusted according to the update times and / or evaluation information of the current data label, thereby improving the accuracy and reliability of the label weight of the current data label.
[0095] Figure 3 A structural schematic diagram of a data acquisition device provided in an embodiment of the present invention. The device and the data acquisition methods of the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiments of the data acquisition device, reference can be made to the embodiments of the above-mentioned data acquisition methods.
[0096] like Figure 3 As shown, the device comprises:
[0097] An acquisition module 310 is used to acquire task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data, wherein the task data includes data tags, and the tag tree is composed of multiple tag levels, each of which includes at least one data tag;
[0098] A first determination module 320 is used to determine a confidence calculation method for each of the data tags, calculate the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each of the data tags in each of the tag levels;
[0099] The second determination module 330 is used to determine the target task data in the database corresponding to each target data tag according to each target data tag.
[0100] On the basis of the above embodiment, a label tree is established based on the task data, including:
[0101] Taking each of the data labels as a cluster, calculating the similarity between the clusters;
[0102] Merge the two clusters corresponding to the minimum similarity into one cluster, and return to calculate the similarity between the clusters until the minimum similarity is greater than a preset threshold, thereby obtaining multiple clusters;
[0103] For each of the clusters, constructing the label hierarchy corresponding to the cluster according to each of the data labels corresponding to the cluster;
[0104] The intra-cluster similarity of each of the clusters is calculated, the hierarchical relationship between the label levels corresponding to each of the clusters is determined according to the intra-cluster similarity, and the label tree is established according to the hierarchical relationship between the label levels.
[0105] On the basis of the above embodiment, the confidence calculation method of each data tag is determined, including:
[0106] A confidence calculation method for the data label corresponding to the task data obtained from each database is determined according to the data amount of each database.
[0107] On the basis of the above embodiment, the confidence of each data tag is calculated based on the tag level to which each data tag belongs in the tag tree and the confidence calculation method, including:
[0108] For a current data tag, determining a data volume of task data corresponding to the current data tag;
[0109] The confidence of the current data label is calculated based on the data volume of the task data corresponding to the current data label, the database weight of the database of the task data corresponding to the current data label, the label weight of the current data label, the label level to which the current data label belongs in the label tree, and the confidence calculation method.
[0110] Based on the above embodiment, the device further includes:
[0111] A weight module is used to adjust the label weight of the current data label according to the update times and / or evaluation information of the current data label before calculating the confidence of the current data label.
[0112] On the basis of the above embodiment, determining the target data tag according to the confidence of each of the data tags in each of the tag levels includes:
[0113] Sorting the data tags in each of the tag levels according to the confidence of each of the data tags in each of the tag levels to obtain a sorting result corresponding to each of the tag levels;
[0114] The target data label corresponding to each label level is determined according to the sorting result corresponding to each label level.
[0115] Based on the above embodiment, the second determining module 330 is specifically used for:
[0116] Determine a database corresponding to each of the target data tags, and obtain task data corresponding to each of the target data tags in the database corresponding to each of the target data tags;
[0117] The task data corresponding to each of the target data tags is determined as the target task data corresponding to the current task.
[0118] The data acquisition device provided in the embodiment of the present invention can execute the data acquisition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0119] It is worth noting that in the embodiment of the above-mentioned data acquisition device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0120] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 4 A block diagram of an exemplary electronic device 4 suitable for implementing embodiments of the present invention is shown. Figure 4 The electronic device 4 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0121] like Figure 4 As shown, the electronic device 4 is in the form of a general purpose computing electronic device. The components of the electronic device 4 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0122] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0123] The electronic device 4 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 4, including volatile and non-volatile media, removable and non-removable media.
[0124] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 4 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0125] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0126] The electronic device 4 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the electronic device 4, and / or any device that enables the electronic device 4 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 4 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. Figure 4 As shown, the network adapter 20 communicates with other modules of the electronic device 4 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 4, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0127] The processing unit 16 executes various functional applications and page displays by running the programs stored in the system memory 28, for example, implementing the data acquisition method provided in the embodiment of the present invention, which includes:
[0128] Acquire task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data, wherein the task data includes data tags, and the tag tree is composed of multiple tag levels, and each of the tag levels includes at least one data tag;
[0129] Determine a confidence calculation method for each of the data tags, calculate the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each of the data tags in each of the tag levels;
[0130] The target task data is determined in a database corresponding to each of the target data tags according to each of the target data tags.
[0131] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data acquisition method provided by any embodiment of the present invention.
[0132] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, for example, a data acquisition method provided by an embodiment of the present invention is implemented. The method includes:
[0133] Acquire task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data, wherein the task data includes data tags, and the tag tree is composed of multiple tag levels, and each of the tag levels includes at least one data tag;
[0134] Determine a confidence calculation method for each of the data tags, calculate the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each of the data tags in each of the tag levels;
[0135] The target task data is determined in a database corresponding to each of the target data tags according to each of the target data tags.
[0136] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0137] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0138] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0139] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0140] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0141] In addition, the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.
[0142] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data acquisition method, characterized in that: The method comprises: Acquire task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data, wherein the task data includes data tags, and the tag tree is composed of multiple tag levels, and each of the tag levels includes at least one data tag; Determine a confidence calculation method for each of the data tags, calculate the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each of the data tags in each of the tag levels; The target task data is determined in a database corresponding to each of the target data tags according to each of the target data tags.
2. The data acquisition method according to claim 1, characterized in that: A label tree is established based on the task data, including: Taking each of the data labels as a cluster, calculating the similarity between the clusters; Merge the two clusters corresponding to the minimum similarity into one cluster, and return to calculate the similarity between the clusters until the minimum similarity is greater than a preset threshold, thereby obtaining multiple clusters; For each of the clusters, constructing the label hierarchy corresponding to the cluster according to each of the data labels corresponding to the cluster; The intra-cluster similarity of each of the clusters is calculated, the hierarchical relationship between the label levels corresponding to each of the clusters is determined according to the intra-cluster similarity, and the label tree is established according to the hierarchical relationship between the label levels.
3. The data acquisition method according to claim 1, characterized in that: Determine a confidence calculation method for each of the data labels, including: A confidence calculation method for the data label corresponding to the task data obtained from each database is determined according to the data amount of each database.
4. The data acquisition method according to claim 1, characterized in that: Calculating the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method includes: For a current data tag, determining a data volume of task data corresponding to the current data tag; The confidence of the current data label is calculated based on the data volume of the task data corresponding to the current data label, the database weight of the database of the task data corresponding to the current data label, the label weight of the current data label, the label level to which the current data label belongs in the label tree, and the confidence calculation method.
5. The data acquisition method according to claim 4, characterized in that: Before calculating the confidence of the current data label, it also includes: The label weight of the current data label is adjusted according to the update number and / or evaluation information of the current data label.
6. The data acquisition method according to claim 1, characterized in that: Determining a target data label according to the confidence of each of the data labels in each of the label levels includes: Sorting the data tags in each of the tag levels according to the confidence of each of the data tags in each of the tag levels to obtain a sorting result corresponding to each of the tag levels; The target data label corresponding to each label level is determined according to the sorting result corresponding to each label level.
7. The data acquisition method according to claim 1, characterized in that: Determining target task data in a database corresponding to each target data tag according to each target data tag includes: Determine a database corresponding to each of the target data tags, and obtain task data corresponding to each of the target data tags in the database corresponding to each of the target data tags; The task data corresponding to each of the target data tags is determined as the target task data corresponding to the current task.
8. A data acquisition device, characterized in that: The device comprises: An acquisition module, used to acquire task data from multiple databases corresponding to the current task, and establish a tag tree based on the task data, wherein the task data includes data tags, and the tag tree is composed of multiple tag levels, each of which includes at least one data tag; A first determination module is used to determine a confidence calculation method for each of the data tags, calculate the confidence of each of the data tags based on the tag level to which each of the data tags belongs in the tag tree and the confidence calculation method, and determine the target data tag according to the confidence of each of the data tags in each of the tag levels; The second determination module is used to determine the target task data in the database corresponding to each target data tag according to each target data tag.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data acquisition method described in any one of claims 1 to 7.
10. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, the data acquisition method according to any one of claims 1 to 7 is implemented.
