Data retrieval method and device based on data quality management platform

Abstract

The invention discloses a data retrieval method and device based on a data quality management platform. The method comprises the following steps: analyzing business requirements, and determining to-be-optimized data and retrieval efficiency expectation; cleaning and standardizing the to-be-optimized data based on the data quality management platform, and determining a retrieval data set; an index strategy is established for the retrieval data set, and indexes of all data in the retrieval data set are determined; adjusting the index through an index strategy based on the retrieval efficiency expectation, and determining the optimal index of each data in the retrieval data set; and based on the optimal index, according to the retrieval information of the retrieval user, obtaining retrieval data of the retrieval user.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval technology, and more specifically, to a data retrieval method and device based on a data quality management platform. Background Art

[0002] When the amount of data to be retrieved is very large, the database needs to process more data and query operations, which will extend the retrieval time. Therefore, when there is a lot of data, testers will waste a lot of time when performing large-scale retrieval of data, resulting in technical problems such as low data processing efficiency. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a data retrieval method and device based on a data quality management platform.

[0004] According to one aspect of the present invention, a data retrieval method based on a data quality management platform is provided, comprising:

[0005] Analyze business needs, determine the data to be optimized and the retrieval efficiency expectations;

[0006] Clean and standardize the data to be optimized based on the data quality management platform and determine the retrieval data set;

[0007] Establish an indexing strategy for the retrieval data set and determine the index of each data in the retrieval data set;

[0008] Based on the retrieval efficiency expectation, the index is adjusted through the index strategy to determine the optimal index for each data in the retrieval data set;

[0009] Based on the optimal index, the search data of the search user is obtained according to the search information of the search user.

[0010] Optionally, it also includes:

[0011] Use machine learning algorithms to analyze the historical query behavior and patterns of query users and determine the recommended index for query users;

[0012] Based on the recommendation index, the recommended data required by the query user is obtained from the retrieval data set.

[0013] Optionally, the data to be optimized is cleaned and standardized based on the data quality management platform to determine the retrieval data set, including:

[0014] The feature selection algorithm based on data mining is used to remove irrelevant features in the data to be optimized, and the data after the irrelevant features are removed is obtained;

[0015] After removing irrelevant features, the data is standardized to obtain the retrieval data set.

[0016] Optionally, it also includes:

[0017] The data quality management platform maintains metadata information using clustering algorithms and association rule mining algorithms, where the metadata information includes data sources, update frequency, and data quality scores.

[0018] According to another aspect of the present invention, there is provided a data retrieval device based on a data quality management platform, comprising:

[0019] Analysis module, used to analyze business needs, determine the data to be optimized and the retrieval efficiency expectations;

[0020] The first determination module is used to clean and standardize the data to be optimized based on the data quality management platform and determine the search data set;

[0021] The second determination module is used to establish an index strategy for the retrieval data set and determine the index of each data in the retrieval data set;

[0022] An adjustment module is used to adjust the index through an index strategy based on the retrieval efficiency expectation, and determine the optimal index for each data in the retrieval data set;

[0023] The first acquisition module is used to acquire the search data of the search user based on the optimal index and the search information of the search user.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0025] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0026] Therefore, the present invention standardizes data on the data quality management platform to ensure that data from different sources and formats can be processed uniformly, and establishes an effective data indexing strategy based on the importance, access frequency and usage pattern of the data to improve the retrieval speed, optimize the query algorithm, and improve the query response time. Testers can quickly retrieve the required data from a large amount of data, and then manually review the data, greatly reducing the time for retrieving data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0028] Figure 1 It is a flowchart of a data retrieval method based on a data quality management platform provided by an exemplary embodiment of the present invention;

[0029] Figure 2 is a structural schematic diagram of a data retrieval device based on a data quality management platform provided by an exemplary embodiment of the present invention;

[0030] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0031] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0032] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0033] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0034] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0035] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0036] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0037] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0038] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0041] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0042] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0043] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0044] Exemplary Methods

[0045] Figure 1 FIG. 1 is a flow chart of a data retrieval method based on a data quality management platform provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the data retrieval method 100 based on the data quality management platform includes the following steps:

[0046] Step 101, analyzing business requirements, determining data to be optimized and retrieval efficiency expectations;

[0047] Step 102, cleaning and standardizing the data to be optimized based on the data quality management platform, and determining a retrieval data set;

[0048] Step 103, establishing an index strategy for the retrieval data set, and determining the index of each data in the retrieval data set;

[0049] Step 104, adjusting the index through the index strategy based on the retrieval efficiency expectation, and determining the optimal index for each data in the retrieval data set;

[0050] Step 105 , based on the optimal index and according to the search information of the search user, the search data of the search user is obtained.

[0051] Specifically, data standardization is performed on the data quality management platform to ensure that data from different sources and formats can be processed uniformly, and effective data indexing strategies are established based on the importance, access frequency, and usage patterns of the data to improve retrieval speed, optimize query algorithms, and improve query response time. Implementation steps:

[0052] 1. Analyze business needs to determine which types of data are retrieved most frequently and users' expectations for data retrieval efficiency.

[0053] 2. Data cleaning and standardization: Use the data quality management platform to perform in-depth data cleaning, including removing duplicate records, correcting errors and filling missing values, while ensuring that all data follows unified naming and formatting standards to reduce ambiguity and improve efficiency during retrieval.

[0054] The present invention introduces a feature selection algorithm based on data mining, which is used to preprocess data on the data quality management platform to improve the efficiency of subsequent retrieval. The feature selection algorithm based on data mining automatically selects the most informative features by evaluating the correlation between each feature in the data set and the target attribute. This algorithm helps optimize the data preprocessing process on the data quality management platform, reduces the negative impact of irrelevant features on retrieval efficiency, and improves the speed and accuracy of data processing and query.

[0055] 3. Establish data indexes. Establish effective indexing strategies for data sets to ensure that key search attributes are indexed to improve query speed. By combining the nearest neighbor search (ANN) algorithm and local sensitive hashing (LSH), a multi-level index structure can be constructed to accelerate the retrieval process of large-scale data. The nearest neighbor search (ANN) can achieve fast nearest neighbor search on large-scale data sets, sacrificing a little accuracy in exchange for higher query speed, and can also save storage space. It can also maintain good performance when processing large-scale data, so that the system can cope with the growing amount of data without too much performance degradation. Local sensitive hashing (LSH) can achieve efficient approximate nearest neighbor search through hash mapping when establishing data indexes. It is particularly suitable for processing high-dimensional data and large-scale data sets. It can provide fast search speed and low computational complexity, providing an effective solution for data indexing and retrieval.

[0056] 4. When implementing metadata management, clustering algorithms and association rule mining algorithms can be used to maintain detailed metadata information on the data quality management platform, including data sources, update frequency, data quality scores, etc., to help users understand data content more quickly. Clustering algorithms, such as some K-means clustering and hierarchical clustering, can be used to group data sets in metadata management to help identify similarities and associations between data, which helps with data classification and organization. Association rule mining algorithms, such as some Apriori algorithms and FP-Growth algorithms. These algorithms can be used to analyze the association relationships in data sets, help discover potential association rules between metadata, and then guide the relationship modeling and management between data.

[0057] 5. Optimize query performance. You can use query optimizer algorithms and cache replacement algorithms to optimize query performance. By optimizing query performance, you can effectively improve data retrieval efficiency. The query optimizer algorithm can analyze query statements and select the best execution plan. It uses statistical information and cost models to estimate the cost of various execution plans and select the execution plan with the lowest cost. The cache replacement algorithm can use the cache to store frequently accessed data and reduce the number of accesses to the underlying storage system. The cache replacement algorithm is used to manage data items in the cache. The LRU (Least Recently Used) algorithm eliminates the longest unused data items based on the most recent access time, and the LFU (Least Frequently Used) algorithm eliminates less frequently accessed data items based on the access frequency.

[0058] 6. Provide advanced search functions and integrate advanced search tools into the data quality management platform, allowing the use of complex query statements and filtering conditions so that users can quickly locate the required data.

[0059] 7. Machine learning and intelligent recommendation: Use machine learning models to intelligently recommend relevant data based on the user's historical query behavior and patterns, reducing manual retrieval time. Intelligent recommendation algorithms use machine learning and data mining techniques to predict items or content that users may be interested in based on user historical behavior, interest preferences, and other relevant information, thereby providing personalized recommendations to users.

[0060] 8. User interface optimization: Design an intuitive and easy-to-use user interface so that users can quickly understand and operate data retrieval tools.

[0061] 9. Regular review and maintenance: Regularly review data quality and retrieval process to ensure that the data continues to maintain high quality, and adjust and optimize retrieval tools based on feedback.

[0062] Therefore, the present invention standardizes data on the data quality management platform to ensure that data from different sources and formats can be processed uniformly, and establishes an effective data indexing strategy based on the importance, access frequency and usage pattern of the data to improve the retrieval speed, optimize the query algorithm, and improve the query response time. Testers can quickly retrieve the required data from a large amount of data, and then manually review the data, greatly reducing the time for retrieving data.

[0063] Exemplary Devices

[0064] Figure 2 FIG. 1 is a schematic diagram of a data retrieval device based on a data quality management platform provided by an exemplary embodiment of the present invention. Figure 2 As shown, the device 200 includes:

[0065] Analysis module 210, used to analyze business needs, determine data to be optimized and retrieval efficiency expectations;

[0066] A first determination module 220 is used to clean and standardize the data to be optimized based on the data quality management platform to determine the search data set;

[0067] The second determination module 230 is used to establish an index strategy for the retrieval data set and determine the index of each data in the retrieval data set;

[0068] An adjustment module 240 is used to adjust the index through an index strategy based on the retrieval efficiency expectation, and determine the optimal index for each data in the retrieval data set;

[0069] The first acquisition module 250 is used to acquire the search data of the search user based on the optimal index and the search information of the search user.

[0070] Optionally, the apparatus 200 further includes:

[0071] The third determination module is used to analyze the historical query behavior and pattern of the query user by using a machine learning algorithm to determine the recommended index for the query user;

[0072] The second acquisition module is used to acquire the recommended data required by the query user from the retrieval data set based on the recommendation index.

[0073] Optionally, the first determining module 220 includes:

[0074] A cleaning submodule is used to clean irrelevant features in the data to be optimized based on a feature selection algorithm of data mining, and obtain data after cleaning irrelevant features;

[0075] The acquisition submodule is used to standardize the data after removing irrelevant features and obtain the retrieval data set.

[0076] Optionally, the apparatus 200 further includes:

[0077] The maintenance module is used to use clustering algorithms and association rule mining algorithms to maintain metadata information on the data quality management platform, where the metadata information includes data source, update frequency, and data quality score.

[0078] Exemplary Electronic Devices

[0079] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes one or more processors 31 and a memory 32 .

[0080] The processor 31 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0081] The memory 32 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0082] In addition, the input device 33 may also include, for example, a keyboard, a mouse, etc.

[0083] The output device 34 can output various information to the outside. The output device 34 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0084] Of course, to simplify, Figure 3 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device may also include any other appropriate components.

[0085] Exemplary computer program products and computer-readable storage media

[0086] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0087] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0088] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method for information mining of historical change records according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0089] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0090] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0091] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0092] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0093] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0094] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0095] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A data retrieval method based on a data quality management platform, characterized in that: include: Analyze business needs, determine the data to be optimized and the retrieval efficiency expectations; Cleaning and standardizing the data to be optimized based on the data quality management platform to determine the retrieval data set; Establishing an indexing strategy for the retrieval data set, and determining an index for each data in the retrieval data set; Based on the retrieval efficiency expectation, the index is adjusted through an index strategy to determine an optimal index for each data in the retrieval data set; Based on the optimal index, according to the search information of the search user, the search data of the search user is obtained.

2. The method according to claim 1, characterized in that Also includes: Analyze the historical query behavior and pattern of the query user using a machine learning algorithm to determine the recommended index for the query user; The recommended data required by the querying user is obtained from the retrieval data set based on the recommendation index.

3. The method according to claim 1, characterized in that The data to be optimized is cleaned and standardized based on the data quality management platform to determine the retrieval data set, including: Using a feature selection algorithm based on data mining to remove irrelevant features in the data to be optimized, and obtaining data after the irrelevant features are removed; The data after removing irrelevant features is standardized to obtain the retrieval data set.

4. The method according to claim 1, characterized in that Also includes: The metadata information maintained by the data quality management platform is processed using a clustering algorithm and an association rule mining algorithm, wherein the metadata information includes data source, update frequency, and data quality score.

5. A data retrieval device based on a data quality management platform, characterized in that: include: Analysis module, used to analyze business needs, determine the data to be optimized and the retrieval efficiency expectations; A first determination module is used to clean and standardize the data to be optimized based on a data quality management platform to determine a search data set; A second determination module is used to establish an index strategy for the retrieval data set and determine the index of each data in the retrieval data set; An adjustment module, configured to adjust the index through an index strategy based on the retrieval efficiency expectation, and determine an optimal index for each data in the retrieval data set; The first acquisition module is used to acquire the search data of the searching user based on the optimal index and the searching information of the searching user.

6. The device according to claim 5, characterized in that Also includes: A third determination module is used to analyze the historical query behavior and pattern of the query user by using a machine learning algorithm to determine the recommended index of the query user; The second acquisition module is used to acquire the recommended data required by the query user from the search data set based on the recommendation index.

7. The device according to claim 5, characterized in that The first determination module includes: A clearing submodule, used to clear irrelevant features in the data to be optimized based on a feature selection algorithm of data mining, and obtain data after the irrelevant features are cleared; The acquisition submodule is used to perform standardization processing on the data after removing irrelevant features to obtain the retrieval data set.

8. The device according to claim 5, characterized in that Also includes: The maintenance module is used to use clustering algorithm and association rule mining algorithm to maintain metadata information of the data quality management platform, wherein the metadata information includes data source, update frequency and data quality score.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 4.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 4.

Images