Data query method and device, equipment, medium and program product

Abstract

The invention provides a data query method which can be applied to the technical field of big data. The data query method comprises the following steps: in response to received user input information, identifying the type of the input information; under the condition that the type of the input information is the first type, determining a keyword according to the input information; performing association matching on nodes in the knowledge graph based on the keywords to obtain entity information; wherein the knowledge graph is used for reflecting structured representation of data lake content; querying target information in a pre-constructed data lake according to the entity information; wherein the data lake comprises a plurality of data partitions, the sources of the data in each data partition are different, the data in different data partitions have an association relationship, and the target information comprises the data with the association relationship in the plurality of different data partitions. The invention further provides a data query device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and more specifically to a data query method, device, equipment, medium and program product. Background Art

[0002] Most companies will sort out relevant content based on the prohibitive regulations of government regulatory departments so that internal staff can query and study, ensure compliance operations, and prevent potential risks.

[0003] Currently, the query of relevant regulations only supports users to search by keywords, which has low query flexibility and cannot meet the fuzzy query scenario. In addition, the query results only include the regulations themselves, and are not related to the relevant cases of the regulations. The query results are relatively simple and not conducive to user understanding. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data query method, apparatus, device, medium and program product for improving data query efficiency and accuracy.

[0005] According to a first aspect of the present disclosure, a data query method is provided, including: in response to receiving user input information, identifying the type of the input information; when the type of the input information is a first type, determining keywords based on the input information; performing association matching on nodes in a knowledge graph based on the keywords to obtain entity information; wherein the knowledge graph is a structured representation for reflecting the content of a data lake; querying target information in a pre-constructed data lake based on the entity information; wherein the data lake includes multiple data partitions, the source of data in each data partition is different, data in different data partitions are associated with each other, and the target information includes data in multiple different data partitions that are associated with each other.

[0006] According to an embodiment of the present disclosure, the method also includes: when the type of the input information is the second type, querying the target information in a pre-built data lake based on the input information; wherein the data lake includes multiple data partitions, the source of data in each data partition is different, and there is a correlation relationship between the data in different data partitions.

[0007] According to an embodiment of the present disclosure, in response to receiving user input information, identifying the type of the input information includes: parsing the input information; matching the parsed input information with predefined specified information to obtain a matching result; if the matching result does not meet a preset condition, determining that the type of the input information is a first type; if the matching result meets the preset condition, determining that the type of the input information is a second type.

[0008] According to an embodiment of the present disclosure, association matching is performed on nodes in a knowledge graph based on keywords to obtain entity information, including: obtaining a target node matching the keyword in the knowledge graph and a first node associated with the target node based on string matching; obtaining attribute information of the target node and the first node and relationship information between the first node and the target node respectively; determining entity information used to query the target information in a data lake based on the attribute information of the target node, the attribute information of the first node, and the relationship information.

[0009] According to an embodiment of the present disclosure, the nodes in the knowledge graph are associated and matched based on keywords to obtain entity information, including: constructing a first vector based on the keyword; matching the first vector with the vector representation of each node in the knowledge graph, and determining the target node according to the matching result; respectively obtaining the attribute information of the target node and the attribute information of the first node associated with the target node and the relationship information between the first node and the target node; determining the entity information used to query the target data in the data lake based on the attribute information of the target node, the attribute information of the first node and the relationship information.

[0010] According to an embodiment of the present disclosure, querying target information in a pre-constructed data lake based on entity information includes: constructing a query statement based on the entity information; inputting the query statement into the data lake, and the data lake locating the target partition based on the query statement and extracting target data from the target partition based on the query statement; and outputting the target data based on a specified format.

[0011] According to an embodiment of the present disclosure, the data lake includes at least a first partition and a second partition, and there is a dependency relationship between the first partition and the second partition. The method also includes: in response to a specified content update in the first platform, capturing the updated content and transferring the updated content to the first partition of the data lake; querying the second platform for target content that has an association relationship with the updated content, capturing the target content and transferring the target content to the second partition of the data lake; wherein there is an association relationship between the target content and the updated content.

[0012] According to an embodiment of the present disclosure, the method also includes: generating a first instruction based on the content in the first partition; wherein the first instruction is used to query the target content in the second platform; storing the queried target content in the second partition and establishing an association relationship between the target content and related content in the first partition.

[0013] A second aspect of the present disclosure provides a data query device, including: an identification module, for identifying the type of input information in response to receiving user input information; a determination module, for determining keywords based on the input information when the type of the input information is a first type; a matching module, for performing association matching on nodes in a knowledge graph based on keywords to obtain entity information; wherein the knowledge graph is a structured representation for reflecting the content of a data lake; a query module, for querying target information in a pre-constructed data lake based on the entity information; wherein the data lake includes multiple data partitions, the source of data in each data partition is different, there is an association relationship between data in different data partitions, and the target information includes data with an association relationship in multiple different data partitions.

[0014] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario diagram of the data query method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0019] Figure 2 A flowchart of a data query method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 3 A flowchart of a data query method according to another embodiment of the present disclosure is schematically shown;

[0021] Figure 4 Schematically shows a flow chart of identifying the type of input information in response to receiving user input information according to an embodiment of the present disclosure;

[0022] Figure 5aA flowchart schematically shows how to perform association matching on nodes in a knowledge graph based on keywords to obtain entity information according to an embodiment of the present disclosure;

[0023] Figure 5b A flowchart schematically shows how to perform association matching on nodes in a knowledge graph based on keywords to obtain entity information according to another embodiment of the present disclosure;

[0024] Figure 6 A flowchart of querying target information in a pre-built data lake according to entity information according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 7 A flowchart of updating the content in a data lake in real time according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 8 Schematically shows a flow chart of updating second partition data according to an embodiment of the present disclosure;

[0027] Fig. 9 The structure block diagram of the data query device according to the embodiment of the present disclosure is schematically shown;

[0028] Fig.10 A block diagram of an electronic device suitable for implementing a data query method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0033] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0034] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0036] An embodiment of the present disclosure provides a data query method, including: in response to receiving user input information, identifying the type of the input information; when the type of the input information is a first type, determining keywords based on the input information; performing association matching on nodes in a knowledge graph based on the keywords to obtain entity information; wherein the knowledge graph is a structured representation for reflecting the content of a data lake; querying target information in a pre-constructed data lake based on the entity information; wherein the data lake includes multiple data partitions, the source of data in each data partition is different, data in different data partitions are associated with each other, and the target information includes data in multiple different data partitions that are associated with each other.

[0037] Figure 1 The application scenario diagram of the data query method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0038] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0039] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0041] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the data query method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data query device provided in the embodiment of the present disclosure can generally be set in the server 105. The data query method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data query device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] The following will be based on Figure 1 The scene described by Figure 2~Figure 8 The data query method of the disclosed embodiment is described in detail.

[0045] Figure 2 The flowchart of a data query method according to an embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the data query of this embodiment includes operations S210 to S240.

[0047] In operation S210, in response to receiving user input information, a type of the input information is identified.

[0048] In some embodiments, the user inputs information through a specified interface or interface, and the input information is used to represent the user's query needs. In the first case, when the user needs to obtain very specific and accurate information, the user's input information will be relatively accurate, for example, the input information includes the file number and file name that the user wants to query. In the second case, when the user only knows part of the information or keywords, or the user wants to obtain a wider range of results, the user's input information will also be relatively vague, and may only include some keywords.

[0049] Therefore, the user's query needs can be analyzed by identifying the type of user input information. Different types of input information correspond to different data query processes, which effectively improves the efficiency and accuracy of data query and flexibly adapts to different query scenarios and needs.

[0050] In operation S220, in a case where the type of the input information is the first type, a keyword is determined according to the input information.

[0051] In some embodiments, the query scenario corresponding to the first type is a fuzzy scenario, and when the input information is of the first type, keywords for the query are determined based on the input information.

[0052] Preprocess the input information. The preprocessing operations can include, for example, text cleaning, word segmentation, and stop word removal. Among them, text cleaning is used to remove irrelevant characters, punctuation marks, stop words, etc. from the user input to reduce noise. Word segmentation is used to split the text into individual words or phrases. Stop words that frequently appear in the input information but contribute little to the meaning of the text (such as "de", "le", etc.) are removed. Extract keywords from the preprocessed input information. For example, keywords can be extracted from the preprocessed input information through methods such as the term frequency-inverse document frequency (TF-IDF) algorithm and the TextRank algorithm. Extracting keywords for data query from the input information can effectively narrow the search scope, quickly locate relevant information in the data lake, and effectively improve the query efficiency and accuracy.

[0053] In operation S230, perform associative matching on the nodes in the knowledge graph based on the keywords to obtain entity information; among them, the knowledge graph is a structured representation used to reflect the content of the data lake.

[0054] In some embodiments, the knowledge graph is constructed according to the data lake, which stores data from different sources and in different formats under the target scenario. The knowledge graph uses a graph-structured data model or topology to integrate data and is a method of representing information about real-world entities and their interrelationships. The knowledge graph can clearly display the hierarchical structure and association relationships among data partitions, data tables, and fields in the data lake.

[0055] By matching the keywords with the nodes in the knowledge graph, quickly locate the relevant entity information, and perform precise queries in the data lake based on the entity information, which can effectively improve the efficiency and accuracy of data query. And by performing associative matching in the knowledge graph according to the keywords, flexible combination and query of different input information can be realized, improving the flexibility of data query.

[0056] In operation S240, query the target information in the pre-constructed data lake according to the entity information; among them, the data lake includes multiple data partitions, the data sources in each data partition are different, and there are association relationships among the data in different data partitions. The target information includes data with association relationships in multiple different data partitions.

[0057] In some embodiments, the data lake stores data from different sources and in different formats. Based on the entity information in the knowledge graph, relevant data sets or records in the data lake can be quickly located, avoiding blind searches and a large amount of data screening in traditional query methods, effectively improving the query efficiency, and reducing query time and resource consumption.

[0058] In the specific implementation process, each entity in the data graph will have a unique identifier or label. The identifier or label of the target entity can be used as a query condition to perform query operations in the data lake to obtain the target data.

[0059] The data query method provided by the embodiment of the present disclosure, by constructing a knowledge graph in the data lake, cleans and displays the hierarchical structure and association relationship between each data partition, data table, and field in the data lake, helps to intuitively understand the association relationship between each data and the data overview and distribution characteristics in the data lake, realizes the rapid positioning of the target data, avoids blind search and large-scale data screening, significantly improves the query efficiency, and reduces the query time and resource consumption. In addition, the knowledge graph helps to realize cross-dataset and cross-field association queries, helps to meet diversified queries, and improves the comprehensiveness of query results. Compared with traditional databases, the present disclosure is based on databases for data storage, which can effectively improve the diversity of data storage. Compared with databases that only support structured data storage, data lakes can support the storage of multiple data types and data formats. In addition, the data lake adopts distributed storage, has good scalability, can be easily expanded as the amount of data increases, meets the needs of large amounts of data and high concurrent access, and the data lake can integrate data from multiple data sources into a centralized storage system, reducing the cost of data storage and management.

[0060] Figure 3 The following schematically shows a flow chart of a data query method according to another embodiment of the present disclosure.

[0061] like Figure 3 As shown, the data query of this embodiment includes operations S210' to S220'.

[0062] In operation S210, in response to receiving user input information, a type of the input information is identified.

[0063] In some embodiments, operation S210′ is performed with Figure 2 The operation details of operation S210 in the embodiments are the same and will not be described in detail here.

[0064] In operation S220', when the type of the input information is the second type, the target information is queried in the pre-built data lake based on the input information; wherein the data lake includes multiple data partitions, the source of data in each data partition is different, and there is a correlation relationship between the data in different data partitions.

[0065] In some embodiments, a query statement is constructed based on the input information and the first data corresponding to the input information is queried in the data lake based on the query statement. The association relationship between the first data and data in other data partitions other than the data partition where the first data is located is analyzed, and second data that has an association relationship with the first data is obtained based on the association relationship, and the first data and the second data are output as target data.

[0066] In the specific implementation process, the first data and the second data can be quickly located by querying the data directory. For example, if the input information is the penalty document number, the data partition used to store the penalty document is directly located based on the data directory, and the target penalty document corresponding to the penalty document number is searched in the data partition. By analyzing the target penalty document, the second data that has an association relationship with the target penalty document in other data partitions, such as the legal provisions / regulatory basis, business type, public opinion information, etc. corresponding to the target penalty document.

[0067] The data query method provided by the embodiment of the present disclosure adopts different query processes for different query scenarios. In the scenario of precise query, precise query can be directly performed in the data lake based on the input information, so that the required data can be quickly located and the query speed can be improved. In the scenario of fuzzy query, the query scope can be quickly narrowed based on the structured information in the knowledge graph, the accuracy and efficiency of the query can be improved, the access method of the data can be optimized, the number of scans of the data lake during the query process can be reduced, the system load can be reduced, and unnecessary resource consumption can be reduced. By adopting different query methods for different input information, the flexibility of the query can be effectively improved, the use of resources can be optimized, and the query efficiency can be improved.

[0068] Figure 4 The flowchart schematically shows a method of identifying the type of input information in response to receiving user input information according to an embodiment of the present disclosure.

[0069] like Figure 4 As shown, in response to receiving user input information, identifying the type of the input information in this embodiment includes operations S410 to S440.

[0070] In operation S410, input information is parsed.

[0071] In operation S420, the parsed input information is matched with predefined designated information to obtain a matching result.

[0072] In some embodiments, after receiving the user's input information, the input information is parsed, for example, irrelevant characters in the input information are removed and the input information is segmented, etc., to facilitate subsequent processing. Based on the parsing of the input information, the parsed input information is matched with predefined specified information. The matching method may include, for example, string matching, similarity calculation, etc., and the matching degree between the input information and the specified information is used as the matching result.

[0073] When the matching method is string matching, for example, a substring contraction algorithm (such as the KMP algorithm and the Boyer-Moore algorithm) can be used to find whether the input information contains a part of the specified information, and the matching degree is evaluated according to the length or proportion of the contained part. When the matching method is similarity calculation, the input information and the specified information are converted into vector representations respectively. For example, the input information and the specified information can be converted into vector representations through a bag-of-words model, a TF-IDF algorithm, or a word embedding algorithm, and then the cosine value between the two vectors is calculated. The cosine value can be used to reflect the matching degree between the input information and the specified information. The closer the cosine value is to 1, the more similar the two information are.

[0074] In operation S430, when the matching result does not satisfy the preset condition, it is determined that the type of the input information is the first type.

[0075] In operation S440, when the matching result satisfies the preset condition, it is determined that the type of the input information is the second type.

[0076] In some embodiments, for example, a preset threshold may be used to determine whether the matching result satisfies the preset condition. For example, when the matching result is greater than the preset threshold, the matching result satisfies the preset condition, indicating that the input information contains the specified information for accurate retrieval, and the type of the input information is determined to be the second type. When the matching result is less than the preset threshold, the matching result does not satisfy the preset condition, indicating that the input information does not contain the specified information for accurate retrieval, and the type of the input information is determined to be the first information.

[0077] The data query method provided by the present disclosure quickly determines the type of input information by matching the input information with predefined specified information, and thus selects different query strategies according to the type of input information, thereby optimizing query efficiency and query accuracy and meeting diverse query needs.

[0078] Figure 5a The flowchart schematically shows a method of performing association matching on nodes in a knowledge graph based on keywords to obtain entity information according to an embodiment of the present disclosure.

[0079] like Figure 5aAs shown, this embodiment performs association matching on nodes in the knowledge graph based on keywords to obtain entity information, including operations S510a to S530a.

[0080] In operation S510a, a target node matching the keyword in the knowledge graph and a first node associated with the target node are obtained based on string matching.

[0081] In some embodiments, a string matching algorithm (such as KMP algorithm, BM algorithm, etc.) is used to search for nodes matching the keyword, and these nodes are target nodes. For each target node, its adjacent nodes are traversed to find the first node associated with the target node. These first nodes may have a direct relationship or attribute association with the target node.

[0082] In operation S520a, attribute information of the target node and the first node and relationship information between the first node and the target node are acquired respectively.

[0083] In some embodiments, for the target node and the first node, their attribute information is extracted from the knowledge graph. The attribute information generally includes the name, type, description, etc. of the node. Such attribute information helps to further understand the characteristics and meaning of the node.

[0084] The relationship information between the target node and the first node is extracted from the knowledge graph. The relationship information generally includes the type, direction, weight, etc. of the relationship between the target node and the first node.

[0085] In operation S530a, entity information for querying target information in the data lake is determined based on the attribute information of the target node, the attribute information of the first node, and the relationship information.

[0086] In some embodiments, for a target node, the attribute information of the target node, the attribute information of the first node, and the relationship information therebetween are directly obtained and integrated to form a complete information set, and entity information used to query the target information in the data lake is determined based on the integrated information set. Such entity information generally includes the name, type, attributes, and relationship with other entities of the entity, and is used to quickly query the target information in the data lake.

[0087] Figure 5b The flowchart schematically shows a method of performing association matching on nodes in a knowledge graph based on keywords to obtain entity information according to another embodiment of the present disclosure.

[0088] like Figure 5b As shown, this embodiment performs association matching on nodes in the knowledge graph based on keywords to obtain entity information, including operations S510b to S530b.

[0089] In operation S510b, a first vector is constructed based on the keyword.

[0090] In some embodiments, for example, the keyword can be passed into a pre-trained word vector model (such as a Word2Vec model, a GloVe model, a BERT model, etc.), and a first vector is obtained based on the pre-trained word vector model, where the first vector is a vector representation of the keyword.

[0091] In operation S520b, the first vector is matched with the vector representation of each node in the knowledge graph, and the target node is determined according to the matching result.

[0092] In some embodiments, each node in the knowledge graph corresponds to a vector representation. The vector representation is obtained through knowledge graph embedding technology. Embedding technology can map entities and relationships in the knowledge graph to a continuous vector space, thereby facilitating numerical operations and semantic information acquisition.

[0093] In the specific implementation process, the similarity (such as cosine similarity) is calculated between the first vector and the vector representation of each node in the knowledge graph, and based on the similarity sorting results, the node with the highest similarity to the first vector is selected as the target node.

[0094] In operation S530b, attribute information of the target node, attribute information of a first node associated with the target node, and relationship information between the first node and the target node are acquired respectively.

[0095] In some embodiments, the nodes in the knowledge graph are interconnected through relationships, and all nodes associated with the target node are determined as first nodes (i.e., nodes that have a direct relationship with the target node). Each node in the knowledge graph is associated with a series of attribute information, which can be obtained by querying the database or graph database of the knowledge graph. For the target node and the first node, all relevant attribute information needs to be obtained to further understand the characteristics and meaning of the node. In addition, the relationship information between the target node and the first node is extracted from the knowledge graph. The relationship information generally includes the type, direction, weight, etc. of the relationship between the target node and the first node. The relationship information helps to understand the relevance and context between the nodes.

[0096] In operation S540b, entity information for querying target information in the data lake is determined based on the attribute information of the target node, the attribute information of the first node, and the relationship information.

[0097] In some embodiments, the attribute information of the target node, the attribute information of the first node, and the relationship information between the target node and the first node are integrated and processed to form entity information for querying the target information in the data lake, wherein the entity information has strong context and relevance, which helps to accurately find the target information in the data lake.

[0098] The data query method provided by the embodiment of the present disclosure quickly finds knowledge graph nodes related to keywords through vector representation and similarity calculation, improves the query scope and time, and constructs entity information based on the relationship information between nodes, thereby improving the context and correlation between nodes, meeting query requirements, and improving the comprehensiveness of query results.

[0099] The embodiments of the present disclosure provide two methods for querying entity information: string matching query and vector representation query. The string matching method is more accurate and has a lower query cost. The vector representation method can handle complex semantic relationships and queries of different data types. The selection can be made according to the actual application scenario, and the present disclosure is not limited here.

[0100] Figure 6 The flowchart of querying target information in a pre-built data lake based on entity information according to an embodiment of the present disclosure is schematically shown.

[0101] like Figure 6 As shown, the method of querying target information in a pre-built data lake according to entity information in this embodiment includes operations S610 to S630.

[0102] In operation S610, a query statement is constructed based on the entity information.

[0103] In some embodiments, the entity information may include the attributes, ranges, and conditions of the data to be queried, and a structured query statement is constructed based on the entity information, wherein the query statement should include elements such as which fields to select, which data partition or table to query from, and the query conditions.

[0104] In operation S620, a query statement is input into the data lake, and the data lake locates the target partition based on the query statement and extracts target information from the target partition based on the query statement.

[0105] In some embodiments, the constructed query statement is input into a query interface in the data lake, and the data lake queries the target information according to the information in the query statement.

[0106] In the specific implementation process, in response to receiving a query statement, the query statement is parsed, and an execution plan is determined based on the grammatical structure of the query statement and the query intent, such as selecting the best index, adjusting the table connection order, merging or splitting query operations, etc., to reduce resource consumption during query execution. Specifically, the data lake will quickly locate the target partition based on the conditions in the query statement (such as time range, data type, etc.), and after locating the target partition, the data lake will extract the target information from the target partition based on the field selection requirements in the query statement.

[0107] In operation S630, the target information is output based on a specified format.

[0108] In some embodiments, the extracted target data is further processed, such as data cleaning, conversion, aggregation, etc. The processing operations make the data more in line with the query requirements, improve the availability of the query results, and select an appropriate format to output the target information based on the actual needs of the user, where the specified format may include, for example, tables, charts, etc.

[0109] Furthermore, when the entity information contains multiple data to be queried, multiple query statements can be constructed respectively, where each query statement targets a specific data to be queried and includes corresponding field selection, table or data set selection, query conditions, etc. The data lake performs concurrent queries based on multiple query statements to improve query efficiency.

[0110] The disclosed embodiment constructs query statements through entity information, which can accurately locate the target partition in the data lake and realize fast query of target data. And constructing query statements based on entity information can effectively improve the flexibility of query, adjust query statements according to actual needs, and meet diverse data analysis needs.

[0111] Figure 7 The flowchart of real-time updating of content in a data lake according to an embodiment of the present disclosure is schematically shown.

[0112] like Figure 7 As shown, the real-time updating of the content in the data lake in this embodiment includes operations S710 to S720.

[0113] In operation S710 , in response to a content update specified in the first platform, the updated content is captured and delivered to a first partition of the data lake.

[0114] In operation S720, target content that is associated with the updated content is searched in the second platform, the target content is captured and transferred to the second partition of the data lake; wherein the target content is associated with the updated content.

[0115] In some embodiments, the data in the data lake may come from different first and second platforms. The data lake includes at least a first partition and a second partition, and the first partition and the second partition are in a one-way dependency relationship, wherein the first partition is used to store the first data from the first platform, and the second partition is used to store the second data from the second platform. The second data in the second partition needs to reflect the latest status of the first data in the first partition, and the first partition and the second partition are in a one-way dependency relationship, that is, when the first data in the first partition is updated, the second data in the second partition will also be updated accordingly.

[0116] The first platform can be, for example, an official platform, such as a website of a government department, court, or financial institution, and the designated object is an object that needs to be stored in the data lake, such as updated legal provisions, relevant regulations, etc. By monitoring the content source designated in the first platform, such as a web page, database, or API interface, etc., the update of the designated content is captured in real time, the captured updated content is preprocessed, and the preprocessed updated content is transmitted to the first partition of the data lake in accordance with the specified format and protocol and stored in the first partition as the first data.

[0117] The second platform may be, for example, an unofficial platform, such as a news website, social media, etc., for obtaining public opinion information related to the first data, such as a news report corresponding to a certain legal provision, etc. In response to the first data in the first partition being updated, the second platform is queried for target content that is associated with the updated content, the queried target content is preprocessed, such as deduplication, format conversion, etc., and the preprocessed target content is transferred to the second partition of the data lake as the second data storage.

[0118] The data query method provided in the embodiment of the present disclosure effectively improves the timeliness of data lake updates and ensures the real-time nature of data storage by real-time monitoring and capturing of designated content in the first platform. By partitioning data obtained from different sources, it helps to achieve rapid retrieval and access of data and improve the efficiency of data query.

[0119] Figure 8 The flowchart of updating the second partition data according to the embodiment of the present disclosure is schematically shown.

[0120] like Figure 8 As shown, updating the second partition data in this embodiment includes operations S810 to S820.

[0121] In operation S810, a first instruction is generated based on content in a first partition; wherein the first instruction is used to search for target content in a second platform.

[0122] In operation S820, the searched target content is stored in the second partition and an association relationship is established between the target content and related content in the first partition.

[0123] In some embodiments, a first instruction is generated based on the content in the first partition, and the first instruction clearly indicates the target content to be searched in the second platform. For example, if the first partition includes provision 1, the first command is used to search for content A related to provision 1 in real time in the second platform. The first instruction is used to search or query the content in the second platform, find the target content, store the target content in the specified location of the second content, and indicate the association relationship between the target content and the related content in the first partition (for example, the association relationship between the target content and the related content in the first partition can be recorded through an association table or index)

[0124] The data query method provided by the embodiment of the present disclosure can more effectively utilize data resources by establishing association relationships between different partitions. When it is necessary to access or analyze related data, relevant content can be quickly found, thereby improving the efficiency and accuracy of data processing. After storing the target content in the second partition and establishing an association relationship, the data can be accessed more conveniently. Whether in the first partition or the second partition, the required content can be quickly found through the association table or index.

[0125] Based on the above data query method, the present disclosure also provides a data query device. Fig. 9 The device is described in detail.

[0126] Fig. 9 The structural block diagram of the data query device according to the embodiment of the present disclosure is schematically shown.

[0127] like Fig. 9 As shown, the data query device 900 of this embodiment includes an identification module 910 , a determination module 920 , a matching module 930 and a query module 940 .

[0128] The identification module 910 is used to identify the type of the input information in response to receiving the user input information. In one embodiment, the identification module 910 can be used to perform the operation S210 described above, which will not be described in detail here.

[0129] The determination module 920 is used to determine the keyword according to the input information when the type of the input information is the first type. In one embodiment, the determination module 920 can be used to perform the operation S220 described above, which will not be described in detail here.

[0130] The matching module 930 is used to perform association matching on nodes in the knowledge graph based on keywords to obtain entity information; wherein the knowledge graph is a structured representation used to reflect the content of the data lake. In one embodiment, the matching module 930 can be used to perform the operation S230 described above, which will not be repeated here.

[0131] The query module 940 is used to query the target information in the pre-built data lake according to the entity information; wherein the data lake includes multiple data partitions, the data in each data partition has different sources, the data between different data partitions have associations, and the target information includes data with associations in multiple different data partitions. In one embodiment, the query module 940 can be used to perform the operation S240 described above, which will not be repeated here.

[0132] According to an embodiment of the present disclosure, any multiple modules of the identification module 910, the determination module 920, the matching module 930 and the query module 940 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the identification module 910, the determination module 920, the matching module 930 and the query module 940 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the identification module 910, the determination module 920, the matching module 930 and the query module 940 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0133] Fig.10 A block diagram of an electronic device suitable for implementing a data query method according to an embodiment of the present disclosure is schematically shown.

[0134] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0135] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0136] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0137] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0138] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0139] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data query method provided by the embodiment of the present disclosure.

[0140] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1001. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0141] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0142] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0143] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0144] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0146] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data query method, characterized in that: The method comprises: In response to receiving user input information, identifying a type of the input information; In a case where the type of the input information is the first type, determining a keyword according to the input information; Based on the keywords, the nodes in the knowledge graph are matched with each other to obtain entity information; wherein the knowledge graph is a structured representation used to reflect the content of the data lake; Query target information in a pre-built data lake according to the entity information; wherein the data lake includes multiple data partitions, the source of data in each data partition is different, data between different data partitions are associated, and the target information includes data with an associated relationship in multiple different data partitions.

2. The data query method according to claim 1, characterized in that: The method further comprises: When the type of the input information is the second type, target information is queried in a pre-built data lake based on the input information; wherein the data lake includes multiple data partitions, data in each data partition comes from a different source, and data in different data partitions are associated with each other.

3. The data query method according to claim 1 or 2, characterized in that: In response to receiving user input information, identifying the type of the input information includes: Parsing the input information; Match the parsed input information with the predefined specified information to obtain a matching result; If the matching result does not satisfy the preset condition, determining that the type of the input information is the first type; When the matching result satisfies a preset condition, it is determined that the type of the input information is the second type.

4. The data query method according to claim 1, characterized in that: The step of performing association matching on the nodes in the knowledge graph based on the keywords to obtain entity information includes: Acquire a target node matching the keyword in the knowledge graph and a first node associated with the target node based on string matching; Respectively obtaining attribute information of the target node and the first node and relationship information between the first node and the target node; Entity information for querying target information in the data lake is determined based on the attribute information of the target node, the attribute information of the first node, and the relationship information.

5. The data query method according to claim 1, characterized in that: The step of performing association matching on the nodes in the knowledge graph based on the keywords to obtain entity information includes: constructing a first vector based on the keyword; Matching the first vector with the vector representation of each node in the knowledge graph, and determining the target node according to the matching result; Respectively acquiring attribute information of the target node, attribute information of a first node associated with the target node, and relationship information between the first node and the target node; Entity information for querying target data in the data lake is determined based on the attribute information of the target node, the attribute information of the first node, and the relationship information.

6. The data query method according to claim 1, characterized in that: The querying target information in the pre-built data lake according to the entity information includes: Constructing a query statement based on the entity information; Input the query statement into the data lake, and the data lake locates the target partition based on the query statement and extracts target data from the target partition based on the query statement; The target data is output based on a specified format.

7. The data query method according to claim 1, characterized in that: The data lake includes at least a first partition and a second partition, and there is a dependency relationship between the first partition and the second partition. The method further includes: In response to a specified content update in the first platform, capturing the updated content and delivering the updated content to a first partition of the data lake; Querying the second platform for target content associated with the updated content, capturing the target content and transferring the target content to a second partition of the data lake; There is an association relationship between the target content and the update content.

8. The data query method according to claim 7, characterized in that: The method further comprises: Generate a first instruction based on the content in the first partition; wherein the first instruction is used to query the target content in the second platform; The searched target content is stored in the second partition and an association relationship is established between the target content and related content in the first partition.

9. A data query device, characterized in that: The device comprises: an identification module, configured to identify the type of the input information in response to receiving the user input information; a determination module, configured to determine a keyword according to the input information when the type of the input information is the first type; A matching module, used to perform association matching on nodes in the knowledge graph based on the keywords to obtain entity information; wherein the knowledge graph is a structured representation used to reflect the content of the data lake; A query module is used to query target information in a pre-built data lake according to the entity information; wherein the data lake includes multiple data partitions, the data in each data partition has a different source, and there is a correlation relationship between the data in different data partitions, and the target information includes data with a correlation relationship in multiple different data partitions.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images