Data processing method and device

Abstract

The embodiment of the invention provides a data processing method and device, in the process of constructing a knowledge graph by using a data table, according to an entity relation graph of the data table, matched concepts and entity models can be generated, and then attributes extracted from the entity relation graph are clustered to form a concept value set and a mapping relation, so that the knowledge graph is obtained. The method can be used for optimizing the generated SPG model. And entity attributes of the knowledge graph are extracted from the optimized SPG model through the large model to assist construction of the knowledge graph and data analysis based on the knowledge graph. According to the embodiment, accurate data analysis can be better facilitated.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of computer technology, and in particular, to methods and devices for data processing. Background Art

[0002] Big data analysis typically relies on massive data tables, often using SQL for data processing. However, for data analysis cases involving complex relationships, analysis efficiency is low. Knowledge graphs, as structured semantic knowledge bases, can symbolically describe concepts in the physical world and their interrelationships. Based on a graph structure, knowledge graphs, through visual analysis and expert rule expression, can delve deeper into fluctuations in various indicators and their causes.

[0003] The basic building blocks of a knowledge graph are typically "entity-relationship-entity" triples, along with entity and related attribute-value pairs. Entities are connected through relationships, forming a network-like knowledge structure. An entity (which can serve as a node in a knowledge graph) can be any real-world object, such as a person, a payment device, or a company, or an abstract concept like artificial intelligence. Relationships are represented by edges in the knowledge graph, describing the associated attributes between entities, such as investment relationships and payment relationships. In specific business domains, knowledge graphs can be constructed based on large amounts of data. However, big data technicians may be familiar with the ER model for data and have no prior knowledge of the SPG model and knowledge terminology associated with knowledge graphs. Therefore, providing an automated way to transform a table-based data system into a knowledge graph is crucial for lowering the barrier to entry for knowledge graph users and improving the efficiency of big data analytics. Summary of the Invention

[0004] One or more embodiments of this specification describe a method and device for data preprocessing to construct a knowledge graph, which is used to solve one or more problems mentioned in the background technology.

[0005] According to a first aspect, a data processing method is provided for data processing in the process of constructing a knowledge graph based on a business data table, the method comprising: obtaining an entity relationship diagram, the entity relationship diagram being used to describe a data table structure related to a business objective of the current knowledge graph construction; extracting attribute information according to the entity relationship diagram, and embedding the attribute values ​​to determine corresponding embedding vectors; optimizing attributes using the embedding vectors to obtain an optimized concept set, wherein the optimized concept set comprises: clustering the embedding vectors obtained by embedding the attribute values ​​for enumerable attributes, and forming concept types according to the clustering categories; constructing at least one concept tree of the knowledge graph by optimizing the concept set, and the knowledge semantic framework SPG model constituted by the concept trees is used to construct the knowledge graph.

[0006] In one embodiment, a single attribute value corresponds to a single or multiple data items in a data table, and an enumerable attribute is an attribute whose attribute values ​​can be exhaustively enumerated; the method of using each embedding vector to optimize the attributes to obtain an optimized concept set includes: for enumerable attributes, comparing each attribute value with a known concept type, and in the case of a match, attributing the corresponding attribute value to the known concept type; for attribute values ​​in the enumerable attributes that do not match the known concept type, using the corresponding embedding vector to cluster the attributes, and determining a new concept type for each cluster category.

[0007] In a further embodiment, clustering attributes using corresponding embedding vectors includes several iterations of the following clustering step: taking a predetermined number of concept values ​​ranked top in descending order of embedding vector similarity as the current same cluster category.

[0008] In another further embodiment, determining a new concept type for each cluster category includes: inputting the embedding vector corresponding to each cluster category and the predetermined concept type as prompt information into the large language model, and determining the new concept type according to the output result of the large language model.

[0009] In one embodiment, extracting attribute information according to the entity relationship diagram includes: summarizing attribute value distribution of attributes in the entity relationship diagram; and sampling data in the data table according to the attribute value distribution summary to extract attribute information.

[0010] According to a second aspect, a data processing device is provided for processing data in a process of constructing a knowledge graph based on a business data table, the device comprising:

[0011] An acquisition unit configured to acquire an entity relationship diagram, wherein the entity relationship diagram is used to describe a data table structure related to a business objective constructed by the current knowledge graph;

[0012] an embedding unit configured to embed the attribute values ​​and determine corresponding embedding vectors;

[0013] an optimization unit configured to optimize attributes using each embedding vector to obtain an optimized concept set, wherein the optimized concept set includes: clustering each embedding vector obtained by embedding attribute values ​​for enumerable attributes, and forming concept types according to clustering categories;

[0014] The construction unit is configured to construct at least one concept tree of the knowledge graph by optimizing the concept set, and the knowledge semantic framework SPG model composed of each concept tree is used to construct the knowledge graph.

[0015] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0016] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method of the first aspect is implemented.

[0017] Through the method and apparatus provided in the embodiments of this specification, in the data analysis business of constructing a knowledge graph using a data table, based on the entity relationship diagram that describes the data table structure related to the business objectives of the current knowledge graph construction, attribute information can be extracted according to the entity relationship diagram, and the attribute values ​​can be embedded to determine the corresponding embedding vectors. Then, for the attribute values ​​of the enumerable attributes, the concept types are optimized using the respective embedding vectors to obtain an optimized concept set. At least one concept tree of the knowledge graph is constructed by the optimized concept set. The knowledge semantic framework SPG model composed of the respective concept trees can assist in the construction of the knowledge graph and data analysis based on the knowledge graph. Since the concepts in the data table are classified and optimized through clustering of the embedding vectors, the accumulated knowledge in the data table can be fully utilized to improve the effectiveness of the constructed knowledge graph, which is more conducive to accurate data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of the implementation architecture under the technical concept of this specification;

[0020] Figure 2 is a flowchart of data processing according to one embodiment of this specification;

[0021] Figure 3 is a schematic diagram of an attribute clustering architecture according to a specific example of this specification;

[0022] Figure 4 is a schematic block diagram of a data processing apparatus according to an embodiment of the present specification. DETAILED DESCRIPTION

[0023] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0024] First, several conceptual terms that may be involved in this manual are introduced.

[0025] ER model: Entity Relationship Diagram (ERD), also known as ER diagram, entity relationship model, entity relationship pattern diagram, etc., is a structural diagram used in database design. An ER model can contain different symbols and connectors to show the main entities within the system and the relationships between these entities. For example, a data table is a student performance table, which contains entities such as Student ID, Subject 1, Subject 2, etc. Data is typically recorded using the student ID as a tag, and Subject 1, Subject 2, etc. can be used as attributes, with the specific product values ​​under the corresponding subjects being the attribute values.

[0026] Conceptual modeling: Conceptual modeling is the first level of abstraction from the real world to the information world. It organizes concrete real-world entities through abstraction, forming an information structure that directly guides subsequent logical and physical models. This user-oriented, objective-world model uses abstract concepts to abstractly represent business operations, following the structure of the data model. Conceptual models typically define business objects and business activities. For example, a "business object" in a conceptual model might be: "Related entity objects or meaningful virtual objects involved in business operations, which generally guide the creation of dimension logical tables and the granularity of metric statistics (common examples include users, products, and locations)." A "business activity" might be: "Actions and events that occur in entities or systems during business operations, which generally guide the creation of fact logical tables (common examples include order placement and payment)."

[0027] SPG model: A semantic representation framework based on attribute graphs that can assist in generating knowledge graphs. The SPG model clearly defines the formal representation and programmable framework of "knowledge", making knowledge definable, programmable, and machine-understandable and processable. The SPG model typically defines entity types (Entity Type) and event types (Event Type). The SPG model can achieve compatible progression between knowledge levels, support the construction and continuous iterative evolution of graphs of incomplete data states in industrial-level scenarios, and connect big data and AI technology systems, supporting efficient knowledge-based conversion of massive data, helping to improve data value and application value. The SPG model can contain one or more concept trees.

[0028] Entity type: used to define the metadata information of an entity. For example, an enterprise is an entity category, and the corresponding attribute information such as enterprise name, legal representative, and registered place can be used as metadata information.

[0029] Relationship type: Indicates the type of association between two entities. For example, the connection between "enterprise" and "enterprise" can have association types such as "holding" and "business cooperation".

[0030] Meta-concept (Concept Type): used to describe the classification of entities. It is information at the Schema metadata level. For example, an enterprise is an entity type, and the enterprise type is a meta-concept used to create a classification system for enterprises.

[0031] Event type: describes an event-like entity with temporal characteristics. It is a derivative type of entity type. Event types usually have four major attributes: time, space, subject, object, and other attributes that describe the event. For example, a company listing event is an event type.

[0032] Embedding vectors: Embedding is the process of converting raw data from non-numerical form into numerical vectors. The purpose is to better capture potential semantic relationships and patterns, adapt them to the input requirements of deep learning models, and thus improve the performance and generalization capabilities of the models. Embedding vectors are the result of the embedding process.

[0033] Vector indexing: Vector indexing is a data structure and algorithmic technique in computer science and information retrieval that uses vectors as indices to enable fast and efficient searches across large vector datasets in high-dimensional spaces. In fields such as image recognition, speech recognition, word embedding search in natural language processing (NLP), and recommender systems, it is often necessary to process floating-point vectors spanning tens of thousands or even thousands of dimensions and to find the vector with the highest similarity to the query vector. Vector indexing can rapidly perform these searches.

[0034] Large Model: A Large Language Model (LLM), also known as a large language model, is a pre-trained AI model designed to understand and generate human language. Large models are often trained on large text corpora and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. LLMs are typically large in scale, containing billions of parameters, and can learn complex patterns in language data. These models are often based on deep learning architectures such as transformers, which help them achieve good results on various natural language processing (NLP) tasks.

[0035] The application scenario provided in this specification is, for example, a big data analysis scenario. It performs business analysis through a large number of data tables. In conventional technical solutions, the ER diagram can be converted into a knowledge graph, usually a conversion of entities and the connection relationships between entities. However, some common sense information may also be accumulated in the massive data of the ER diagram. This common sense information may have an impact on the construction of the knowledge graph and may improve the accuracy of the data analysis results. To this end, this specification provides a data processing method that can mine this common sense information by generating a conceptual model, thereby providing more reliable data support for data analysis and improving the effectiveness of data analysis.

[0036] Figure 1 The technical architecture diagram of this specification is shown in FIG. Figure 1 In the illustration, this specification can construct a knowledge graph through a computing platform in a big data analysis scenario, and use the displayability and scalability of the knowledge graph to analyze relevant data.

[0037] like Figure 1 As shown, the computing platform can first obtain an ER diagram describing the data table structure. For example, database management tools such as MySQL Workbench, SQL Server Management Studio (SSMS), and Oracle SQL Developer provide intuitive graphical interfaces that can easily generate ER diagrams. ER diagrams can be used to define the data structure within a data table to generate the data table. The data tables here can be various data tables within the corresponding business, such as user tables, teacher tables, and grades tables. Taking the grades table as an example, the ER diagram can define attribute columns such as key values ​​(such as student ID), subjects, and so on. The key values ​​and various attributes can be understood as the entities involved in the data table. When constructing the data table.

[0038] ER diagrams can define the various entities involved in the data table, as well as the relationships between entities. The data IDs corresponding to each piece of data in the data table (such as user ID, enterprise unique code, etc.), the attributes corresponding to each attribute column, etc., can all be used as entities, and the relationships between entities can be determined by the logical relationships between attributes. As a specific example, if the entity "Enterprise A" corresponds to Region B in the region attribute column, then both Enterprise A and Region B can be used as entities, and their relationship can be: Enterprise A "is located in" Region B. It can be understood that in some embodiments, there can be a hierarchical relationship between different entities in the ER diagram, for example, Region B (one of the entities) "belongs to" City D, Province C (another entity), and so on.

[0039] It can be understood that in order to generate an SPG model, entity types (Entity Type) and event types (Event Type) can be defined. For the entities in the ER diagram, they can be manually screened or summarized through a large language model, so as to classify each entity and obtain multiple entity types, such as enterprise type, user type, merchant type, product type, etc. A single entity type can correspond to at least one entity in the ER diagram, and an event type can correspond to an association type in the ER diagram, such as subordinate to, located in, and other association relationships. In addition, the SPG model can also include concept types for conceptual-level abstraction of entity types. For example, entity types can include occupation types, and concept types can include types such as teachers and doctors. Concept types are knowledge types that integrate more common sense.

[0040] In order to mine concept types, data in the corresponding data table can be obtained based on the ER diagram. Considering that the data in the data table may be massive (such as hundreds of millions of entries), in some embodiments, distribution calculation, sampling and other processing can be performed on the data obtained from the data table. Distribution calculation can be a summary calculation of the distribution of attribute values. For example, the distribution calculation of the enterprise size attribute includes the number of enterprises in individual segments such as less than 20 people, 20 to 50 people, 50 to 100 people, and more than 100 people. Sampling can be random sampling, or sampling can be performed according to the data distribution, for example, the same number of entities are sampled in each statistical segment.

[0041] Based on the obtained attribute values, the concept types in the concept tree can be optimized. On the one hand, the obtained attribute values ​​can be matched with known concept types. When a known concept type is matched, it is stored corresponding to the known concept type. When a known concept type is not matched, a new concept type is created. Under the technical concept of this specification, the determination of the concept type can be performed only for enumerable attributes in the data table. For non-enumerable attributes, its data type (such as integer int, floating point float, etc.) can be determined.

[0042] To create new concept types, attribute embedding can be performed on the acquired data. This involves representing the textual information (such as geographic location) corresponding to each attribute value as a vector. Attribute embedding can be implemented using various text semantic embedding models, such as one-hot representation, word2Vec, and Bert. Attribute embedding allows attribute values ​​to be represented as vectors, such as embedded vectors.

[0043] Furthermore, attribute values ​​can be clustered by comparing vector similarities. Because embedding vectors are semantic representations of attribute values, attribute values ​​can be clustered by calculating their vector similarity. The higher the similarity of the embedding vectors, the more likely the corresponding attribute values ​​are to belong to the same cluster category. During the attribute clustering process, all attribute values ​​in the ER model can be traversed.

[0044] It is understandable that there may be semantic duplication of values ​​between attribute values ​​corresponding to different entities, such as class and grade. Therefore, in a possible implementation, the attribute name (such as the field corresponding to the data column in the data table) and the attribute value can be spliced ​​together to embed the semantics of the attribute value. In this way, after clustering, similar attribute columns are clustered together, and attribute values ​​with large differences in the same attribute column may be clustered into different categories. For example, the province, city, and county attribute columns can be clustered into regional attributes, the primary school and middle school in the school attribute column may be clustered into two different categories, and so on.

[0045] The clustering process can be based on K-means, for example, where K embedding vectors are selected as initial centers and clustered iteratively based on the similarity of the embedding vectors. The mean of the embedding vectors that are clustered into the same category in each iteration is used as the new cluster center. The above process is repeated until clustering is completed. In the attribute clustering process, all attributes in the ER model can be clustered.

[0046] Based on the clustering results, concepts in the conceptual model can be modified, or new concepts can be generated for each cluster category. This process of creating new concepts can be performed manually or through the large model. When using the large model to determine new concepts, the embedding vectors corresponding to the attribute values ​​in the cluster category, along with the corresponding attribute names, can be input into the large model as prompts. The large model then references the corresponding attribute names to generate new concept types. Optionally, the attribute types generated by the large model can be manually modified to better align with business practices.

[0047] New concept types and known concept types can be combined to form an optimized concept set to generate the concept tree in the SPG model. Furthermore, the schema information of the knowledge graph can be determined based on the concept set. It can be understood that schema information is used to define the relevant information about entities and relationships in the knowledge graph and can be used to generate the knowledge graph.

[0048] Figure 2 The data processing flow according to one embodiment of this specification is shown. The execution subject of this flow can be a computer, device, or server with certain computing capabilities, such as Figure 1 The computing platform in Figure 2 As shown, the data processing flow according to one embodiment includes the following steps: Step 201, obtaining an entity relationship diagram, which is used to describe the data table structure related to the business objectives of the current knowledge graph construction; Step 202, extracting attribute information based on the entity relationship diagram; Step 203, using the obtained attribute information to perform concept extraction to obtain an optimized concept set, wherein the optimized concept set includes: clustering each embedding vector obtained by embedding the attribute value for enumerable attributes, and forming concept types according to clustering categories; Step 204, constructing at least one concept tree of the knowledge graph through the optimized concept set, and the knowledge semantic framework SPG model composed of each concept tree is used to construct the knowledge graph.

[0049] First, in step 201, an entity relationship diagram is obtained. Here, the entity relationship diagram is used to describe the data table structure related to the business objectives of the current knowledge graph construction. The data table can be various data tables related to the current business, such as user tables, merchant tables, product tables, etc. The entity relationship diagram is an ER diagram. The entity relationship diagram can include various entities involved in the data table, as well as the association relationships between entities. For example, the data ID corresponding to each piece of data in the data table (such as user ID, enterprise unique code, etc.), the attributes corresponding to each attribute column, etc., can all be entities defined in the ER diagram. The association relationship between entities can describe the connection relationship between the attribute values ​​on the attribute columns of the data table.

[0050] Then, in step 202, attribute information is extracted according to the entity relationship diagram.

[0051] The attribute information here can include information such as attribute values. In order to generate a concept and entity model (i.e., an SPG model), entity types (Entity Type) and event types (Event Type) can be defined, corresponding to the entity types and association relationship types in the ER diagram, respectively. The connection relationship types in the ER diagram can correspond to the event types in the SPG model and the association relationship types in the ER diagram. In addition, the SPG model can also include concept types in the concept tree. Concept types can be common sense information extracted from entity types.

[0052] Entities in ER diagrams can be manually filtered or aggregated through large models to categorize each entity into multiple entity types, such as enterprise type, user type, merchant type, product type, and so on. A single entity type can correspond to at least one entity in the ER diagram, and an event type can correspond to an association type in the ER diagram, such as "belongs to" or "is located in."

[0053] Next, in step 203, the acquired attribute information is used to perform concept extraction to obtain an optimized concept set.

[0054] Under the technical concept of this specification, the optimized concept set may include known concept types, and may also include optimized concept types determined after mining attribute values ​​and determining deeper associations. Specifically, the known concept types may be concept types in a known concept tree, or they may be concept types that have been obtained through other reasonable means (such as manual definition, etc.). The optimized concept types may be new concept types generated by utilizing the semantic relationships between attribute values.

[0055] To explore the semantic relationships between attribute values, attribute values ​​can be semantically embedded. Through semantic embedding, attribute values ​​can be vectorized, such as by representing them as embedding vectors. Attribute embedding can be implemented using various text semantic embedding models, such as one-hot representation, word2Vec, and Bert. Furthermore, attributes can be clustered by comparing vector similarities. Because embedding vectors are semantic representations of attribute values, attribute values ​​can be clustered by calculating vector similarity between embedding vectors. The higher the similarity of the embedding vectors, the more likely the corresponding attribute values ​​belong to the same cluster category.

[0056] In one embodiment, to generate new concept types, attribute values ​​can be subjected to distribution calculations, sampling, and other processing. Distribution calculations can be aggregated calculations of the distribution of attribute values. For example, the distribution calculation of the enterprise size attribute can include statistics on the number of enterprises with fewer than 20 employees, 20 to 50 employees, 50 to 100 employees, and more than 100 employees. Sampling can be random or based on data distribution, for example, sampling the same number of entities in each statistical segment. Attribute embedding can be performed on the sampled samples, where each attribute value is represented by a vector.

[0057] During attribute clustering, all attribute values ​​in the ER model can be traversed. In one possible implementation, attribute names (such as the fields corresponding to data columns in a data table) and attribute values ​​can be spliced ​​together to embed the attribute values. In this way, after clustering, similar attribute columns are clustered together, and attribute values ​​with significant differences in the same attribute column may be clustered into different categories. For example, the province, city, and county attribute columns can be clustered into regional attributes, and the primary and secondary schools in the school attribute column may be clustered into two different categories, and so on.

[0058] The clustering process can be performed using methods such as K-means, DBSCAN, and GMM, which are not limited here. The types of entities after clustering will usually increase or decrease accordingly.

[0059] Figure 3 A specific process diagram of attribute clustering is shown in Figure 2. Figure 3 As shown, all attributes corresponding to a data table can be obtained from the ER diagram. Attributes can be divided into enumerable and non-enumerable attributes. Non-enumerable attributes are generally difficult to enumerate one by one, such as consumer expenditure attributes, which are usually non-enumerable. For non-enumerable attributes, regular matching can be used to match each attribute value to a predetermined computer data type, such as integer or floating point. Regular matching is a matching method that defines a matching pattern, such as a matching pattern that defines a fixed vocabulary of beginning and ending words. Enumerable attributes are generally attributes whose attribute values ​​fall within a limited range, such as regional attributes such as provinces, cities, and counties. For enumerable attributes, a check is performed to see if they can be matched with existing concepts (i.e., known concept types), that is, whether they can be attached to existing concepts. If so, the concept value of the known concept type is recalled and stored in conjunction with the embedding vector. Otherwise, if no match is found with an existing concept, a new concept type can be created and an embedding vector for the corresponding attribute value is determined. The resulting embedding vector is stored in conjunction with the new concept type. The newly created concept type can be mined from the attributes and their description information (attribute metadata) using a large model. The mining method is, for example, to input some existing concept types and attributes and their description information as prompt words into the large model. Optionally, the newly created concept types output by the large model can also be manually modified.

[0060] Attribute clusters can be determined for the stored concept values ​​and corresponding embedding vectors. Figure 3 As shown, attribute clustering can include clustering of similar attributes, such as provinces, cities, and counties can be clustered into regional attributes, and attribute value clustering, such as similar attribute values ​​in consumer expenditure attributes are clustered into corresponding consumer expenditure gear attributes, etc. According to an optional embodiment, during the clustering process, the concept values ​​corresponding to a predetermined number (e.g., N) of the top embedding vector similarities ranked from high to low can be regarded as the current same cluster category.

[0061] After clustering, some concept categories can be merged or some new concept categories can be added to optimize the concepts in the SPG concept tree.

[0062] Furthermore, in step 204, entity attributes of the knowledge graph are extracted from the optimized concept and entity models through the big model to assist in the construction of the knowledge graph and data analysis based on the knowledge graph.

[0063] It can be understood that the optimized concept types can be used to construct the concept tree in the SPG model, thereby assisting in the generation of the knowledge graph. Clustering is performed on the embedding vectors obtained through vector embedding, which makes full use of the common sense knowledge accumulated in the attributes and attribute values ​​in the data table, and can provide more accurate concept segmentation. The entity attributes of the knowledge graph can be extracted from the optimized SPG model to construct the knowledge graph schema. Generally, the knowledge graph schema can define entity types and entity attributes, connection relationship types and connection relationship attributes. The attributes in the entity type can be selected from the concept type. In order to extract the entity attributes of the knowledge graph through the concept and entity model, a small number of knowledge graph entity attributes can be provided to the large model as samples, and the concept and entity model can be used as the input of the large model, so that the large model can extract the entity type and entity attributes from them.

[0064] The connection relationship types in the SPG model can also be generated based on the ER model. Specifically, the fields and field descriptions of each data table can be input as prompts into the large model, which then outputs the foreign key associations between tables, that is, the association relationships. For example, the subject table can describe student exam subjects, such as using the number 1 as the subject ID to describe the Chinese subject, and the number 2 as the subject ID to describe the mathematics subject. The student table has a subject field, and the attribute value of this subject field corresponds to the subject ID of the subject table, which can be used to discover the association relationship between students and subjects.

[0065] In this way, the knowledge graph Schema can be constructed through the obtained entity types, concept types, and connection relationship types, thereby constructing a knowledge graph for corresponding data analysis business processing.

[0066] Looking back at the above process, in the data analysis business using data tables, after obtaining the entity relationship diagram that describes the data table structure, attribute information in the entity relationship diagram can be extracted, and concept extraction can be performed based on the extracted attribute information to obtain an optimized concept set, which is used to optimize the concept tree in the SPG model used to construct the knowledge graph. The optimized concept set includes new concept types formed by clustering the various embedding vectors obtained by embedding the attribute values ​​for enumerable attributes. The optimized concept types formed by clustering are used to classify the concepts in the data table through clustering, so that the accumulated knowledge in the data table can be fully utilized, the effectiveness of the constructed knowledge graph can be improved, and it is more conducive to accurate data analysis.

[0067] According to another embodiment, a data processing device is also provided, which can be installed in any computer, device, or server with a certain computing capability. Figure 4 FIG. 1 shows a structural block diagram of a data processing device according to an embodiment. Figure 4 As shown, the data processing device 400 may include:

[0068] An acquisition unit 41 is configured to acquire an entity relationship diagram, where the entity relationship diagram is used to describe a data table structure related to a business objective of the current knowledge graph construction;

[0069] an embedding unit 42 configured to embed the attribute values ​​and determine corresponding embedding vectors;

[0070] The optimization unit 43 is configured to optimize the attributes using the respective embedding vectors to obtain an optimized concept set, wherein the optimized concept set includes: clustering the respective embedding vectors obtained by embedding the attribute values ​​of the enumerable attributes, and forming concept types according to the clustering categories;

[0071] The construction unit 44 is configured to construct at least one concept tree of the knowledge graph by optimizing the concept set, and the knowledge semantic framework SPG model composed of each concept tree is used to construct the knowledge graph.

[0072] It is worth mentioning that Figure 4 The device 400 is shown with Figure 2 The illustrated method embodiment corresponds, therefore, to Figure 2 The corresponding description in the method embodiment also applies to Figure 4 The device 400 shown is not described in detail here.

[0073] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described by et al.

[0074] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the system realizes the combination of Figure 2 The method described by et al.

[0075] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the embodiments of this specification may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0076] The specific implementation methods described above further explain in detail the purpose, technical solutions and beneficial effects of the technical concept of this specification. It should be understood that the above description is only a specific implementation method of the technical concept of this specification and is not intended to limit the scope of protection of the technical concept of this specification. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of this specification should be included in the scope of protection of the technical concept of this specification.

Claims

1. A data processing method for data processing in the process of constructing a knowledge graph based on a business data table, the method comprising: Obtain an entity relationship diagram, where the entity relationship diagram is used to describe a data table structure related to a business objective of the current knowledge graph construction; Extract attribute information based on entity relationship diagram; Using the acquired attribute information to perform concept extraction, an optimized concept set is obtained, wherein the optimized concept set includes: clustering each embedding vector obtained by embedding attribute values ​​for enumerable attributes, and forming concept types according to clustering categories; At least one concept tree of the knowledge graph is constructed by optimizing the concept set, and the knowledge semantic framework SPG model composed of each concept tree is used to construct the knowledge graph.

2. The method of claim 1, wherein: A single attribute value corresponds to one or more data in the data table. An enumerable attribute is an attribute whose attribute value can be exhaustively enumerated. The optimization of attributes by using each embedding vector to obtain the optimized concept set includes: For enumerable attributes, each attribute value is compared with the known concept type. If there is a match, the corresponding attribute value is attributed to the known concept type. For attribute values ​​in enumerable attributes that do not match known concept types, the corresponding embedding vectors are used to perform attribute clustering and determine new concept types for each cluster category.

3. The method of claim 2, wherein: The attribute clustering using the corresponding embedding vectors includes several iterations of the following clustering steps: A predetermined number of concept values ​​ranked top in descending order of embedding vector similarity are taken as the current same clustering categories.

4. The method of claim 2, wherein: Determining new concept types for each cluster category includes: The embedding vectors corresponding to each cluster category and the predetermined concept type are input into the large language model as prompt information, and the new concept type is determined according to the output result of the large language model.

5. The method of claim 1, wherein: The extracting of attribute information according to the entity relationship diagram includes: Summarize the attribute value distribution of the attributes in the entity relationship diagram; The data in the data table is sampled based on the attribute value distribution summary to extract attribute information.

6. A data processing device for data processing in the process of building a knowledge graph based on a business data table, the device comprising: An acquisition unit is configured to acquire an entity relationship diagram, wherein the entity relationship diagram is used to describe a data table structure related to a business objective constructed by a current knowledge graph; An embedding unit configured to embed the attribute values ​​and determine corresponding embedding vectors; The optimization unit is configured to optimize the attributes by using each embedding vector to obtain an optimized concept set, wherein the optimized concept set includes: clustering each embedding vector obtained by embedding the attribute value for the enumerable attribute, and forming a concept type according to the clustering category; The construction unit is configured to construct at least one concept tree of the knowledge graph by optimizing the concept set, and the knowledge semantic framework SPG model composed of each concept tree is used to construct the knowledge graph.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.

8. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 5 is implemented.

Images