Data processing methods and apparatus

Abstract

This specification provides a data processing method and apparatus. In the process of constructing a knowledge graph using a data table, matching concept and entity models can be generated based on the entity relationship graph of the data table. Furthermore, the concept value set and mapping relationship formed by clustering the attributes extracted from the entity relationship graph can be used to optimize the generated SPG model. Entity attributes of the knowledge graph are extracted from the optimized SPG model using a large model to assist in the construction of the knowledge graph and data analysis based on the knowledge graph. This implementation method can facilitate more accurate data analysis.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to methods and apparatus for data processing. Background Technology

[0002] Big data analytics typically relies on large datasets and often utilizes SQL for data processing. However, this approach is inefficient for complex data analysis scenarios. Knowledge graphs, as structured semantic knowledge bases, can describe concepts and their relationships in the physical world using symbolic representations. Based on a graph structure, knowledge graphs, through visualization and expert rule expression, can continuously delve deeper into the fluctuations of various indicators and their underlying causes.

[0003] The basic building blocks of a knowledge graph are typically "entity-relationship-entity" triples, along with entity-value pairs containing their associated attributes. Entities are interconnected through relations, forming a network-like knowledge structure. An entity (which can serve as a node in a knowledge graph) can be any thing in the real world, such as a person, a payment device, or a company, or an abstract concept like artificial intelligence. Relationships are represented by edges in the knowledge graph, describing the associations between entities, such as investment relationships or payment relationships. In specific business domains, knowledge graphs can be built based on massive amounts of data. However, for technical personnel in the big data field, they may be familiar with the ER model of data but lack prior knowledge of the SPG model corresponding to knowledge graphs and the terminology used in knowledge graphs. Therefore, providing an automated method for constructing a knowledge graph system from a data system primarily based on data tables is a crucial technological bridge to lower the barrier to entry for using knowledge graphs and improve the efficiency of big data analytics. Summary of the Invention

[0004] This specification describes one or more embodiments of a data preprocessing method and apparatus for constructing knowledge graphs, which is used to solve one or more problems mentioned in the background art.

[0005] According to the first aspect, a data processing method is provided for data processing in the process of constructing a knowledge graph based on business data tables. The method includes: obtaining an entity relationship graph, which describes the data table structure related to the business objectives of the current knowledge graph construction; extracting attribute information from the entity relationship graph and embedding attribute values ​​to determine corresponding embedding vectors; optimizing attributes using the embedding vectors to obtain an optimized concept set, wherein the optimized concept set includes: clustering enumerable attributes based on the embedding vectors obtained from embedding attribute values, and forming concept types based on the clustering categories; constructing at least one concept tree of the knowledge graph through the optimized concept set, wherein the knowledge semantic framework (SPG) model composed of the concept trees is used for the construction of the knowledge graph.

[0006] In one embodiment, a single attribute value corresponds to one or more data entries in a data table, and an enumerable attribute is an attribute whose attribute value can be exhaustively enumerated. The step of optimizing the attribute using each embedding vector to obtain an optimized concept set includes: for enumerable attributes, comparing each attribute value with a known concept type, and assigning the corresponding attribute value to a known concept type if they match; for attribute values ​​in the enumerable attributes that do not match a known concept type, performing attribute clustering using the corresponding embedding vector, and determining a new concept type for each cluster category.

[0007] In a further embodiment, the attribute clustering using the corresponding embedding vectors includes several iterations of the following clustering steps: taking a predetermined number of concept values ​​whose embedding vector similarity ranks highest to lowest as the current same cluster category.

[0008] In another further embodiment, determining new concept types for each cluster category includes: inputting the embedding vectors corresponding to each cluster category and the predetermined concept type as prompt information into the large language model, and determining new concept types based on the output of the large language model.

[0009] In one embodiment, extracting attribute information from the entity relationship diagram includes: summarizing the attribute value distribution of the attributes in the entity relationship diagram; and sampling the data in the data table based on the summarized attribute value distribution to extract attribute information.

[0010] According to a second aspect, a data processing apparatus is provided for data processing in the process of constructing a knowledge graph based on a business data table, the apparatus comprising:

[0011] The acquisition unit is configured to acquire an entity relationship graph, which is used to describe the data table structure related to the business objectives of the current knowledge graph construction.

[0012] The embedding unit is configured to embed attribute values ​​and determine the corresponding embedding vectors.

[0013] The optimization unit is configured to perform attribute optimization using each embedding vector to obtain an optimized concept set, wherein the optimized concept set includes: clustering enumerable attributes based on each embedding vector obtained by embedding attribute values, and forming concept types according to the clustering categories;

[0014] The building unit is configured to construct at least one concept tree of the knowledge graph by optimizing the concept set, and the knowledge semantic framework SPG model composed of the concept trees is used for the construction of the knowledge graph.

[0015] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0016] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect.

[0017] The methods and apparatus provided in the embodiments of this specification enable data analysis operations that utilize data tables to construct knowledge graphs. Based on an entity relationship diagram describing the structure of the data table related to the business objectives of the current knowledge graph construction, attribute information is extracted from the entity relationship diagram, and attribute values ​​are embedded to determine corresponding embedding vectors. Then, for enumerable attributes, the concept types are optimized using the embedding vectors to obtain an optimized concept set. At least one concept tree of the knowledge graph is constructed using this optimized concept set. The knowledge semantic framework (SPG) model, composed of these concept trees, assists in the construction of the knowledge graph and data analysis based on it. Because the concepts in the data table are categorized and optimized through clustering of embedding vectors, the accumulated knowledge in the data table can be fully utilized, improving the effectiveness of the constructed knowledge graph and thus facilitating more accurate data analysis. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the implementation architecture based on the technical concept of this manual;

[0020] Figure 2 This is a schematic diagram of a data processing flow according to one embodiment of this specification;

[0021] Figure 3 This is a schematic diagram of an attribute clustering architecture based on a specific example of this specification.

[0022] Figure 4 This is a schematic block diagram of a data processing apparatus according to one embodiment of this specification. Detailed Implementation

[0023] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0024] First, let's introduce some concepts and terms that may be involved in this manual.

[0025] ER Model: Also known as Entity-Relationship Diagram, Entity-Relationship Schema Diagram, etc., it is a structural diagram used for database design. An ER model can contain different symbols and connectors to display the main entities within a system and the relationships between them. For example, a data table might be a student grade table, containing entities such as Student ID, Subject 1, Subject 2, etc. Data is typically recorded using Student ID as the identifier, Subject 1, Subject 2, etc., can be used as attributes, and the specific product values ​​under the corresponding subjects are the attribute values.

[0026] Conceptual Model: Conceptual modeling is the first layer of abstraction from the real world to the information world. It organizes concrete things in the real world through abstraction, forming an information structure that can directly guide subsequent logical and physical models. It is a user-oriented and objective-world-oriented model that uses abstract conceptual terms to abstractly express business operations, following the structure of a data model. A conceptual model typically defines business objects and business activities. For example, a "business object" defined in a conceptual model can be: "Relevant entity objects or virtual objects with historical significance involved in business operations, generally used to guide the creation of dimension logical tables and the granularity of indicator statistics (common examples include users, products, and locations)"; while a "business activity" can be: "Actions and events that occur in entities or systems during business operations, generally used to guide the creation of fact logical tables (common examples include placing orders and making payments)."

[0027] The SPG model is a semantic representation framework based on attribute graphs that can assist in the generation of knowledge graphs. The SPG model explicitly defines the formal representation and programmable framework of "knowledge," making knowledge definable, programmable, and machine-understandable and processable. The SPG model typically defines entity types and event types. It enables compatibility and progression between knowledge levels, supports the construction and continuous iterative evolution of knowledge graphs in incomplete data states in industrial scenarios, and can connect big data and AI technology systems, supporting efficient knowledge transformation of massive amounts of data and helping to improve data and application value. An SPG model can contain one or more concept trees.

[0028] Entity type: Used to define the metadata information of an entity. For example, an enterprise is an entity category, and its metadata information can be attributed to the enterprise name, legal representative, registered location, etc.

[0029] Relation type: Represents the type of association between two entities. For example, the connection between "enterprise" and "enterprise" can be related to "holding" or "business cooperation".

[0030] Meta-concept (Concept Type): Used to describe the classification of entities. It is information at the schema metadata level. For example, enterprise is an entity type, and enterprise type is a meta-concept used to create a classification system for enterprises.

[0031] Event type: Describes event-type entities with temporal characteristics. It is a derivative type of entity type. Event types usually have four main attributes: time, space, subject, and object, as well as other attributes that describe the event. For example, a company's listing event is an event type.

[0032] Embedding: Embedding is the process of transforming raw data from a non-numerical form into a numerical vector. The goal is to better capture underlying semantic relationships and patterns, adapting them to the input requirements of deep learning models, thereby improving the model's performance and generalization ability. Embedding vectors are the result of this process.

[0033] Vector indexing is a data structure and algorithm technique in computer science and information retrieval that uses vectors as indexes to achieve fast and efficient searches on large vector datasets in high-dimensional space. In fields such as image recognition, speech recognition, word embedding search in natural language processing (NLP), and recommender systems, it is often necessary to process floating-point vectors with thousands or even more dimensions and find the vector with the highest similarity to the query vector. Vector indexing can quickly achieve this query process.

[0034] Large Language Models (LLMs) are pre-trained artificial intelligence models designed to understand and generate human language. They are typically trained on massive amounts of text corpora and can perform a wide range of tasks, including text summarization, translation, and sentiment analysis. LLMs are usually enormous, containing billions of parameters, and can learn complex patterns in language data. These models are often based on deep learning architectures, such as transformers, which helps them achieve better results on various Natural Language Processing (NLP) tasks.

[0035] The application scenarios provided in this specification include, for example, big data analytics. These scenarios involve business analysis using large amounts of data tables. Conventional technical solutions convert ER diagrams into knowledge graphs, typically involving the transformation of entities and their inter-entity relationships. However, the massive amounts of data within ER diagrams may also accumulate common-sense information. This common-sense information can influence the construction of the knowledge graph and potentially improve the accuracy of data analysis results. Therefore, this specification provides a data processing method that, by generating a conceptual model, can mine this common-sense information, thereby providing more reliable data support for data analysis and improving its effectiveness.

[0036] Figure 1 A schematic diagram of the technical architecture of this specification is shown. Figure 1 As illustrated in the diagram, this manual demonstrates that in a big data analytics scenario, a knowledge graph can be constructed via a computing platform, and the displayability and extensibility of the knowledge graph can be used to analyze relevant data.

[0037] like Figure 1 As shown, the computing platform can first obtain an ER diagram describing the data table structure. Database management tools such as MySQL Workbench, SQL Server Management Studio (SSMS), and Oracle SQL Developer provide intuitive graphical interfaces that facilitate the generation of ER diagrams. ER diagrams can be used to define the data structure of a data table to generate the table. These data tables can be various tables within the corresponding business logic, such as user tables, teacher tables, grade tables, etc. Taking a grade table as an example, the ER diagram can define key values ​​(such as student ID), subject, and other attribute columns. The key values ​​and various attributes can be understood as entities involved in the data table, which are then constructed during the data table construction process.

[0038] An ER diagram can define both the various entities involved in a data table and the relationships between them. Each data entry in the table, along with its corresponding data ID (such as a user ID, company unique code, etc.) and attributes in its attribute columns, can be considered an entity. The relationships between entities can be determined through the logical relationships between attributes. For example, if entity "Company A" corresponds to Region B in the region attribute column, then both Company A and Region B can be considered entities, and their relationship could be: Company A is "located" in Region B. It is understood that in some implementations, different entities in the ER diagram can have hierarchical relationships; for example, Region B (one entity) "belongs" to Province C, City D (another entity), and so on.

[0039] To generate an SPG model, it's understandable that entity types and event types can be defined. Entities in the ER diagram can be manually selected or aggregated using large language models to categorize them into multiple entity types, such as enterprise, user, merchant, and product types. A single entity type can correspond to at least one entity in the ER diagram, while event types can correspond to relationship types, such as belonging to or located in. Furthermore, the SPG model can include concept types for conceptual abstraction of entity types. For example, entity types can include occupation types, and concept types can include teacher, doctor, etc. Concept types incorporate more common-sense knowledge.

[0040] To uncover concept types, data from corresponding tables can be obtained based on the ER diagram. Considering that the data tables may contain massive amounts of data (e.g., hundreds of millions of records), in some embodiments, the data obtained from the tables can undergo distribution calculations, sampling, and other processing. Distribution calculations can be summative calculations of the distribution of attribute values. For example, the distribution calculation of the enterprise size attribute includes statistics on the number of enterprises in segments such as less than 20 employees, 20 to 50 employees, 50 to 100 employees, and more than 100 employees. Sampling can use random sampling or sampling according to the data distribution, such as sampling the same number of entities in each statistical segment.

[0041] Based on the obtained attribute values, the concept types in the concept tree can be optimized. On one hand, the obtained attribute values ​​can be matched with known concept types. If a known concept type is matched, the attribute value is stored accordingly; otherwise, a new concept type is created. Under the technical conception described in this specification, the determination of concept types can be performed only on enumerable attributes in the data table. For non-enumerable attributes, their data type (such as integer (int), floating-point type (float), etc.) can be determined.

[0042] To create new concept types, text embedding can be performed on the acquired data for attributes. This involves representing the text information (such as geographic location) corresponding to each attribute value as a vector. Attribute embedding can be implemented using various text semantic embedding models, such as one-hot encoding, word2Vec, and BERT. Through attribute embedding, attribute values ​​can be vectorized, denoted as embedding vectors.

[0043] Furthermore, attribute values ​​can be clustered by comparing vector similarity. Since embedding vectors are semantic representations of attribute values, clustering can be performed by calculating the vector similarity of embedding vectors. The higher the similarity of the embedding vectors, the more likely the corresponding attribute values ​​belong to the same cluster category. During the attribute clustering process, all attribute values ​​in the ER model can be traversed.

[0044] It's understandable that there might be semantic overlap in the attribute values ​​corresponding to different entities, such as "class" and "grade." Therefore, in one possible implementation, attribute names (like fields corresponding to data columns in a data table) and attribute values ​​can be concatenated together to semantically embed the attribute values. In this way, after clustering, similar attribute columns are clustered together, while attribute values ​​that differ significantly within the same attribute column may be clustered into different categories. For example, province, city, and county attribute columns can be clustered as regional attributes, and primary and secondary schools in the school attribute column might be clustered into two different categories, and so on.

[0045] Clustering can be based on methods such as K-means, which involves selecting K embedding vectors as initial centers and iteratively clustering based on the similarity of the embedding vectors. In each iteration, the mean of the embedding vectors that cluster into the same category is used as the new cluster center. This process is repeated until clustering is complete. In attribute clustering, all attributes in the ER model can be clustered.

[0046] Then, based on the clustering results, the concepts in the conceptual model can be modified, or new concepts can be generated for each cluster category. This process of creating new concepts can be done manually or through a large model. When using a large model to determine new concepts, the embedding vectors corresponding to the attribute values ​​in the cluster categories, along with the corresponding attribute names, can be input into the large model as prompts. The large model will then generate new concept types by referring to the corresponding attribute names. Optionally, the attribute types generated by the large model can be manually modified to better align with business practices.

[0047] New concept types and known concept types can be combined to form an optimized concept set, which generates the concept tree in the SPG model. Furthermore, the schema information of the knowledge graph can be determined based on the concept set. Schema information is used to define relevant information about entities and relationships in the knowledge graph and can be used to generate the knowledge graph.

[0048] Figure 2 A data processing flow according to one embodiment of this specification is illustrated. The entity executing this flow can be a computer, device, or server with a certain computing power, for example... Figure 1 The computing platform in [the context]. For example... Figure 2 As shown, the data processing flow according to one embodiment includes the following steps: Step 201, obtaining an entity relationship graph, which describes the data table structure related to the business objectives of the current knowledge graph construction; Step 202, extracting attribute information based on the entity relationship graph; Step 203, performing concept extraction using the obtained attribute information to obtain an optimized concept set, wherein the optimized concept set includes: clustering enumerable attributes based on each embedding vector obtained by embedding attribute values, and forming concept types according to the clustering categories; Step 204, constructing at least one concept tree of the knowledge graph through the optimized concept set, wherein the knowledge semantic framework SPG model composed of each concept tree is used for the construction of the knowledge graph.

[0049] First, in step 201, an entity relationship diagram is obtained. Here, the entity relationship diagram describes the data table structure related to the business objectives of the current knowledge graph construction. The data tables can be various tables related to the current business, such as user tables, merchant tables, product tables, etc. The entity relationship diagram is also known as an ER diagram. It can include various entities involved in the data tables, as well as the relationships between entities. For example, the data IDs (such as user IDs, enterprise unique codes, etc.) corresponding to each piece of data in the data table, and the attributes corresponding to each attribute column, can all be entities defined in the ER diagram. The relationships between entities describe the connections between attribute values ​​in the attribute columns of the data tables.

[0050] Then, in step 202, attribute information is extracted from the entity relationship diagram.

[0051] The attribute information here can include attribute values ​​and other information. To generate the concept and entity model (i.e., the SPG model), entity types and event types can be defined, corresponding to the entity types and relationship types in the ER diagram, respectively. The connection types in the ER diagram can correspond to the event types in the SPG model and the relationship types in the ER diagram. Furthermore, the SPG model can also include concept types from the concept tree. Concept types can be common-sense information extracted from entity types.

[0052] Entities in an ER diagram can be manually filtered or aggregated using large models to categorize them into multiple entity types, such as enterprise type, user type, merchant type, product type, etc. A single entity type can correspond to at least one entity in the ER diagram, and event types can correspond to relationship types in the ER diagram, such as "belonging to" or "located in".

[0053] Next, through step 203, the obtained attribute information is used to extract concepts and obtain an optimized concept set.

[0054] Under the technical concept described in this specification, the optimized concept set can include known concept types, or optimized concept types determined by mining attribute values ​​to identify deeper relationships. Specifically, known concept types can be concept types in a known concept tree, or concept types already obtained through other reasonable methods (such as manual definition). Optimized concept types can be new concept types generated using the semantic relationships between attribute values.

[0055] To uncover the semantic relationships between attribute values, semantic embedding can be performed. Semantic embedding allows attribute values ​​to be vectorized, denoted as embedding vectors. Attribute embedding can be implemented using various text semantic embedding models, such as one-hot encoding, word2Vec, and BERT. Furthermore, attributes can be clustered by comparing vector similarity. Since embedding vectors are semantic representations of attribute values, clustering can be performed by calculating the similarity between embedding vectors. The higher the similarity of the embedding vectors, the more likely the corresponding attribute values ​​belong to the same cluster category.

[0056] In one embodiment, to generate new concept types, attribute values ​​can be processed through distribution calculations, sampling, and other techniques. Distribution calculations can be a summary calculation of the distribution of attribute values; for example, the distribution calculation of the enterprise size attribute includes statistical statistics on the number of enterprises in segments such as fewer than 20 employees, 20 to 50 employees, 50 to 100 employees, and more than 100 employees. Sampling can be random sampling or sampling according to the data distribution, such as sampling the same number of entities in each statistical segment. For the sampled data, attribute embedding can be performed, that is, representing each attribute value as a vector.

[0057] During attribute clustering, all attribute values ​​in the ER model can be traversed. In one possible implementation, attribute names (such as fields corresponding to data columns in a data table) and attribute values ​​can be concatenated together to embed the attribute values. In this way, after clustering, similar attribute columns are clustered together, while attribute values ​​that differ significantly within the same attribute column may be clustered into different categories. For example, province, city, and county attribute columns can be clustered as regional attributes, and primary and secondary schools in the school attribute column may be clustered into two different categories, and so on.

[0058] Clustering can be performed using methods such as K-means, DBSCAN, and GMM, and will not be limited to these methods here. The types of entities after clustering will typically increase or decrease accordingly.

[0059] Figure 3 This illustrates a specific process diagram for attribute clustering. For example... Figure 3 As shown, all attributes corresponding to the data table can be obtained from the ER diagram, and these attributes can be divided into enumerable and non-enumerable attributes. Non-enumerable attributes are usually those that are difficult to list one by one, such as consumption expenditure attributes, which are typically non-enumerable. For non-enumerable attributes, regular expression matching can be used to match each attribute value to a predetermined computer data type, such as integer or floating-point. Regular expression matching is a matching method that defines a matching pattern, such as defining a matching pattern with fixed words at the beginning and end. Enumerable attributes are usually attributes whose values ​​take values ​​within a limited range, such as regional attributes like provinces and cities / counties. For enumerable attributes, it is checked whether they can be matched with existing concepts (i.e., known concept types), that is, whether they can be attached to existing concepts. If so, the concept values ​​of the known concept types are recalled and stored correspondingly with the embedding vectors. Otherwise, if they cannot be matched with existing concepts, a new concept type can be created, and the embedding vectors of the corresponding attribute values ​​can be determined. The resulting embedding vectors are stored correspondingly with the new concept type. New concept types can be mined from attributes and their descriptive information (meta-information of attributes) through a large model. Mining methods include inputting some existing concept types, attributes, and their descriptive information as prompts into a large model. Optionally, the newly created concept types output by the large model can also be manually modified.

[0060] Attribute clustering can be determined based on the stored concept values ​​and corresponding embedding vectors. (See reference) Figure 3 As shown, attribute clustering can include clustering of similar attributes, such as provinces, cities, and counties being clustered as regional attributes, and clustering of attribute values, such as similar attribute values ​​in consumption expenditure being clustered as corresponding consumption expenditure level attributes, and so on. According to an optional embodiment, during the clustering process, the concept values ​​corresponding to the top number (e.g., N) of embedding vector similarity ranked from high to low can be used as the current same cluster category.

[0061] Clustering can merge some concept categories or add some new concept categories, thereby optimizing the concepts in the SPG concept tree.

[0062] Furthermore, in step 204, entity attributes of the knowledge graph are extracted from the optimized concept and entity models using a large model to assist in the construction of the knowledge graph and data analysis based on the knowledge graph.

[0063] It is understandable that optimized concept types can be used to construct the concept tree in the SPG model, thereby assisting in knowledge graph generation. Clustering using embedding vectors obtained through vector embedding fully utilizes the common-sense knowledge accumulated in the attributes and attribute values ​​of the data table, providing more accurate concept segmentation. Entity attributes of the knowledge graph can be extracted from the optimized SPG model to construct the knowledge graph schema. Typically, a knowledge graph schema can define entity types and entity attributes, connection relationship types and connection relationship attributes. Attributes in entity types can be selected from concept types. To extract entity attributes from the knowledge graph through concept and entity models, a small number of knowledge graph entity attributes can be provided as examples to the large model, and the concept and entity models can be used as input to the large model, from which the large model can extract entity types and entity attributes.

[0064] The join types in the SPG model can also be generated based on the ER model. Specifically, the fields and field descriptions of each data table can be input into the larger model as prompts, and the larger model outputs the foreign key relationships between the tables, i.e., the association relationships. For example, the subject table can describe the subjects students take, such as using the value 1 as the subject ID to describe the Chinese subject, the value 2 as the subject ID to describe the Mathematics subject, etc. The student table has a subject field, and the attribute value of this subject field corresponds to the subject ID in the subject table, which allows us to explore the association relationships between students and subjects.

[0065] In this way, a knowledge graph schema can be constructed using the acquired entity types, concept types, and connection relationship types, thereby building a knowledge graph for corresponding data analysis business processing.

[0066] Reviewing the above process, in data analysis using data tables, after obtaining the entity-relationship graph describing the data table structure, attribute information can be extracted from the graph. Based on this extracted attribute information, concept extraction is performed to obtain an optimized concept set, which is used to optimize the concept tree in the SPG model used to build the knowledge graph. The optimized concept set includes new concept types formed by clustering enumerable attributes based on the embedding vectors obtained from attribute value embeddings. By forming optimized concept types through clustering, the concepts in the data table are categorized, thus fully utilizing the accumulated knowledge in the data table, improving the effectiveness of the constructed knowledge graph, and ultimately facilitating more accurate data analysis.

[0067] According to another embodiment, a data processing apparatus is also provided. This data processing apparatus can be located in any computer, device, or server with a certain computing power. Figure 4 A structural block diagram of a data processing apparatus according to one embodiment is shown. Figure 4 As shown, the data processing apparatus 400 may include:

[0068] Acquisition unit 41 is configured to acquire an entity relationship graph, which is used to describe the data table structure related to the business objectives of the current knowledge graph construction.

[0069] Embedding unit 42 is configured to embed attribute values ​​and determine the corresponding embedding vectors;

[0070] The optimization unit 43 is configured to perform attribute optimization using each embedding vector to obtain an optimized concept set, wherein the optimized concept set includes: clustering of enumerable attributes based on each embedding vector obtained by embedding attribute values, and concept types formed according to the clustering categories;

[0071] Construction unit 44 is configured to construct at least one concept tree of a knowledge graph by optimizing a concept set, and the knowledge semantic framework SPG model composed of the concept trees is used for the construction of the knowledge graph.

[0072] It is worth noting that, Figure 4 The device 400 shown is Figure 2 The method embodiment shown corresponds to this, therefore, Figure 2 The corresponding descriptions in the method embodiments also apply to Figure 4 The device 400 shown will not be described in detail here.

[0073] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The methods described above.

[0074] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The methods described above.

[0075] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0076] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the technical concept in this specification. It should be understood that the above description is only a specific embodiment of the technical concept in this specification and is not intended to limit the scope of protection of the technical concept in this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments in this specification should be included within the scope of protection of the technical concept in this specification.

Claims

1. A data processing method for data processing in a process of constructing a knowledge graph based on a business data table, the method comprising: obtaining an entity relationship graph, the entity relationship graph being used to describe a data table structure related to a business target of current knowledge graph construction; extracting attribute information according to the entity relationship graph; performing concept extraction using the obtained attribute information to obtain an optimized concept set, wherein the optimized concept set comprises known concept types and optimized concept types, and the optimized concept types comprise clustering each embedding vector obtained by embedding an attribute value in an enumerable attribute that does not match a known concept type, and determining a new concept type according to a clustering category; and constructing at least one concept tree of the knowledge graph through the optimized concept set, and a knowledge semantic framework SPG model formed by the concept trees being used for construction of the knowledge graph. A single attribute value corresponds to a single or multiple data in a data table, and an enumerable attribute is an attribute whose attribute values can be enumerated. The attribute optimization using each embedding vector to obtain the optimized concept set comprises: for an enumerable attribute, comparing each attribute value with a known concept type, and in a matching case, attributing the corresponding attribute value to the known concept type. The attribute clustering using the corresponding embedding vector comprises several iterations of the following clustering steps: A predetermined number of concept values ranked from high to low in embedding vector similarity are taken as a same clustering category.

2. The method of claim 1, wherein, The new concept type is determined by: inputting embedding vectors corresponding to each clustering category and a predetermined concept type as prompt information into a large language model, and determining a new concept type according to an output result of the large language model. The attribute information extraction according to the entity relationship graph comprises: Attribute value distribution summarization is performed on attributes in the entity relationship graph; and 3. The method of claim 2, wherein, Sampling is performed on data in the data table according to the attribute value distribution summarization to extract attribute information.

6. A data processing apparatus for data processing in a process of constructing a knowledge graph based on a business data table, the apparatus comprising:

4. The method of claim 2, wherein, an obtaining unit configured to obtain an entity relationship graph, the entity relationship graph being used to describe a data table structure related to a business target of current knowledge graph construction; an embedding unit configured to embed attribute values to determine corresponding embedding vectors; 5. The method of claim 1, wherein, an optimization unit configured to perform attribute optimization using each embedding vector to obtain an optimized concept set, wherein the optimized concept set comprises known concept types and optimized concept types, and the optimized concept types comprise clustering each embedding vector obtained by embedding an attribute value in an enumerable attribute that does not match a known concept type, and determining a new concept type according to a clustering category; a construction unit configured to construct at least one concept tree of the knowledge graph through the optimized concept set, and a knowledge semantic framework SPG model formed by the concept trees being used for construction of the knowledge graph.

7. A computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-5. ​ ​ ​ ​ ​ ​ 8. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-5.

Images