Method, device, equipment and readable storage medium for generating general data relation structure

Abstract

The present application provides a method, device, equipment and computer-readable storage medium for generating a general data relationship structure. The method for generating a general data relationship structure includes: automatically extracting entity information from a data table; using a model algorithm to normalize entity information; extracting commonalities and labeling semantic tags for entity information after entity normalization; extracting entity relationships between entities; storing mapping relationships between data tables and entities, mapping relationships between entities and semantic tags, and entity relationships; generating entity relationship graphs; and generating data table relationship graphs. The present application can handle situations where the relationship data of target data is stored in multiple two-dimensional tables, adding more analyzable scenarios to data analysis, and the structure supports drill-down operations. The target data's relationship can be used to find its related data, and the next step of related data query can be performed based on the related data, so that the relationship data is connected to form an effective data network structure.

Description

Technical Field

[0001] The present application belongs to the technical field of general data relationship structure generation, and in particular, relates to a general data relationship structure generation method, device, equipment and computer-readable storage medium. Background Art

[0002] With the rapid development of the Internet, a large amount of data has been mined. Data analysis is to obtain effective information from this data so that people can use this effective information to assist in judgment and decision-making.

[0003] Data relations are an important foundation in data analysis. They are used to connect data, so that data is no longer isolated and to increase the accuracy of data analysis. Currently, mainstream relational databases such as MySQL and Oracle are used to store these relational data.

[0004] Currently, the data relationship structure of most data analyses is stored in a flat two-dimensional table. However, this two-dimensional table data relationship structure lacks the ability to express data relationships. In daily practical applications, related data may be stored in multiple two-dimensional tables. In this case, it is difficult to analyze data relationships. This two-dimensional table data relationship structure is not suitable for today's data analysis scenarios. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and computer-readable storage medium for generating a general data relationship structure, which can handle the situation where the relationship data of the target data is stored in multiple two-dimensional tables, adding more analyzable scenarios to data analysis, and the structure supports drill-down operations. The related data can be found through the relationship of the target data, and the next step of related data query can be performed based on the related data, so that the relationship data is connected to form an effective data network structure.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a general data relationship structure, including:

[0007] Automatically extract entity information from data tables;

[0008] Use model algorithms to normalize entity information to eliminate contradictions and ambiguities;

[0009] For the entity information after entity normalization, extract commonalities and add semantic labels;

[0010] Extract entity relationships between entities based on entity information with semantic tags;

[0011] Store the mapping relationship between data tables and entities, the mapping relationship between entities and semantic tags, and entity relationships for background maintenance and management;

[0012] Generate entity relationship diagram based on entity relationship;

[0013] Generate a data table relationship diagram based on the entity relationship diagram.

[0014] Optionally, the method further includes:

[0015] Based on semantic tags, split the target data and determine the entities associated with the target data;

[0016] Based on the mapping relationship between data tables and entities and the entity relationship diagram, determine the table information of the entity associated with the target data;

[0017] Based on the table information and entity relationship diagram, a query path for the target data is formed.

[0018] Optionally, the entity information includes at least person name information, organization name information, time information and status information.

[0019] Optionally, based on the entity information after semantic labeling, entity relationships between entities are extracted, including:

[0020] Based on the entity information with semantic labels, the entity relationships between entities are automatically extracted using supervised learning algorithms.

[0021] In a second aspect, an embodiment of the present application provides a general data relationship structure generation device, including:

[0022] Entity information extraction module, used to automatically extract entity information from data tables;

[0023] The entity normalization module is used to normalize entity information using a model algorithm to eliminate contradictions and ambiguities;

[0024] The semantic tag adding module is used to extract commonalities and add semantic tags to entity information after entity normalization;

[0025] The entity relationship extraction module is used to extract entity relationships between entities based on entity information with semantic tags;

[0026] The relationship storage module is used to store the mapping relationship between data tables and entities, the mapping relationship between entities and semantic tags, and the entity relationship for background maintenance and management;

[0027] An entity relationship diagram generation module is used to generate an entity relationship diagram according to entity relationships;

[0028] The data table relationship diagram generation module is used to generate a data table relationship diagram based on the entity relationship diagram.

[0029] Optionally, the device further comprises:

[0030] The target data splitting module is used to split the target data based on semantic tags and determine the entities associated with the target data;

[0031] A table information determination module, used to determine the table information of the entity associated with the target data based on the mapping relationship between the data table and the entity and the entity relationship diagram;

[0032] The query path forming module is used to form a query path for target data based on table information and entity relationship diagram.

[0033] Optionally, the entity information includes at least person name information, organization name information, time information and status information.

[0034] Optionally, an entity relationship extraction module is used to automatically extract entity relationships between entities based on entity information with semantic labels using a supervised learning algorithm.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;

[0036] When the processor executes the computer program instructions, the general data relationship structure generation method shown in the first aspect is implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the general data relationship structure generation method shown in the first aspect is implemented.

[0038] The general data relationship structure generation method, device, equipment and computer-readable storage medium of the embodiments of the present application can handle the situation where the relationship data of the target data is stored in multiple two-dimensional tables, adding more analyzable scenarios to data analysis, and the structure supports drill-down operations. The related data can be found through the relationship of the target data, and the next related data query can be performed based on the related data, so that the relationship data is connected to form an effective data network structure.

[0039] The method for generating a universal data relationship structure comprises: automatically extracting entity information of a data table; using a model algorithm to normalize the entity information to eliminate contradictions and ambiguities; extracting commonalities and marking semantic tags on the entity information after entity normalization; extracting entity relationships between entities based on the entity information marked with semantic tags; storing mapping relationships between data tables and entities, mapping relationships between entities and semantic tags, and entity relationships for background maintenance and management; generating an entity relationship diagram based on the entity relationship; and generating a data table relationship diagram based on the entity relationship diagram.

[0040] It can be seen that this method can handle the situation where the relational data of the target data is stored in multiple two-dimensional tables, which adds more analyzable scenarios for data analysis. In addition, this structure supports drill-down operations. The related data can be found through the relationship of the target data, and the next step of related data query can be performed based on the related data, so that the relational data can be connected to form an effective data network structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flowchart of a method for generating a general data relationship structure provided by an embodiment of the present application;

[0043] Figure 2 is an entity relationship diagram provided by an embodiment of the present application;

[0044] Figure 3 This is a query path diagram provided by an embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of generating a general relationship structure provided by an embodiment of the present application;

[0046] Figure 5 It is a structural diagram of a general data relationship structure generating device provided by an embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0049] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0050] With the rapid development of the Internet, a large amount of data has been mined. Data analysis is to obtain effective information from this data so that people can use this effective information to assist in judgment and decision-making.

[0051] Data relations are an important foundation in data analysis. They are used to connect data, so that data is no longer isolated and to increase the accuracy of data analysis. Currently, mainstream relational databases such as MySQL and Oracle are used to store these relational data.

[0052] Currently, the data relationship structure of most data analyses is stored in a flat two-dimensional table. However, this two-dimensional table data relationship structure lacks the ability to express data relationships. In daily practical applications, related data may be stored in multiple two-dimensional tables. In this case, it is difficult to analyze data relationships. This two-dimensional table data relationship structure is not suitable for today's data analysis scenarios.

[0053] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment and computer-readable storage medium for generating a general data relationship structure. The following first introduces the method for generating a general data relationship structure provided by the embodiments of the present application.

[0054] Figure 1 FIG. 1 is a flow chart showing a method for generating a general data relationship structure according to an embodiment of the present application. Figure 1 As shown, the general data relationship structure generation method includes:

[0055] S101, automatically extracting entity information of the data table;

[0056] In one embodiment, the entity information includes at least person name information, organization name information, time information and status information.

[0057] S102, using a model algorithm to normalize entity information to eliminate contradictions and ambiguities;

[0058] S103, extracting commonalities from entity information after entity normalization and adding semantic tags;

[0059] S104, extracting entity relationships between entities based on entity information labeled with semantic tags;

[0060] In one embodiment, based on entity information with semantic tags, entity relationships between entities are extracted, including:

[0061] Based on the entity information with semantic labels, the entity relationships between entities are automatically extracted using supervised learning algorithms.

[0062] S105, storing the mapping relationship between the data table and the entity, the mapping relationship between the entity and the semantic tag, and the entity relationship for background maintenance and management;

[0063] S106, generating an entity relationship diagram according to the entity relationship;

[0064] S107. Generate a data table relationship diagram based on the entity relationship diagram.

[0065] In one embodiment, the method further comprises:

[0066] Based on semantic tags, split the target data and determine the entities associated with the target data;

[0067] Based on the mapping relationship between data tables and entities and the entity relationship diagram, determine the table information of the entity associated with the target data;

[0068] Based on the table information and entity relationship diagram, a query path for the target data is formed.

[0069] In order to explain the general data relationship structure generation method provided by the present application in detail, a specific embodiment is described below, including the following steps:

[0070] (1) Data extraction: automatically extract entity information elements from data tables, including person names, organization names, time, status, and other entity information, and supports multiple relational libraries.

[0071] (2) Based on the entity information extracted from the data in (1), the model algorithm is used to eliminate contradictions and ambiguities and normalize the extracted entity information. For example, some entities may have multiple expressions, and a specific title may correspond to multiple different entities.

[0072] (3) After normalizing the entities in (2), extract their commonalities and label them with semantic labels.

[0073] (4) Based on the entity information extracted in (3), the relationship between entities is extracted based on the template, and the entity relationship template is maintained in the background. Later, algorithms can be introduced based on a large amount of labeled data to automatically extract entity relationships using supervised learning algorithms.

[0074] (5) Design the underlying storage method for the correspondence between the relational table extracted in (1) and the entities, the mapping between entities and semantic labels in (3), and the relationship between entities in (4). The storage method will directly affect the query efficiency and application effect.

[0075] (6) After storing these relationships, add a background to facilitate the maintenance and management of these relationships.

[0076] (7) Data splitting: split the target data according to the semantic tags in (3) to determine the entities associated with the target data. For example, refrigeration income will be split into two entities: refrigeration and income.

[0077] (8) Generate an entity relationship diagram based on the entity relationships in (4). For example, refrigeration (data under the organizational dimension) is related to TableA, TableB, TableC, and TableD, and revenue is related to advertising investment, order execution rate, and return rate. For details, see Figure 2 .

[0078] Among them, TableA: period, organization, region, revenue, return rate;

[0079] Table B: period, organization, region, channel, revenue, advertising investment;

[0080] TableC: period, organization, product, revenue, order execution rate;

[0081] TableD: Period, organization, income....

[0082] (9) According to the relationship table and entity correspondence maintained in the background in (6), and the entity relationship diagram generated in (8), find the table information where the target data entity is located, and form the query path of the target data based on the table information and the entity relationship diagram. For example, to find the query path for refrigeration (data under the organizational dimension) revenue, please refer to Figure 3 .

[0083] (10) The general data relationship structure is shown in Table 1 below:

[0084] Table 1

[0085] Property Name type describe targetDimen List<String> Target Dimension targetlndicator String Target indicators indicatorRelation HashMap<String, lndicatorlnfo> Indicator correlation query Path List<tabnlelnfo> Query Path

[0086] Tablelnfo (two-dimensional table information) is shown in Table 2 below:

[0087] Table 2

[0088] Property Name type describe dimlnfo List<Dimlnfo> Dimension information of a two-dimensional table indicatorlnfo lndicatorlnfo Index information of two-dimensional table tableName String The name of the two-dimensional table

[0089] (11) Generate general relationship structure. For details, see Figure 4 .

[0090] This universal relational structure expands the ability of relational databases to express data relationships and can be used in various relational databases.

[0091] This embodiment can handle the situation where the relational data of the target data is stored in multiple two-dimensional tables, which adds more analyzable scenarios to data analysis. The structure supports drill-down operations. The related data can be found through the relationship of the target data, and the next step of related data query can be performed based on the related data, so that the relational data can be connected to form an effective data network structure.

[0092] like Figure 5 As shown, the present application provides a general data relationship structure generation device, including:

[0093] An entity information extraction module 501 is used to automatically extract entity information from a data table;

[0094] The entity normalization module 502 is used to normalize the entity information by using a model algorithm to eliminate contradictions and ambiguities;

[0095] The semantic tag adding module 503 is used to extract commonalities and add semantic tags to the entity information after entity normalization;

[0096] An entity relationship extraction module 504 is used to extract entity relationships between entities based on entity information marked with semantic tags;

[0097] The relationship storage module 505 is used to store the mapping relationship between data tables and entities, the mapping relationship between entities and semantic tags, and entity relationships for background maintenance and management;

[0098] An entity relationship diagram generation module 506, used to generate an entity relationship diagram according to entity relationships;

[0099] The data table relationship diagram generation module 507 is used to generate a data table relationship diagram based on the entity relationship diagram.

[0100] In one embodiment, the apparatus further comprises:

[0101] The target data splitting module is used to split the target data based on semantic tags and determine the entities associated with the target data;

[0102] A table information determination module, used to determine the table information of the entity associated with the target data based on the mapping relationship between the data table and the entity and the entity relationship diagram;

[0103] The query path forming module is used to form a query path for target data based on table information and entity relationship diagram.

[0104] In one embodiment, the entity information includes at least person name information, organization name information, time information and status information.

[0105] In one embodiment, the entity relationship extraction module 504 is used to automatically extract entity relationships between entities using a supervised learning algorithm based on entity information with semantic tags.

[0106] Figure 5 Each module / unit in the device shown has the function of realizing Figure 1 The functions of each step in the process can achieve the corresponding technical effects, which will not be described in detail here for the sake of brevity.

[0107] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.

[0108] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0109] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0110] The memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 602 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 602 may be inside or outside the electronic device. In a particular embodiment, the memory 602 may be a non-volatile solid-state memory.

[0111] In one embodiment, the memory 602 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0112] The processor 601 implements any one of the general data relationship structure generation methods in the above embodiments by reading and executing computer program instructions stored in the memory 602 .

[0113] In one example, the electronic device may further include a communication interface 603 and a bus 610. Figure 6 As shown, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 and communicate with each other.

[0114] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0115] Bus 610 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 610 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0116] In addition, in combination with the general data relationship structure generation method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the general data relationship structure generation methods in the above embodiment is implemented.

[0117] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0118] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0119] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0120] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0121] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for generating a general data relation structure, characterized in that: include: Automatically extract entity information from data tables; Using a model algorithm, the entity information is normalized to eliminate contradictions and ambiguities; For the entity information after entity normalization, extract commonalities and add semantic labels; Extract entity relationships between entities based on entity information with semantic tags; Storing the mapping relationship between the data table and the entity, the mapping relationship between the entity and the semantic tag, and the entity relationship for background maintenance and management; According to the entity relationship, an entity relationship diagram is generated; Based on the entity relationship diagram, generate a data table relationship diagram; Based on the semantic tags, split the target data and determine the entities associated with the target data; Based on the mapping relationship between the data table and the entity and the entity relationship diagram, determine the table information of the entity associated with the target data; A query path for the target data is formed according to the table information and the entity relationship diagram.

2. The method for generating a general data relationship structure according to claim 1, characterized in that: The entity information at least includes person name information, organization name information, time information and status information.

3. The method for generating a general data relationship structure according to claim 1, characterized in that: The extracting entity relationships between entities based on entity information marked with semantic tags includes: Based on the entity information with semantic labels, the entity relationships between entities are automatically extracted using supervised learning algorithms.

4. A general data relationship structure generating device, characterized in that: include: Entity information extraction module, used to automatically extract entity information from data tables; An entity normalization module, used to perform entity normalization on the entity information by using a model algorithm to eliminate contradictions and ambiguities; The semantic tag adding module is used to extract commonalities and add semantic tags to entity information after entity normalization; The entity relationship extraction module is used to extract entity relationships between entities based on entity information with semantic tags; A relationship storage module, used to store the mapping relationship between the data table and the entity, the mapping relationship between the entity and the semantic label, and the entity relationship, for background maintenance and management; An entity relationship diagram generation module, used to generate an entity relationship diagram according to the entity relationship; A data table relationship diagram generation module, used to generate a data table relationship diagram based on the entity relationship diagram; A target data splitting module, used to split the target data based on the semantic tags and determine the entities associated with the target data; A table information determination module, used to determine the table information of the entity associated with the target data based on the mapping relationship between the data table and the entity and the entity relationship diagram; The query path forming module is used to form a query path for the target data according to the table information and the entity relationship diagram.

5. The general data relationship structure generating device according to claim 4, characterized in that: The entity information at least includes person name information, organization name information, time information and status information.

6. The general data relationship structure generating device according to claim 4, characterized in that: The entity relationship extraction module is used to automatically extract entity relationships between entities using a supervised learning algorithm based on entity information marked with semantic tags.

7. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for generating a general data relationship structure as described in any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for generating a general data relationship structure as described in any one of claims 1 to 3 is implemented.

Images