Heterogeneous data processing method, device, server and storage medium

By receiving and processing heterogeneous data sets and generating query type word sets, the problem of inefficient current heterogeneous data processing methods is solved, and faster data processing and resource utilization is achieved.

CN114003731BActive Publication Date: 2024-11-26STATE GRID HEBEI ELECTRIC POWER RES INST +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111274904.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-29
Publication Date
2024-11-26
Estimated Expiration
2041-10-29

AI Technical Summary

Technical Problem

The existing heterogeneous data processing methods are inefficient and time-consuming. Especially in the process of data cleaning and classification of automatic terminals in distribution networks, cleaning rules need to be re-established when changes occur, resulting in a long processing time. .

Method used

By receiving heterogeneous data sets, determining the data type and performing structured or standardized processing, generating attribute vocabulary lists, performing semantic calculations to determine the query type word set, and uniformly constructing the query type word set, shortening data processing time and improving efficiency.

Benefits of technology

By uniformly processing heterogeneous data sets, the data processing time is shortened, the processing efficiency is improved, the network resource occupation is reduced, and the time cost of data processing is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114003731B_ABST
    Figure CN114003731B_ABST
Patent Text Reader

Abstract

The present invention provides a heterogeneous data processing method, device, server and storage medium. The method includes: receiving an uploaded heterogeneous data set, and determining the data type of each heterogeneous data in the heterogeneous data set; the data type includes structured or unstructured; for each heterogeneous data whose data type is unstructured, the unstructured heterogeneous data is structured, and the unstructured heterogeneous data is converted into structured heterogeneous data; for each structured heterogeneous data, the structured heterogeneous data is standardized to obtain an attribute word table of the structured heterogeneous data; all attribute word tables are collected to form an attribute word table set of the heterogeneous data set; semantic calculation is performed on the attribute word table set, and the query type word set of the heterogeneous data set is determined according to the result of the semantic calculation. The present invention can improve the efficiency of heterogeneous data processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a heterogeneous data processing method, device, server and storage medium. Background Art

[0002] As an integral part of the distribution network automation operation, the distribution network automation terminal provides guarantee for the stable, safe and reliable operation of the distribution network. The quality inspection of the distribution network automation terminal has always been the focus of attention of all parties. With the continuous deepening of informatization construction, it is a future development trend to uniformly upload the original inspection data of the distribution network automation terminal of terminal manufacturers, power supply companies, distribution network automation master stations and other personnel to the master station server for centralized and unified management. Due to the different retention methods of the inspection data of each party, the original inspection data uploaded to the master station server can be considered as heterogeneous data.

[0003] In the existing heterogeneous data processing, most of the upload terminals establish corresponding data cleaning rules with the main station server. The original detection data will be cleaned in their respective terminals, and the data uploaded to the main station server is in a standardized format. The main station server then classifies the data according to the standardized format and establishes the query type. However, if the main station server or the upload terminal changes, the cleaning rules need to be re-established between the two, and the main station server needs to re-classify the data, which takes a long time. That is, the existing heterogeneous data processing method is inefficient and time-consuming. Summary of the invention

[0004] The embodiments of the present invention provide a heterogeneous data processing method, device, server and storage medium to solve the problems of low processing efficiency and long processing time in existing heterogeneous data processing methods.

[0005] In a first aspect, an embodiment of the present invention provides a heterogeneous data processing method, including:

[0006] Receive uploaded heterogeneous data sets and determine the data type of each heterogeneous data in the heterogeneous data sets; the data type includes structured or unstructured;

[0007] For each type of heterogeneous data that is unstructured, the unstructured heterogeneous data is structured and converted into structured heterogeneous data;

[0008] For each structured heterogeneous data, standardize the structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data;

[0009] All attribute word lists are collected to form an attribute word list set of the heterogeneous data set; semantic calculation is performed on the attribute word list set, and the query type word list of the heterogeneous data set is determined according to the result of the semantic calculation.

[0010] In a possible implementation, semantic computation is performed on the attribute vocabulary set, and the query type vocabulary set of the heterogeneous data set is determined according to the result of the semantic computation, including:

[0011] For every two attribute words in the attribute word table in the attribute word table set, calculate the semantic similarity of the two attribute words;

[0012] When the semantic similarity of the two attribute words is greater than a preset similarity threshold, the two attribute words are determined to be similar, and any one of the two attribute words is selected as the query type word;

[0013] All query type vocabulary sets are combined to obtain the query type vocabulary sets of heterogeneous datasets.

[0014] In a possible implementation, semantic computation is performed on the attribute vocabulary set, and the query type vocabulary set of the heterogeneous data set is determined according to the result of the semantic computation, including:

[0015] For every two attribute words in the attribute word table in the attribute word table set, calculate the semantic similarity of the two attribute words;

[0016] When the semantic similarity of the two attribute words is less than or equal to a preset similarity threshold, the two attribute words are determined to be dissimilar, and both of the two attribute words are used as query type words;

[0017] All query type vocabulary sets are combined to obtain the query type vocabulary sets of heterogeneous datasets.

[0018] In a possible implementation, calculating the semantic similarity between the two attribute words includes:

[0019] The semantic distances between all the semantic primitives of the two attribute words are calculated, and the average of the semantic distances between all the semantic primitives of the two attribute words is taken as the semantic similarity of the two attribute words.

[0020] In a possible implementation, the structured heterogeneous data is standardized to obtain an attribute vocabulary of the structured heterogeneous data, including:

[0021] Decomposing the structured heterogeneous data to obtain a data word segmentation table of the structured heterogeneous data;

[0022] The data words in the data word table are clustered to obtain an attribute word table of the structured heterogeneous data.

[0023] In a possible implementation, the unstructured heterogeneous data is structured and converted into structured heterogeneous data, including:

[0024] Convert the unstructured heterogeneous data into an XML document;

[0025] According to the preset conversion relationship, the XML document is mapped into structured heterogeneous data.

[0026] In a second aspect, an embodiment of the present invention provides a heterogeneous data processing device, including:

[0027] A receiving module is used to receive the uploaded heterogeneous data set and determine the data type of each heterogeneous data in the heterogeneous data set; the data type includes structured or unstructured;

[0028] A conversion module is used for performing structured processing on each unstructured heterogeneous data of which the data type is unstructured heterogeneous data, and converting the unstructured heterogeneous data into structured heterogeneous data;

[0029] A processing module, for performing standardization processing on each structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data;

[0030] A collection module is used to collect all attribute vocabularies to form an attribute vocabularies set of heterogeneous data sets;

[0031] The calculation module is used to perform semantic calculation on the attribute word set and determine the query type word set of the heterogeneous data set according to the result of the semantic calculation.

[0032] In a possible implementation, the calculation module includes:

[0033] A first calculation unit is used to calculate the semantic similarity of any two attribute words in the attribute word table in the attribute word table set;

[0034] A first judgment unit is used to judge that the two attribute words are similar when the semantic similarity of the two attribute words is greater than a preset similarity threshold, and select any one of the two attribute words as a query type word;

[0035] The first aggregation unit is used to collect all query type vocabulary sets to obtain the query type word set of the heterogeneous data set.

[0036] In a third aspect, an embodiment of the present invention provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the heterogeneous data processing method according to the first aspect or any possible implementation method of the first aspect are implemented.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the heterogeneous data processing method as described in the first aspect or any possible implementation method of the first aspect.

[0038] The embodiment of the present invention provides a heterogeneous data processing method, device, server and storage medium, which receives an uploaded heterogeneous data set and determines the data type of each heterogeneous data in the heterogeneous data set; the data type includes structured or unstructured; for each heterogeneous data whose data type is unstructured, the unstructured heterogeneous data is structured and converted into structured heterogeneous data; for each structured heterogeneous data, the structured heterogeneous data is standardized to obtain an attribute word table of the structured heterogeneous data; all attribute word tables are collected to form an attribute word table set of the heterogeneous data set; semantic calculation is performed on the attribute word table set, and the query type word set of the heterogeneous data set is determined according to the result of the semantic calculation. By uniformly constructing a query type word set for the heterogeneous data set, the processing time of the heterogeneous data processing method is shortened, the efficiency of heterogeneous data processing is improved, and the occupation of network resources is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 is a flow chart of the implementation of the heterogeneous data processing method provided by an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of the structure of a heterogeneous data processing device provided by an embodiment of the present invention;

[0042] Figure 3 It is a schematic diagram of a server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0045] See also Figure 1 , which shows a flowchart of the implementation of the heterogeneous data processing method provided by an embodiment of the present invention. Figure 1 As shown, a heterogeneous data processing method may include:

[0046] S101, receiving an uploaded heterogeneous data set, and determining the data type of each heterogeneous data in the heterogeneous data set; the data type includes structured or unstructured;

[0047] Optionally, heterogeneous data refers to data of different structures. The server can receive data uploaded from different terminals, for example, heterogeneous data of detection information about distribution network automation terminals uploaded by terminal manufacturers, power supply companies, distribution network automation master stations, etc., can be received respectively. The heterogeneous data uploaded by multiple parties constitute a heterogeneous data set.

[0048] Optionally, structured data is data that can be represented by data or a unified structure, such as numbers, letters, etc. Unstructured data is data that cannot be represented by data or a unified structure, such as pictures, videos, etc. The data types of heterogeneous data in the heterogeneous data set can be determined by selecting appropriate existing technologies according to actual needs.

[0049] S102, for each unstructured heterogeneous data, perform structured processing on the unstructured heterogeneous data to convert the unstructured heterogeneous data into structured heterogeneous data;

[0050] Optionally, for unstructured heterogeneous data, data extraction combined with data cleaning can be used to convert unstructured heterogeneous data into structured heterogeneous data. Structured heterogeneous data is easier to process.

[0051] S103, for each structured heterogeneous data, perform standardization processing on the structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data;

[0052] Optionally, after the unstructured heterogeneous data is converted into structured heterogeneous data, it and the original structured heterogeneous data together form a structured heterogeneous data set. In the embodiment of the present invention, "each structured heterogeneous data" refers to the data in the structured heterogeneous data set.

[0053] Optionally, for the detection data of the distribution network automation terminal, it can be the detection results and evaluation results recorded in the form of text. The attribute word list of the heterogeneous data obtained by standardizing the structured data is to decompose the recorded data of the text into attribute words, and each attribute word is represented in the form of a table, which is the attribute word list. The attribute word list can represent the decomposed vocabulary of the structured heterogeneous data. In addition, the attribute word can also represent the specific information of the structured heterogeneous data.

[0054] S104, collecting all attribute vocabularies to form an attribute vocabularies set of the heterogeneous data set;

[0055] Optionally, the attribute word lists of each structured heterogeneous data set can be aggregated to form an attribute word list set of the heterogeneous data set. The attribute word list set includes all information of the heterogeneous data set.

[0056] S105, performing semantic calculation on the attribute word set, and determining the query type word set of the heterogeneous data set according to the result of the semantic calculation.

[0057] Optionally, the attribute word set includes all attribute words of the heterogeneous data set, and some attribute words are exactly the same, and some attribute words have the same meaning even though they are expressed in different forms. Therefore, it is necessary to screen all attribute words, screen out all query type words that can represent the heterogeneous data set, and integrate all query type words into the query type word set. When there is a need to query specific data in the heterogeneous data set, the query type words can be used to determine the content to be queried, which greatly improves the query efficiency.

[0058] In addition, for two heterogeneous data sets, they are represented as a first heterogeneous data set and a second heterogeneous data set. After determining the first query type word set corresponding to the first heterogeneous data set and the second query type word set corresponding to the second heterogeneous data set, semantic calculations can be performed on all query type words in the first query type word set and the second query type word set, and screening can be performed to finally obtain a common query type word set that can represent the first heterogeneous data set and the second heterogeneous data set. For the case of multiple heterogeneous data sets, the common query type word set can be determined by referring to the above method, which will not be repeated here.

[0059] The embodiment of the present invention receives an uploaded heterogeneous data set and determines the data type of each heterogeneous data in the heterogeneous data set; the data type includes structured or unstructured; for each heterogeneous data whose data type is unstructured, the unstructured heterogeneous data is structured and converted into structured heterogeneous data; for each structured heterogeneous data, the structured heterogeneous data is standardized to obtain an attribute word table of the structured heterogeneous data; all attribute word tables are collected to form an attribute word table set of the heterogeneous data set; semantic calculation is performed on the attribute word table set, and the query type word set of the heterogeneous data set is determined according to the result of the semantic calculation. By uniformly constructing a query type word set for the heterogeneous data set, the processing time of the heterogeneous data processing method is shortened, the processing efficiency of the heterogeneous data is improved, and the occupation of network resources is reduced.

[0060] In some embodiments of the present invention, the above S105 may include:

[0061] For every two attribute words in the attribute word table in the attribute word table set, calculate the semantic similarity of the two attribute words;

[0062] When the semantic similarity of the two attribute words is greater than a preset similarity threshold, the two attribute words are determined to be similar, and any one of the two attribute words is selected as the query type word;

[0063] All query type vocabulary sets are combined to obtain the query type vocabulary sets of heterogeneous datasets.

[0064] Optionally, the cosine similarity, Euclidean distance, Manhattan distance, Minkowski distance or Pearson correlation coefficient of two attribute words can be calculated to characterize the semantic similarity. The greater the semantic similarity, the closer the semantics of the two attribute words are. A similarity threshold can be preset. When the semantic similarity between two attribute words is greater than the preset similarity threshold, it indicates that the two attribute words have the same semantics, and any one of the two attribute words can be selected as the query type word. Finally, all query type vocabulary sets are obtained to obtain the query type word set of the heterogeneous data set.

[0065] In some embodiments of the present invention, the above S105 may further include:

[0066] For every two attribute words in the attribute word table in the attribute word table set, calculate the semantic similarity of the two attribute words;

[0067] When the semantic similarity of the two attribute words is less than or equal to a preset similarity threshold, the two attribute words are determined to be dissimilar, and both of the two attribute words are used as query type words;

[0068] All query type vocabulary sets are combined to obtain the query type vocabulary sets of heterogeneous datasets.

[0069] Optionally, the smaller the semantic similarity, the greater the semantic difference between the two attribute words. A similarity threshold can be preset. When the semantic similarity between the two attribute words is less than or equal to the preset similarity threshold, it indicates that the two attribute words are semantically different, and both attribute words can be used as query type words. Finally, all query type vocabulary sets are combined to obtain the query type word set of the heterogeneous data set.

[0070] In some embodiments of the present invention, calculating the semantic similarity of the two attribute words includes:

[0071] The semantic distances between all the semantic primitives of the two attribute words are calculated, and the average of the semantic distances between all the semantic primitives of the two attribute words is taken as the semantic similarity of the two attribute words.

[0072] Optionally, each attribute word may have multiple meanings, that is, each attribute word may include multiple semantic primitives. The semantic distances between the multiple semantic primitives corresponding to the two attribute words are calculated, and the average of the semantic distances between all the semantic primitives of the two attribute words is taken as the semantic similarity of the two attribute words. Alternatively, the average of the maximum semantic distance and the minimum semantic distance between all the semantic primitives of the two attribute words is selected as the semantic similarity of the two attribute words.

[0073] In some embodiments of the present invention, the above S103 "standardizing the structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data" may include:

[0074] Decomposing the structured heterogeneous data to obtain a data word segmentation table of the structured heterogeneous data;

[0075] The data words in the data word table are clustered to obtain an attribute word table of the structured heterogeneous data.

[0076] Optionally, the structured heterogeneous data can be decomposed to obtain the corresponding data word segmentation table. The data word segmentation table may contain completely identical words or words with obvious errors in content. These words can be simply corrected, and finally the corrected data word segmentation table can be clustered using existing clustering methods to obtain the attribute word table of the structured heterogeneous data. For example, the K-means clustering algorithm can be used to cluster the corrected data word segmentation table, or a suitable clustering algorithm can be selected according to actual needs.

[0077] In some embodiments of the present invention, the above S103 “structuring the unstructured heterogeneous data to convert the unstructured heterogeneous data into structured heterogeneous data” may include:

[0078] Convert the unstructured heterogeneous data into an XML document;

[0079] According to the preset conversion relationship, the XML document is mapped into structured heterogeneous data.

[0080] Optionally, unstructured data can be converted into structured data by adopting a step-by-step conversion method of “unstructured data – semi-structured data – structured data”.

[0081] Specifically, the conversion process can be expressed as:

[0082] First, by analyzing the structural characteristics of unstructured heterogeneous data and XML documents, the conversion of unstructured heterogeneous data into XML documents is achieved.

[0083] Second, by studying the mapping method between XML documents and structured heterogeneous data, the conversion from XML documents to structured heterogeneous data is realized. The preset conversion relationship is the mapping method.

[0084] The mapping method can be template-driven mapping, which is a template-based mapping that embeds SQL statements into XML documents; or model-driven mapping, which does not require embedded SQL commands and establishes a corresponding relationship between the tree structure of the XML document and the relational structure of structured heterogeneous data.

[0085] The beneficial effects of the embodiments of the present invention are as follows: by uploading heterogeneous data sets to a server and having the server uniformly process the heterogeneous data, data processing time can be greatly reduced, network resource usage can be reduced, and heterogeneous data processing efficiency can be improved.

[0086] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0087] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0088] Figure 2 A schematic diagram of the structure of a heterogeneous data processing device provided by an embodiment of the present invention is shown. For the sake of convenience, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

[0089] like Figure 2 As shown, the heterogeneous data processing device 20 may include:

[0090] The receiving module 201 is used to receive the uploaded heterogeneous data set and determine the data type of each heterogeneous data in the heterogeneous data set; the data type includes structured or unstructured;

[0091] The conversion module 202 is used for performing structured processing on each unstructured heterogeneous data whose data type is unstructured heterogeneous data, and converting the unstructured heterogeneous data into structured heterogeneous data;

[0092] The processing module 203 is used to perform standardization processing on each structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data;

[0093] A collection module 204, used to collect all attribute vocabularies to form an attribute vocabularies set of heterogeneous data sets;

[0094] The calculation module 205 is used to perform semantic calculation on the attribute word set, and determine the query type word set of the heterogeneous data set according to the result of the semantic calculation.

[0095] In some embodiments of the present invention, the calculation module 205 may include:

[0096] A first calculation unit is used to calculate the semantic similarity of any two attribute words in the attribute word table in the attribute word table set;

[0097] A first judgment unit is used to judge that the two attribute words are similar when the semantic similarity of the two attribute words is greater than a preset similarity threshold, and select any one of the two attribute words as a query type word;

[0098] The first aggregation unit is used to collect all query type vocabulary sets to obtain the query type word set of the heterogeneous data set.

[0099] In some embodiments of the present invention, the calculation module 205 may further include:

[0100] A second calculation unit is used to calculate the semantic similarity of any two attribute words in the attribute word table in the attribute word table set;

[0101] A second judgment unit is used to judge that the two attribute words are not similar when the semantic similarity of the two attribute words is less than or equal to a preset similarity threshold, and to use the two attribute words as query type words;

[0102] The second aggregation unit is used to collect all query type vocabulary sets to obtain the query type word set of the heterogeneous data set.

[0103] In some embodiments of the present invention, the first calculation unit or the second calculation unit can be used to calculate the semantic distance between all the semantic primitives of the two attribute words, and take the average of the semantic distances between all the semantic primitives of the two attribute words as the semantic similarity of the two attribute words.

[0104] In some embodiments of the present invention, the processing module 203 may include:

[0105] A decomposition unit, used for decomposing the structured heterogeneous data to obtain a data word segmentation table of the structured heterogeneous data;

[0106] The clustering unit is used to cluster the data words in the data word table to obtain the attribute word table of the structured heterogeneous data.

[0107] In some embodiments of the present invention, the conversion module 202 may include:

[0108] A first conversion unit, used for converting the unstructured heterogeneous data into an XML document;

[0109] The second conversion unit is used to map the XML document into structured heterogeneous data according to a preset conversion relationship.

[0110] Figure 3 Schematic diagram of a server provided by an embodiment of the present invention. Figure 3 As shown, the server 30 of this embodiment includes: a processor 300, a memory 301, and a computer program 302 stored in the memory 301 and executable on the processor 300. When the processor 300 executes the computer program 302, the steps in the above-mentioned various heterogeneous data processing method embodiments are implemented, for example Figure 1 Alternatively, when the processor 300 executes the computer program 302, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 2 Functionality of modules / units 201 to 205 is shown.

[0111] Exemplarily, the computer program 302 may be divided into one or more modules / units, one or more modules / units are stored in the memory 301 and executed by the processor 300 to implement the present invention. One or more modules / units may be a series of computer program instruction segments that can implement specific functions, and the instruction segments are used to describe the execution process of the computer program 302 in the server 30. For example, the computer program 302 may be divided into Figure 2 Modules / units 201 to 205 are shown.

[0112] The server 30 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The server 30 may include, but is not limited to, a processor 300 and a memory 301. Those skilled in the art will appreciate that Figure 3This is only an example of the server 30 and does not constitute a limitation on the server 30 . The server 30 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the server may also include input and output devices, network access devices, buses, etc.

[0113] The processor 300 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] The memory 301 may be an internal storage unit of the server 30, such as a hard disk or memory of the server 30. The memory 301 may also be an external storage device of the server 30, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server 30. Further, the memory 301 may also include both an internal storage unit of the server 30 and an external storage device. The memory 301 is used to store computer programs and other programs and data required by the server. The memory 301 may also be used to temporarily store data that has been output or is to be output.

[0115] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] In the embodiments provided by the present invention, it should be understood that the disclosed devices / servers and methods can be implemented in other ways. For example, the device / server embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various heterogeneous data processing method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0122] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for processing heterogeneous data, characterized in that: include: Receiving an uploaded heterogeneous data set, and determining a data type of each heterogeneous data in the heterogeneous data set; The data type includes structured or unstructured; For each type of heterogeneous data that is unstructured, the unstructured heterogeneous data is structured and converted into structured heterogeneous data; For each structured heterogeneous data, standardize the structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data; Collecting all attribute vocabularies to form an attribute vocabularies set of the heterogeneous data set; Performing semantic calculation on the attribute word set, and determining the query type word set of the heterogeneous data set according to the result of the semantic calculation; The performing semantic calculation on the attribute word set and determining the query type word set of the heterogeneous data set according to the result of the semantic calculation includes: For every two attribute words in the attribute word table in the attribute word table set, the semantic similarity of the two attribute words is calculated; wherein the semantic similarity of the two attribute words is any one of the cosine similarity, Euclidean distance, Manhattan distance, Minkowski distance or Pearson correlation coefficient of the two attribute words; When the semantic similarity of the two attribute words is greater than a preset similarity threshold, the two attribute words are determined to be similar, and any one of the two attribute words is selected as the query type word; All query type vocabulary sets are combined to obtain the query type vocabulary set of the heterogeneous data set.

2. The heterogeneous data processing method according to claim 1, characterized in that: The performing semantic calculation on the attribute word set and determining the query type word set of the heterogeneous data set according to the result of the semantic calculation includes: For every two attribute words in the attribute word table in the attribute word table set, calculating the semantic similarity of the two attribute words; When the semantic similarity of the two attribute words is less than or equal to a preset similarity threshold, the two attribute words are determined to be dissimilar, and both of the two attribute words are used as query type words; All query type vocabulary sets are combined to obtain the query type vocabulary set of the heterogeneous data set.

3. The heterogeneous data processing method according to claim 1 or 2, characterized in that: The calculating the semantic similarity of the two attribute words includes: The semantic distances between all the semantic primitives of the two attribute words are calculated, and the average of the semantic distances between all the semantic primitives of the two attribute words is taken as the semantic similarity of the two attribute words.

4. The heterogeneous data processing method according to claim 1, characterized in that: The step of performing standardization on the structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data includes: Decomposing the structured heterogeneous data to obtain a data word segmentation table of the structured heterogeneous data; The data words in the data word table are clustered to obtain an attribute word table of the structured heterogeneous data.

5. The heterogeneous data processing method according to claim 1, characterized in that: The step of performing structural processing on the unstructured heterogeneous data to convert the unstructured heterogeneous data into structured heterogeneous data includes: Convert the unstructured heterogeneous data into an XML document; According to the preset conversion relationship, the XML document is mapped into structured heterogeneous data.

6. A heterogeneous data processing device, characterized in that: include: A receiving module, used to receive an uploaded heterogeneous data set and determine the data type of each heterogeneous data in the heterogeneous data set; The data type includes structured or unstructured; A conversion module is used for performing structured processing on each unstructured heterogeneous data of which the data type is unstructured heterogeneous data, and converting the unstructured heterogeneous data into structured heterogeneous data; A processing module, for performing standardization processing on each structured heterogeneous data to obtain an attribute vocabulary of the structured heterogeneous data; A collection module, used for collecting all attribute word lists to form an attribute word list set of the heterogeneous data set; A calculation module, used for performing semantic calculation on the attribute word set, and determining the query type word set of the heterogeneous data set according to the result of the semantic calculation; Wherein, the calculation module includes: A first calculation unit is used to calculate the semantic similarity between any two attribute words in the attribute word table in the attribute word table set; wherein the semantic similarity between the two attribute words is any one of the cosine similarity, Euclidean distance, Manhattan distance, Minkowski distance or Pearson correlation coefficient of the two attribute words; A first judgment unit is used to judge that the two attribute words are similar when the semantic similarity of the two attribute words is greater than a preset similarity threshold, and select any one of the two attribute words as a query type word; The first collection unit is used to collect all query type vocabulary sets to obtain the query type word set of the heterogeneous data set.

7. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the heterogeneous data processing method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the heterogeneous data processing method as described in any one of claims 1 to 5 are implemented.