Heterogeneous data processing method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment and storage medium for processing heterogeneous data. The method includes: obtaining first source data and second source data of a geographic entity to be processed, wherein the first source data and the second source data are heterogeneous data; performing data conversion on the first source data to obtain first target data, and performing data conversion on the second source data to obtain second target data, wherein the data structures of the first target data and the second target data are the same; performing data fusion on the first target data and the second target data to obtain fused data corresponding to the entity to be processed. Thus, by obtaining multi-source data and converting the multi-source data, the converted data can be automatically fused, reducing manual processing and improving the fusion processing efficiency of multi-source data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and computer-readable storage medium for processing heterogeneous data. Background Art

[0002] With the popularization of big data and artificial intelligence technology, massive multi-source heterogeneous data has increased dramatically. When faced with multi-source heterogeneous data processing, traditional data platforms usually use multi-source heterogeneous data in software through certain data processing rules and a large amount of manual assistance to complete data conversion, fusion and output results to achieve heterogeneous data processing, which in turn requires too much human cost, resulting in a more complicated work process and low work efficiency. Therefore, how to better achieve heterogeneous data processing has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of this application is to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a method for processing heterogeneous data. The method can automatically fuse the converted data by acquiring multi-source data and converting the multi-source data, thereby reducing manual processing and improving the fusion processing efficiency of multi-source data.

[0005] The second objective of the present application is to provide a device for processing heterogeneous data.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] To achieve the above-mentioned objectives, the first aspect of the present application proposes a method for processing heterogeneous data, including: obtaining first source data and second source data of a geographic entity to be processed, wherein the first source data and the second source data are heterogeneous data; performing data conversion on the first source data to obtain first target data, and performing data conversion on the second source data to obtain second target data, wherein the data structures of the first target data and the second target data are the same; performing data fusion on the first target data and the second target data to obtain fused data corresponding to the entity to be processed.

[0009] According to the method for processing heterogeneous data of the embodiment of the present application, the first source data and the second source data of the geographic entity to be processed are obtained, wherein the first source data and the second source data are heterogeneous data, and then the first source data is converted to obtain the first target data, and the second source data is converted to obtain the second target data, wherein the data structure of the first target data and the second target data is the same, and then the first target data and the second target data are data fused to obtain the fused data corresponding to the entity to be processed. Thus, by obtaining multi-source data and automatically converting the multi-source data, the converted data can be automatically fused, which improves the fusion processing efficiency of multi-source data and avoids the time-consuming and cumbersome problems caused by manual data fusion.

[0010] To achieve the above-mentioned objectives, the second aspect of the present application proposes a device for processing heterogeneous data, including: a first acquisition module, used to acquire first source data and second source data of a geographic entity to be processed, wherein the first source data and the second source data are heterogeneous data; a first conversion module, used to perform data conversion on the first source data to obtain first target data, and to perform data conversion on the second source data to obtain second target data, wherein the data structures of the first target data and the second target data are the same; a second acquisition module, used to perform data fusion on the first target data and the second target data to obtain fused data corresponding to the entity to be processed.

[0011] According to the heterogeneous data processing device of the embodiment of the present application, the first source data and the second source data of the geographic entity to be processed are obtained, wherein the first source data and the second source data are heterogeneous data, and then the first source data is converted to obtain the first target data, and the second source data is converted to obtain the second target data, wherein the data structure of the first target data and the second target data is the same, and then the first target data and the second target data are data fused to obtain the fused data corresponding to the entity to be processed. Thus, by obtaining multi-source data and automatically converting the multi-source data, the converted data can be automatically fused, which improves the fusion processing efficiency of multi-source data and avoids the time-consuming and cumbersome problems caused by manual data fusion.

[0012] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the tile data migration method described in the first aspect embodiment of the present application is implemented.

[0013] To achieve the above-mentioned objectives, the fourth aspect of the present application provides a computer-readable storage medium, and when the computer program is executed by a processor, the method for processing heterogeneous data described in the first aspect of the present application is implemented.

[0014] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0016] Figure 1 is a flowchart of a method for processing heterogeneous data according to an embodiment of the present application;

[0017] Figure 2 is a flowchart of a method for processing heterogeneous data according to a specific embodiment of the present application;

[0018] Figure 3 is a schematic diagram of a processing result of heterogeneous data according to the present application;

[0019] Figure 4 is a flowchart for determining that all geographic entities have passed the topology check according to a specific embodiment of the present application;

[0020] Figure 5 It is a schematic diagram of the relationship between type information and topology checking rules according to the present application;

[0021] Figure 6 is a schematic diagram representing element overlay in a topological rule according to the present application;

[0022] Figure 7 It is a schematic diagram of storing erroneous data and abnormal data in a topological layer according to the present application;

[0023] Figure 8 is a schematic diagram of the structure of a device for processing heterogeneous data according to an embodiment of the present application;

[0024] Fig. 9 is a schematic diagram of the structure of a device for processing heterogeneous data according to an embodiment of the present application;

[0025] Fig.10 is a schematic diagram of the structure of a device for processing heterogeneous data according to an embodiment of the present application;

[0026] Fig.11 The present invention is a block diagram of an electronic device according to a method for processing heterogeneous data according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0028] In related technologies, when faced with multi-source heterogeneous data processing, traditional data platforms usually use multi-source heterogeneous data in software through certain data processing rules and a large amount of manual assistance to complete data conversion, fusion and output results to achieve heterogeneous data processing.

[0029] Among them, before data conversion and fusion, all data formats, layers, and field attributes need to be manually identified and screened, which makes the processing time-consuming and cumbersome.

[0030] Among them, after the data is manually identified and screened, the software is used to convert the relevant data, and then the data layer field name and attribute content are processed, and finally the data is extracted and the topology is checked. As a result, the software in the whole process is operated independently and lacks a process-based operation mode, which makes the process cumbersome and inefficient.

[0031] Among them, in data topology inspection, data topology inspection and data topology processing all require manual participation, especially topology error processing, which is very inefficient. Data fusion all requires manual one-to-one comparison, selection of correct data, and deletion of erroneous data.

[0032] Among them, after the data processing is completed, the corresponding result layer data needs to be exported separately. If there are more data layers, more working time will be spent, resulting in low efficiency.

[0033] Therefore, due to the large proportion of the existing manual processing stage, the human cost invested is too high, and the data quality cannot be fully guaranteed. When the data is updated or the new data is merged, the manual processing and other operations will be repeated, resulting in low processing efficiency of heterogeneous data. Therefore, in order to solve the above technical problems, this application proposes a method, device, electronic device and computer-readable storage medium for processing heterogeneous data.

[0034] Figure 1 It is a flowchart of a method for processing heterogeneous data according to an embodiment of the present application. It should be noted that the execution subject of the method for processing heterogeneous data provided in this embodiment is a processing device for heterogeneous data, which can be configured in an electronic device, wherein the electronic device can be a hardware device such as a server, or software installed on a hardware device.

[0035] like Figure 1As shown, the method for processing heterogeneous data may include:

[0036] Step 101: Acquire first source data and second source data of a geographic entity to be processed.

[0037] Among them, the first source data and the second source data are heterogeneous data.

[0038] Among them, heterogeneous data can be understood as data with different structures. Among them, heterogeneous data can be reflected in the following five levels: 1) Heterogeneity of computer architecture: the physical storage of data comes from computers with different architectures, such as: mainframes, minicomputers, workstations, PCs or embedded systems; 2) Heterogeneity of operating systems: the storage of data comes from different operating systems, such as: Unix, Windows, Linux, OS / 400, etc.; 3) Heterogeneity of data formats: the storage management mechanism of data is different, which can be a relational database system, such as: Oracle, SQL Server, DB2, etc., or a two-dimensional data file line, such as: txt, CSV, XLS, etc.; 4) Heterogeneity of data storage location: data is stored in dispersed physical locations. This situation mostly occurs in large organizations, such as: sales data is stored in the local sales systems of multiple branches in Beijing, Shanghai, Japan, South Korea, etc.; 5) Heterogeneity of logical models of data storage: data is stored and maintained in different business logics, so that data with the same meaning has heterogeneous performance; for example, there are inconsistent department codes in independent sales systems and independent procurement systems.

[0039] In an embodiment of the present application, the data providers of the first source data and the second source data are different. For example, the source data is geographic group data, the first source data may be data provided by a first map system, and the second source data may be data provided by a second map system.

[0040] For example, the first source data and the second source data of the geographic entity to be processed may be obtained by means of web crawlers, internal data collection, or manual input.

[0041] Among them, geographic entities can be understood as entities in geographic databases, which refer to phenomena that can no longer be divided into the same category in the real world. For example, a city can be regarded as a geographic entity and can be divided into several parts, but these parts are not called cities, but can only be called districts and streets.

[0042] Step 102: performing data conversion on the first source data to obtain first target data, and performing data conversion on the second source data to obtain second target data, wherein the data structures of the first target data and the second target data are the same.

[0043] In order to reduce data calculation and thus improve reading efficiency, correspondingly, in an embodiment of the present application, relevant conversion parameters may be configured in advance for different data sources.

[0044] For example, after obtaining the first source data and the second source data of the geographic entity to be processed, the first source data can be converted according to conversion parameters of the first source data and the second source data to obtain first target data, and the second source data can be converted to obtain second target data.

[0045] Step 103: fusing the first target data and the second target data to obtain fused data corresponding to the entity to be processed.

[0046] In order to solve the problem that different data sources have the same name but different coordinates within a certain range, in an embodiment of the present application, after obtaining the first target data and the second target data, the first target data and the second target data can be fused to obtain fused data corresponding to the entity to be processed.

[0047] In an embodiment of the present application, the first target data and the second target data can be compared to determine the identical data in the first target data and the second target data and the data in which the first target data and the second target data exist separately, and then the identical data and the data in which the first target data and the second target data exist separately are merged to obtain fused data corresponding to the entity to be processed.

[0048] In order to achieve automatic data update, correspondingly, after the data processing is completed, the data is automatically output to the specified directory and data format according to the requirements of hierarchical classification. In one embodiment of the present application, after the data fusion is completed, the data format and storage location corresponding to the geographic entity to be processed can be obtained, and then the fused data can be formatted according to the data format, and the converted fused data can be stored according to the storage location, so that if the source data changes, the updated data can be re-converted and output without further configuration and processing.

[0049] According to the method for processing heterogeneous data of the embodiment of the present application, the first source data and the second source data of the geographic entity to be processed are obtained, wherein the first source data and the second source data are heterogeneous data, and then the first source data is converted to obtain the first target data, and the second source data is converted to obtain the second target data, wherein the data structure of the first target data and the second target data is the same, and then the first target data and the second target data are data fused to obtain the fused data corresponding to the entity to be processed. Thus, by obtaining multi-source data and converting the multi-source data, the converted data can be automatically fused, reducing manual processing and improving the fusion processing efficiency of multi-source data.

[0050] In order to make it easier for those skilled in the art to understand this application, Figure 2 1 is a flowchart of a method for processing heterogeneous data according to a specific embodiment of the present application. It should be noted that the second embodiment is a further refinement or optimization of the first embodiment. Figure 2 As shown, the method for processing heterogeneous data may include:

[0051] Step 201: Acquire first source data and second source data of a geographic entity to be processed.

[0052] Among them, the first source data and the second source data are heterogeneous data.

[0053] Step 202: Perform data conversion on the first source data to obtain first target data, and perform data conversion on the second source data to obtain second target data.

[0054] The first target data and the second target data have the same data structure.

[0055] Step 203: Determine, based on the first target data and the second target data, whether all geographic entities pass the topology check.

[0056] In the embodiment of the present application, the type information corresponding to the geographic entity to be processed can be determined, and the topology check rule corresponding to the type information can be obtained. Then, according to the first target data, it can be determined whether the geographic entity passes the topology check rule. According to the second target data, it can be determined whether the geographic entity passes the topology check rule. When it is determined based on the first target data and the second target data that the geographic entity passes the topology check, the first target data and the second target data are fused to obtain the fused data corresponding to the entity to be processed. The specific implementation process can be referred to the subsequent embodiments.

[0057] In one embodiment of the present application, when it is determined based on the first target data and the second target data that both geographic entities have failed the topological check, the data that have failed the topological check can be input into the manual processing and inspection link, and after confirming that there is no problem with the processing, the first target data and the second target data are fused to obtain the fused data corresponding to the entity to be processed.

[0058] Step 204: Determine, based on the first target data and the second target data, whether the first target data and the second target data both pass the attribute check.

[0059] In an embodiment of the present application, an attribute check rule corresponding to the geographic entity to be processed is obtained, and then an attribute check is performed on the first target data and the second target data according to the attribute check rule to determine whether both the first target data and the second target data pass the attribute check. When both the first target data and the second target data pass the attribute check, a step of performing data fusion on the first target data and the second target data to obtain fused data corresponding to the entity to be processed is performed.

[0060] Among them, in one embodiment of the present application, the attribute checking rules include but are not limited to verifying the string, checking the unique ID, comparing table attributes, composite checking, checking the correctness of attribute values, etc.

[0061] In one embodiment of the present application, when it is determined based on the first target data and the second target data that both geographic entities have failed the attribute check, the data that has failed the attribute check can be input into the manual processing and inspection link, and after confirming that there is no problem with the processing, the first target data and the second target data are fused to obtain the fused data corresponding to the entity to be processed.

[0062] Step 205 , based on the geographic entities passing the topology check and the first target data and the second target data passing the attribute check, data fusion is performed on the first target data and the second target data to obtain fused data corresponding to the entity to be processed.

[0063] In an embodiment of the present application, the first target data and the second target data can be compared to determine the identical data in the first target data and the second target data and the data in which the first target data and the second target data exist separately, and then the identical data and the data in which the first target data and the second target data exist separately are merged to obtain fused data corresponding to the entity to be processed.

[0064] For example, the coordinates and attributes of both data can be compared, duplicates can be removed, and then merged. For example, the specific implementation process can be achieved in the following ways: 1: Mark the data coordinates and attributes of the data within a specific grid area; 2: Translate the specific grid 5 kilometers in the four directions of east, west, south, and north respectively, and mark it according to the data attributes and coordinates; 3: Then remove duplicates based on the coordinates and data attribute marks; 4: Merge the data after duplicate removal. For example, the processing result can be as follows Figure 3 As shown, before processing ("rectangle" represents the result before de-duplication), after processing ("circle" represents the result after de-duplication).

[0065] For example, in one embodiment of the present application, after data processing is completed, the data can be re-merged according to the layer name or attribute information of the data, or the data can be re-divided into different layers for output. For example, if the data source is national school POI data from two different sources, after the school data from the two sources are processed in the previous stage, the data can be merged or layered according to actual needs, wherein the merger can be based on data type (for example, large, medium and small types) or data area (for example, provinces, cities and counties across the country); the layering can be based on data type (for example, large, medium and small types) or data area (for example, provinces, cities and counties across the country).

[0066] According to the method for processing heterogeneous data of the embodiment of the present application, the first source data and the second source data of the geographic entity to be processed are obtained, and then the first source data is converted to obtain the first target data, and the second source data is converted to obtain the second target data, and then according to the first target data and the second target data, it is determined that the geographic entities have passed the topological check and the attribute check, and then the first target data and the second target data are fused to obtain the fused data corresponding to the entity to be processed. Thus, by obtaining multi-source data and converting the multi-source data, the converted data can be automatically fused, reducing manual processing and improving the fusion processing efficiency of multi-source data, and multi-source heterogeneous data can realize configuration reuse, improve data update efficiency, and can be fused and converted and output in large quantities of multi-source heterogeneous data.

[0067] In order for those skilled in the art to understand this application more easily, Figure 4 is a flowchart of determining that all geographic entities have passed the topology check according to a specific embodiment of the present application, such as Figure 4 As shown, the specific implementation process of determining that all geographic entities have passed the topology check can refer to the following embodiments: Step 401, by determining the type information corresponding to the geographic entity to be processed, and obtaining the topology check rule corresponding to the type information.

[0068] The type information corresponding to the geographic entity to be processed includes but is not limited to point entity, line entity, surface entity and volume entity.

[0069] The corresponding relationship between the type information and the topology checking rules can be established in advance to form a relationship diagram between the type information and the topology checking rules. For example, the relationship between the type information and the topology checking rules can be as follows: Figure 5 shown.

[0070] For example, when the correspondence between type information and topology check rules is established in advance, a new topology rule can be created in advance, the required check features in the feature set can be selected, and then the topology rule can be added. The topology rule defines the spatial relationship allowed between features. Topology rules can control the relationship between features in a feature class, between features in different feature classes, and between subtypes of features.

[0071] For example, the topology rule "Must Not Overlap" is used to manage the integrity of features in the same feature class. If two features overlap, the overlapping geometry is displayed in a prominent color (such as red), such as Figure 6 As shown, the overlapping area between adjacent polygons is the red area, and the overlapping area between two lines is the red linear segment portion).

[0072] It should be noted that topology rules can also be defined between subtypes of feature classes. For example, suppose there are two subtypes of street line features, normal streets (connected to other streets at two nodes) and cul-de-sac streets (dead ends at one node), where topology rules can require street features to be connected to other street features at both ends, unless the street belongs to the cul-de-sac subtype.

[0073] Step 402: Determine whether the geographic entity passes the topology check rule based on the first target data.

[0074] For example, a topology check consists of four processes: 1) cracking and clustering feature vertices to find coincident features that share the same location (have common coordinates); 2) inserting common coordinate vertices into coincident features that share geometry; 3) running a series of integrity checks to determine if the rules defined for the topology are violated; and 4) creating an error log of potential topology errors in the feature dataset.

[0075] Step 403: Determine whether the geographic entity passes the topology check rule based on the second target data.

[0076] In one embodiment of the present application, after the topology check is completed according to the first target data and the second target data, the erroneous data can be corrected according to the check result, wherein the erroneous data will record the location where the topology error is found during the topology check. Some errors are acceptable, in which case the erroneous element can be marked as an exception. For example, Figure 7 In the example shown, the error data ( Figure 7 Marked "1" in the middle) and abnormal data ( Figure 7 Marked "2" in the figure) are stored as features in the topology layer and can be used to present and manage features that do not need to comply with topology rules.

[0077] Step 404 , when it is determined based on the first target data and the second target data that both geographic entities pass the topology check, data fusion is performed on the first target data and the second target data to obtain fused data corresponding to the entity to be processed.

[0078] Corresponding to the methods for processing heterogeneous data provided in the above-mentioned embodiments, an embodiment of the present application further provides a device for processing heterogeneous data. Since the device for processing heterogeneous data provided in the embodiment of the present application corresponds to the methods for processing heterogeneous data provided in the above-mentioned embodiments, the implementation of the method for processing heterogeneous data is also applicable to the device for processing heterogeneous data provided in this embodiment, and will not be described in detail in this embodiment. Figure 8 It is a structural diagram of a device for processing heterogeneous data according to an embodiment of the present application.

[0079] like Figure 8 As shown, the heterogeneous data processing device 800 may include: a first acquisition module 810 , a first conversion module 820 and a second acquisition module 830 .

[0080] Specifically, the first acquisition module 810 is used to acquire first source data and second source data of the geographic entity to be processed, wherein the first source data and the second source data are heterogeneous data.

[0081] The first conversion module 820 is used to perform data conversion on the first source data to obtain first target data, and to perform data conversion on the second source data to obtain second target data, wherein the data structures of the first target data and the second target data are the same.

[0082] The second acquisition module 830 is used to perform data fusion on the first target data and the second target data to obtain fused data corresponding to the entity to be processed. As an example, the second acquisition module 830 includes: a first determination unit, used to compare the first target data and the second target data to determine the same data in the first target data and the second target data and the data that exists separately in the first target data and the second target data; a first acquisition unit, used to merge the same data and the data that exists separately in the first target data and the second target data to obtain the fused data corresponding to the entity to be processed.

[0083] In one embodiment of the present application, Fig. 9As shown, before the second acquisition module 830, the device 800 also includes: a third acquisition module 840, used to determine the type information corresponding to the geographic entity to be processed, and obtain the topology check rule corresponding to the type information; a first determination module 850, used to determine whether the geographic entity passes the topology check rule based on the first target data; a second determination module 860, used to determine whether the geographic entity passes the topology check rule based on the second target data; and a first execution module 870, used to perform data fusion of the first target data and the second target data to obtain fused data corresponding to the entity to be processed when it is determined based on the first target data and the second target data that the geographic entity passes the topology check.

[0084] In one embodiment of the present application, Fig.10 As shown, before the second acquisition module 830, the device 800 also includes: a fourth acquisition module 880, used to obtain the attribute check rule corresponding to the geographic entity to be processed; a third determination module 890, used to perform an attribute check on the first target data and the second target data according to the attribute check rule to determine whether the first target data and the second target data both pass the attribute check; a second execution module 8110, used to perform a step of data fusion on the first target data and the second target data to obtain fused data corresponding to the entity to be processed when the first target data and the second target data both pass the attribute check.

[0085] In one embodiment of the present application, the device also includes: a fifth acquisition module, used to obtain the data format and storage location corresponding to the geographic entity to be processed; a second conversion module, used to convert the format of the fused data according to the data format, and store the converted fused data according to the storage location.

[0086] According to the heterogeneous data processing device of the embodiment of the present application, the first source data and the second source data of the geographic entity to be processed are obtained, wherein the first source data and the second source data are heterogeneous data, and then the first source data is converted to obtain the first target data, and the second source data is converted to obtain the second target data, wherein the data structure of the first target data and the second target data is the same, and then the first target data and the second target data are data fused to obtain the fused data corresponding to the entity to be processed. Thus, by obtaining multi-source data and converting the multi-source data, the converted data can be automatically fused, reducing manual processing and improving the fusion processing efficiency of multi-source data.

[0087] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0088] like Fig.11 , is a block diagram of an electronic device according to a method for processing heterogeneous data according to an embodiment of the present application.

[0089] like Fig.11 As shown, the electronic device includes:

[0090] Memory 1001 , processor 1002 , and computer instructions stored in the memory 1001 and executable on the processor 1002 .

[0091] When the processor 1002 executes the instructions, the method for processing heterogeneous data provided in the above embodiment is implemented.

[0092] Furthermore, the electronic device further comprises:

[0093] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002 .

[0094] The memory 1001 is used to store computer instructions that can be executed on the processor 1002 .

[0095] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0096] The processor 1002 is used to implement the heterogeneous data processing method described in the above embodiment when executing the program.

[0097] If the memory 1001, the processor 1002 and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001 and the processor 1002 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0098] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can communicate with each other through an internal interface.

[0099] The processor 1002 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0100] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0101] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0102] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0103] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0104] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0105] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0106] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0107] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for processing heterogeneous data, characterized in that: include: Acquire first source data and second source data of a geographic entity to be processed, wherein the first source data and the second source data are heterogeneous data; Automatically converting the first source data and the second source data to obtain first target data and second target data, wherein the first target data and the second target data have the same data structure; Automatically fusing the first target data and the second target data to obtain fused data corresponding to the geographic entity to be processed; Obtaining the data format and storage location corresponding to the geographic entity to be processed; Converting the format of the fused data according to the data format, and storing the converted fused data according to the storage location; The method also includes: after determining the first target data and the second target data, determining whether the geographic entity passes the topology check rule and the attribute check rule according to the first target data, and determining whether the geographic entity passes the topology check rule and the attribute check rule according to the second target data; when it is determined based on the first target data and the second target data that the geographic entity passes both the topology check and the attribute check, automatically performing data fusion of the first target data and the second target data to obtain fused data corresponding to the geographic entity to be processed; wherein the attribute check includes verifying the string, checking the unique ID, comparing the table attributes, compound checking, and checking the correctness of the attribute value.

2. The method according to claim 1, characterized in that Automatically fusing the first target data and the second target data to obtain fused data corresponding to the geographic entity to be processed, including: Comparing the first target data with the second target data to determine the same data in the first target data and the second target data and the data that exists separately in the first target data and the second target data; The identical data and the first target data and the second target data that exist separately are merged to obtain fused data corresponding to the geographic entity to be processed.

3. A device for processing heterogeneous data, characterized in that: include: A first acquisition module is used to acquire first source data and second source data of a geographic entity to be processed, wherein the first source data and the second source data are heterogeneous data; A first conversion module is used to automatically perform data conversion on the first source data and the second source data to obtain first target data and second target data, wherein the first target data and the second target data have the same data structure; a second acquisition module is used to automatically perform data fusion on the first target data and the second target data to obtain fused data corresponding to the geographic entity to be processed; A fifth acquisition module, used to acquire the data format and storage location corresponding to the geographic entity to be processed; a second conversion module, configured to convert the format of the fused data according to the data format, and store the converted fused data according to the storage location; The device also includes: after determining the first target data and the second target data, determining whether the geographic entity passes the topology check rule and the attribute check rule according to the first target data, and determining whether the geographic entity passes the topology check rule and the attribute check rule according to the second target data; when it is determined based on the first target data and the second target data that the geographic entity passes both the topology check and the attribute check, automatically performing data fusion of the first target data and the second target data to obtain fused data corresponding to the geographic entity to be processed; wherein the attribute check includes verifying the string, checking the unique ID, comparing the table attributes, compound checking, and checking the correctness of the attribute value.

4. The device according to claim 3, characterized in that The second acquisition module includes: A first determining unit, configured to compare the first target data with the second target data to determine identical data in the first target data and the second target data and data that exists separately in the first target data and the second target data; The first acquisition unit is used to merge the identical data and the first target data and the second target data that exist separately, so as to obtain fused data corresponding to the geographic entity to be processed.

5. An electronic device comprising: Memory, processor; The memory stores computer instructions, and when the computer instructions are executed by the processor, the method for processing heterogeneous data as claimed in claim 1 or 2 is implemented.

6. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the heterogeneous data processing method according to claim 1 or 2.

Images