Multi-source heterogeneous data collaboration method and device

Abstract

The application relates to a multi-source heterogeneous data cooperation method, which comprises the following steps: obtaining target data from a target data source; analyzing relevant information of the target data source; using one or more prefabricated operators to arrange a data cooperation process of the target data according to an analysis result of the target data source; and parsing and processing the target data based on the data cooperation process to generate standardized data. Through the technical scheme, a flexible and variable data cooperation mechanism can be realized, and data cooperation capability is enhanced.

Description

Technical Field

[0001] This application generally relates to the field of computer technology, and more particularly to a method and apparatus for multi-source heterogeneous data collaboration. Background Technology

[0002] Currently, governments and industries at all levels have carried out extensive informatization work, but the silo effect of management systems and data is evident across industries, with widespread duplication of construction and data inconsistencies. The key issue lies in the unresolved critical technical problems of data collaboration among various business systems. Data collaboration involves multiple departments, each with its own business systems. The data generated by these systems primarily runs vertically within various networks; some systems integrate data from internal departments, but these often involve simple applications of single data points, lacking spatial data collaboration and integration with other public resources, making it difficult to support in-depth applications. Furthermore, in the field of remote sensing and mapping, due to the high level of specialization and security concerns, database collaboration among various business systems is extremely difficult. Faced with massive amounts of data, manual entry, physical copying, manual uploading, and direct database connections consume significant manpower, are highly inefficient, and cannot guarantee data accuracy and timeliness.

[0003] Therefore, existing technologies have the disadvantages of having a single collaborative mechanism and weak collaborative capabilities, and cannot meet the collaborative needs of data that are multi-sourced, heterogeneous, stored in a dispersed manner, and have inconsistent rates of change. Summary of the Invention

[0004] To address the aforementioned technical problems in the prior art, this application provides a multi-source heterogeneous data collaboration method and apparatus to achieve a flexible and variable data collaboration mechanism and enhance data collaboration capabilities.

[0005] According to a first aspect of this application, a multi-source heterogeneous data collaboration method is provided, comprising: acquiring target data from a target data source; analyzing relevant information of the target data source, wherein the relevant information includes the architecture, network environment, communication protocol, development technology, authentication and authorization, request process or response result of the target data source, to obtain analysis results of the target data source; based on the analysis results of the target data source, using one or more pre-built operators, arranging a data collaboration process for the target data, wherein the pre-built operators include control operators and data processing operators, the control operators including request operators, variable operators, loop operators, function operators, output operators, file operators, SQL operators and sub-process operators, and the data processing operators including image recognition operators, address matching operators, region matching operators, IP matching operators, image registration operators, coordinate transformation operators, format conversion operators and data cleaning operators; and based on the data collaboration process, parsing and processing the target data to generate standardized data.

[0006] In one embodiment, the step of parsing and processing the target data based on the data collaboration process to generate standardized data includes: parsing the target data; determining whether the target data contains spatial data based on the parsing result; determining whether the spatial data requires coordinate system transformation in response to the target data containing spatial data; and performing coordinate system transformation using a coordinate transformation operator in response to the spatial data requiring coordinate system transformation to obtain the processed target data.

[0007] In one embodiment, the step of parsing and processing the target data based on the data collaboration process to generate standardized data further includes: determining whether the target data contains image data based on the parsing result; in response to the target data containing image data, using the image recognition operator to determine whether the image data contains ground features; and in response to the spatial data containing ground features, using the image registration operator to perform image registration to obtain processed target data.

[0008] In one embodiment, the step of parsing and processing the target data based on the data collaboration process to generate standardized data further includes: determining whether the target data contains address description information based on the parsing result; and, in response to the target data containing address description information, performing address matching using the address matching operator to obtain processed target data.

[0009] In one embodiment, the step of parsing and processing the target data based on the data collaboration process to generate standardized data further includes: determining whether the target data contains zoning information based on the parsing result; and performing zoning matching using a zoning matching operator in response to the target data containing zoning information to obtain processed target data.

[0010] In one embodiment, the step of parsing and processing the target data based on the data collaboration process to generate standardized data further includes: determining whether the processed target data needs cleaning; and in response to the need for cleaning the processed target data, using a data cleaning operator to clean the processed data to obtain the standardized data.

[0011] In one embodiment, the method further includes: determining data update triggering conditions based on the update frequency and mechanism of the target data source; and updating the standardized data according to the data update triggering conditions.

[0012] In one embodiment, obtaining target data from a target data source includes: determining whether the target data source requires login; and in response to the target data source requiring login, performing a simulated login to obtain the target data.

[0013] In one embodiment, the method further includes: providing data analysis services based on the standardized data, wherein the data analysis services include at least one of data interface services, big data spatial analysis services, map services, reporting services, and visualization services.

[0014] According to a second aspect of this application, a multi-source heterogeneous data collaboration device is provided, including a memory and a processor, wherein the memory stores computer-executable instructions, characterized in that, when executed by the processor, the computer-executable instructions implement the multi-source heterogeneous data collaboration method according to the first aspect of this application.

[0015] The technical solution of this application has the following beneficial technical effects:

[0016] According to the technical solution of this application, in the face of complex data sources, control nodes such as requests, variables, loops, functions, outputs, files, and SQL, as well as data processing nodes such as image recognition, address matching, area matching, IP matching, image registration, coordinate transformation, and format conversion are encapsulated to generate independent operators. Based on the type of data source and other relevant information, different operators are arranged and scheduled to achieve flexible and variable data collaboration and enhance data collaboration capabilities. Attached Figure Description

[0017] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0018] Figure 1 This is a flowchart of a multi-source heterogeneous data collaboration method according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating the data collaboration process arrangement according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a data collaboration process according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a multi-source heterogeneous data collaboration device according to an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] It should be understood that when the terms "first," "second," etc., are used in the claims, description, and drawings of this application, they are only used to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" used in the description and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0024] According to the first aspect of this application, this application provides a multi-source heterogeneous data collaboration method, which is applied to collaboration scenarios of data with characteristics such as multi-source heterogeneity, dispersed storage, and inconsistent change rates.

[0025] Figure 1 This is a flowchart of a multi-source heterogeneous data collaboration method according to an embodiment of this application. For example... Figure 1 As shown, the method includes steps S101 to S104, which are explained in detail below:

[0026] S101, Obtain target data from the target data source.

[0027] As an example, the target data source is various vertical management systems, professional systems, and closed management systems existing in ubiquitous networks such as the Internet, government networks, and business networks; the target data is web page data, document data, meeting minutes, surveying and mapping data, etc., which are not limited in this application.

[0028] Some data may require logging into its server to be retrieved. In this case, retrieving the target data from the target data source includes: determining whether the target data source requires login; and in response to the target data source requiring login, performing a simulated login to retrieve the target data.

[0029] S102, analyze the relevant information of the target data source, wherein the relevant information includes the architecture, network environment, communication protocol, development technology, authentication and authorization, request process or response result of the target data source, so as to obtain the analysis result of the target data source.

[0030] Specifically, before processing the target data, it's necessary to understand the relevant information about the target data source, including its architecture, network environment, communication protocols, development technologies, authentication and authorization, request processes, and response results. This information helps us better understand the structure and operation of the target data source, thus enabling us to process the target data more effectively later.

[0031] For example, understanding the architecture of the target data source can help us determine the components of the data source and the relationships between them; understanding the network environment can help us determine the availability and stability of the data source; understanding the communication protocol can help us understand the data transmission method and data format; understanding the development technology can help us understand the development process and technical implementation of the data source; understanding authentication and authorization can help us understand the permissions and security of accessing the data source; and understanding the request process or response result can help us understand the operation process of the data source and the data return result.

[0032] S103, based on the analysis results of the target data source, a data collaboration process for the target data is constructed using one or more pre-built operators. The pre-built operators include control operators and data processing operators. The control operators include request operators, variable operators, loop operators, function operators, output operators, file operators, SQL operators, and sub-process operators. The data processing operators include image recognition operators, address matching operators, region matching operators, IP matching operators, image registration operators, coordinate transformation operators, format conversion operators, and data cleaning operators.

[0033] Specifically, based on the results obtained from the previous analysis of the target data source, pre-built operators can be used to construct a data collaboration process to process and transform the target data. Figure 2 This is a schematic diagram of data collaborative workflow arrangement according to an embodiment of this application. Pre-defined operators include control operators and data processing operators. Control operators are used to control the execution order and flow control of the data flow; for example, request operators are used to issue requests, variable operators are used to define variables, loop operators are used to repeatedly execute certain operations, function operators are used to define functions, output operators are used to output results, file operators are used to read and write files, SQL operators are used to execute SQL queries, and sub-flow operators are used to call other flows. Data processing operators are used to process and transform data; for example, image recognition operators are used to process and recognize images, address matching operators are used to perform address matching, region matching operators are used to perform region matching, IP matching operators are used to perform IP address matching, image registration operators are used to perform image registration, coordinate transformation operators are used to perform coordinate transformation, format conversion operators are used to perform format conversion, and data cleaning operators are used to perform data cleaning.

[0034] Based on the analysis results of the target data source, appropriate pre-built operators can be selected to construct a data collaboration workflow to achieve effective data processing and transformation. For example, if the target data source contains a large amount of image data, image recognition operators can be used to process and recognize the images. If it is necessary to convert the data to different formats, format conversion operators can be used. If it is necessary to clean the data and remove duplicates, data cleaning operators can be used.

[0035] The operators described above are demarcated and encapsulated based on specific application scenarios and requirements. During operator demarcation, the dependencies between operators need to be considered, dividing operators into independent modules so that the inputs and outputs of each module can be clearly defined and described. During operator encapsulation, each module needs to be encapsulated into a reusable component, allowing it to be used by other applications or systems. Furthermore, to improve the maintainability and scalability of operators, thorough documentation and testing are essential.

[0036] S104, Based on the data collaboration process, the target data is parsed and processed to generate standardized data.

[0037] After the data collaboration process is orchestrated, the target data can be parsed and processed based on the data collaboration process. Figure 3 This is a schematic diagram of a data collaboration process according to an embodiment of this application. Figure 3 As shown, after obtaining the target data, the method includes: parsing the target data; determining whether the target data contains spatial data based on the parsing result; in response to the target data containing spatial data, determining whether the spatial data requires coordinate system transformation; and in response to the spatial data requiring coordinate system transformation, performing coordinate system transformation using a coordinate transformation operator to obtain processed target data.

[0038] Specifically, there are many common coordinate systems used in spatial data, such as the WGS84 coordinate system, the GCJ-02 coordinate system, and the BD-09 coordinate system. The 2000 coordinate system refers to the 2000 geodetic coordinate system released by the State Bureau of Surveying and Mapping, also known as the "National 2000 Coordinate System." It is a new version of the geodetic coordinate system released by the State Bureau of Surveying and Mapping in 2000, used in fields such as land resource surveys, geographic information systems, and geospatial information services. This application automatically converts spatial data from other coordinate systems to the 2000 coordinate system to facilitate subsequent data processing.

[0039] like Figure 3As shown, the above data collaboration process further includes: determining whether the target data contains image data based on the parsing result; in response to the target data containing image data, using the image recognition operator to determine whether the image data contains ground features; and in response to the spatial data containing ground features, using the image registration operator to perform image registration to obtain processed target data.

[0040] Specifically, geographic features refer to the shape, size, location, and distribution of various natural and man-made objects on the Earth's surface. Geographic features include mountains, rivers, lakes, oceans, buildings, roads, bridges, forests, grasslands, and so on. Observing and analyzing geographic features helps us better understand the natural and human environment of the Earth's surface, providing fundamental data for geographical research, resource development, and urban planning.

[0041] Image registration refers to the geometric transformation of two or more images to make them overlap as much as possible in space. The purpose of image registration is to compare and analyze images acquired from different sources, at different times, or by different sensors. Image registration is a crucial technique in fields such as remote sensing and geological exploration. Commonly used image registration methods include feature-based registration, region-based registration, and phase-correlation-based registration. Image registration helps us better understand and analyze image data, supporting scientific research and practical applications. In this application, image registration is performed using an image registration operator.

[0042] like Figure 3 As shown, the above data collaboration process further includes: determining whether the target data contains address description information based on the parsing result; and, in response to the target data containing address description information, performing address matching using the address matching operator to obtain the processed target data.

[0043] Address matching refers to comparing an input address with a known address database to find the best match. Typically, address matching is based on string matching methods, determining the degree of match by comparing keywords in the address. Address matching is commonly used in scenarios such as Geographic Information Systems (GIS) to help quickly and accurately locate and match address information. During the address matching process, issues such as address normalization, standardization, and uniformity need to be considered to improve the accuracy and stability of the matching. In this application, address matching is performed using an address matching operator.

[0044] like Figure 3 As shown, the above data collaboration process also includes: determining whether the target data contains zoning information based on the parsing result; and, in response to the target data containing zoning information, performing zoning matching using a zoning matching operator to obtain the processed target data.

[0045] Specifically, zoning matching refers to comparing an input geographic location with a known zoning database to find the best-matching zoning information. Typically, zoning matching is based on spatial location methods, determining the degree of match by comparing the latitude and longitude coordinates or administrative divisions of the input geographic location. Zoning matching is commonly used in scenarios such as Geographic Information Systems (GIS) to help quickly and accurately locate and match geographic location information. In this application, zoning matching is performed using a zoning matching operator.

[0046] like Figure 3 As shown, after the above operator operations are completed, the data collaboration process further includes: determining whether the processed target data needs to be cleaned; in response to the need for cleaning the processed target data, using a data cleaning operator to clean the processed data to obtain the standardized data.

[0047] Data cleaning refers to the preprocessing of data to make it conform to analytical needs and standards. In practical applications, raw data often contains irregular, incomplete, inaccurate, duplicate, or useless data, which may affect the results of subsequent analysis. Therefore, data cleaning is one of the important steps in data analysis.

[0048] The main purpose of data cleaning is to remove noise, outliers, and duplicate values ​​from the data, fill in missing values, and perform format conversion and standardization. Specifically, the steps of data cleaning include: removing duplicate values: deduplicating data to avoid affecting subsequent analysis; handling missing values: filling or deleting missing values ​​to ensure data integrity; handling outliers: detecting and handling outliers to avoid affecting subsequent analysis; format conversion: converting data into a format that meets analytical requirements, such as converting date formats to standard formats; and standardization: performing unit conversions or standardization to ensure data consistency and comparability. This application uses data cleaning operators to clean the target data to generate standardized data for subsequent processing.

[0049] After generating the standardized data, the method further includes: providing data analysis services based on the standardized data, wherein the data analysis services include at least one of data interface services, big data spatial analysis services, map services, reporting services, and visualization services.

[0050] Specifically, data interface services refer to services that provide data interfaces, allowing users to obtain data through APIs. This data can be public data, internal enterprise data, or third-party data. Users can obtain data through data interface services and perform secondary development and analysis.

[0051] Big data spatial analysis services refer to services that provide big data storage and analysis, allowing users to store massive amounts of data in the cloud and then analyze and mine the data using data analysis tools. These services typically include functions such as data storage, data processing, data analysis, and data mining.

[0052] Map services refer to services that provide map services, allowing users to display and analyze data on maps. These services typically include functions such as map data collection, map data processing, and map data display.

[0053] Reporting services refer to services that provide reporting capabilities, allowing users to create and display data reports using reporting tools. These services typically include report design, data import, and report display functions.

[0054] Visualization services refer to services that provide data visualization, allowing users to visualize and display data using visualization tools. These services typically include the design of data visualization tools, data import, and visualization display functions.

[0055] The above services can be provided through a service engine. These services can help users better process and analyze data, improving the efficiency and accuracy of data analysis.

[0056] Furthermore, when the target data is updated, the method further includes: determining data update triggering conditions based on the update frequency and mechanism of the target data source; and updating the standardized data according to the data update triggering conditions. Data updates ensure the freshness and real-time nature of the data, thereby providing a reliable data foundation for subsequent data analysis.

[0057] The technical principles and implementation details of the multi-source heterogeneous data collaboration method of this application have been described above through specific embodiments. According to the technical solution of this application, facing complex data sources, control nodes such as requests, variables, loops, functions, outputs, files, and SQL, as well as data processing nodes such as image recognition, address matching, region matching, IP matching, image registration, coordinate transformation, and format conversion, are encapsulated to generate independent operators. Based on information such as the type of data source, different operators are arranged and scheduled, thereby achieving flexible and variable data collaboration and enhancing data collaboration capabilities.

[0058] According to a second aspect of this application, this application also provides a multi-source heterogeneous data collaboration device. Figure 4This is a schematic diagram of a device 40 integrating an electronic whiteboard into a collaborative office system according to an embodiment of this application. The device includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the multi-source heterogeneous data collaboration method according to the first aspect of this application. The device also includes other components well-known to those skilled in the art, such as a communication bus and a communication interface. Their configuration and functions are known in the art and will not be described further here.

[0059] In this application, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this application can be implemented using computer-readable / executable instructions that can be stored or otherwise retained by such a computer-readable medium.

[0060] Based on the above description in this specification, those skilled in the art will also understand that the terms used, such as "upper" and "lower," which indicate orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings of this specification. They are only for the purpose of facilitating the explanation of the present application and simplifying the description, and do not imply that the device or element involved must have the specific orientation, or be constructed and operated in a specific orientation. Therefore, the above-mentioned orientation or positional relationship terms should not be understood or interpreted as a limitation on the present application.

[0061] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0062] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-source heterogeneous data collaboration method, characterized in that, The method comprises: obtaining target data from a target data source; analyzing relevant information of the target data source, wherein the relevant information comprises architecture, network environment, communication protocol, development technology, authentication and authorization, request flow or response result of the target data source, to obtain an analysis result of the target data source; based on the analysis result of the target data source, using one or more pre-made operators to compile a data collaboration flow of the target data, wherein the pre-made operators comprise control operators and data processing operators, the control operators comprise request operators, variable operators, loop operators, function operators, output operators, file operators, SQL operators and sub-flow operators, and the data processing operators comprise image recognition operators, address matching operators, district matching operators, IP matching operators, image registration operators, coordinate conversion operators, format conversion operators and data cleaning operators; based on the data collaboration flow, parsing and processing the target data to generate standardized data; the parsing and processing of the target data based on the data collaboration flow to generate standardized data comprises: parsing the target data; determining whether the target data contains spatial data according to the parsing result of the target data; in response to the target data containing spatial data, determining whether the spatial data needs coordinate system conversion; in response to the spatial data needing coordinate system conversion, performing coordinate system conversion using a coordinate conversion operator to obtain processed target data.

2. The multi-source heterogeneous data collaboration method according to claim 1, characterized in that, the parsing and processing of the target data based on the data collaboration flow to generate standardized data further comprises: determining whether the target data contains image data according to the parsing result; in response to the target data containing image data, using the image recognition operator to determine whether the image data contains features of geographical objects; in response to the spatial data containing features of geographical objects, performing image registration using the image registration operator to obtain processed target data.

3. The multi-source heterogeneous data collaboration method according to claim 2, characterized in that, the parsing and processing of the target data based on the data collaboration flow to generate standardized data further comprises: determining whether the target data contains address description information according to the parsing result; in response to the target data containing address description information, performing address matching using the address matching operator to obtain processed target data.

4. The multi-source heterogeneous data collaboration method according to claim 3, characterized in that, the parsing and processing of the target data based on the data collaboration flow to generate standardized data further comprises: determining whether the target data contains district information according to the parsing result; in response to the target data containing district information, performing district matching using the district matching operator to obtain processed target data.

5. The multi-source heterogeneous data collaboration method according to any one of claims 2 to 4, characterized in that, the parsing and processing of the target data based on the data collaboration flow to generate standardized data further comprises: determining whether the processed target data needs cleaning; in response to the processed target data needing cleaning, cleaning the processed target data using a data cleaning operator to obtain the standardized data.

6. The multi-source heterogeneous data collaboration method of claim 1, wherein, The method further comprises: determining a data update trigger condition according to the update frequency and mechanism of the target data source; updating the standardized data according to the data update trigger condition.

7. The multi-source heterogeneous data collaboration method of claim 1, wherein, The obtaining of the target data from the target data source comprises: determining whether the target data source requires login; in response to the target data source requiring login, performing simulated login to obtain the target data.

8. The multi-source heterogeneous data collaboration method of claim 7, wherein, The method further comprises providing a data analysis service based on the standardized data, wherein the data analysis service comprises at least one of a data interface service, a big data spatial analysis service, a map service, a report service, and a visualization service. 9.A multi-source heterogeneous data collaboration device, comprising a memory and a processor, wherein the memory stores computer executable instructions, and the computer executable instructions comprise the following steps of: The computer executable instructions, when executed by the processor, implement the multi-source heterogeneous data collaboration method according to any one of claims 1 to 8.

Images