Data processing method, device, equipment and storage medium

Abstract

The present application discloses a data processing method, device, equipment and storage medium. The method includes obtaining a task flow to be executed, the task flow includes task characteristics; according to the task characteristics, determining the scheduling information of at least two operator nodes that execute the logical operation instruction of the task flow and the logical operation instruction, the at least two operator nodes include a first operator node and a second operator node, and the operator types of the at least two operator nodes are different; according to the scheduling information, the first operator node and the second operator node are called by a preset mixed operator node to execute the logical instruction, and the target task result of the task flow is obtained. According to the data processing method provided in the embodiment of the present application, geographic information services can be effectively provided to users in complex service scenarios by hybrid arrangement of nodes from different sources and different operator types.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and storage medium. Background Art

[0002] With the continuous development of Geographic Information System (GIS) and big data technology, the service needs of collecting, storing, managing, calculating, analyzing, displaying and describing relevant geographical distribution data in the whole or part of the earth's surface (including the atmosphere) can be realized through the computing functions provided by GIS and GIS.

[0003] Currently, geographic information services can be provided to users through a single geographic information system and the computing functions it provides. However, as the complexity of service scenarios increases, multiple geographic information systems and the computing functions they provide are required to provide geographic information services to users in complex service scenarios. However, since multiple geographic information systems and the computing functions they provide cannot communicate with each other, the above method of providing geographic information services to users is not applicable to complex service scenarios. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and storage medium to solve the problem that data between multiple geographic information systems and the computing functions they provide cannot be interoperable, and geographic information services cannot be provided to users in complex service scenarios.

[0005] According to a first aspect of an embodiment of the present application, there is provided a data processing method, including:

[0006] Obtaining a task flow to be executed, the task flow includes task features;

[0007] Determine, according to the task characteristics, the scheduling information of at least two operator nodes that execute the logic operation instruction on the task flow and execute the logic operation instruction, wherein the at least two operator nodes include a first operator node and a second operator node, and the operator types of the at least two operator nodes are different;

[0008] According to the scheduling information, the first operator node and the second operator node are called by the preset hybrid operator node to execute the logic instruction to obtain the target task result of the task flow.

[0009] According to a second aspect of an embodiment of the present application, there is provided an operator node generation method, comprising:

[0010] Receiving access requests from at least two operator nodes, where the operator types of the at least two operator nodes are different;

[0011] Obtaining node characteristics of each operator node of at least two operator nodes;

[0012] According to the node characteristics of each operator node and the preset integration conditions, the target hybrid operator node is constructed;

[0013] The target mixing operator node is used to convert the task data and result data of the task flow according to the node characteristics of each target operator node in at least two target operator nodes that execute the task flow during the execution of the task flow, so that the task data and result data are transmitted between at least two target operator nodes, and the result data is the data obtained after each target operator node processes the task data.

[0014] According to a third aspect of an embodiment of the present application, there is provided a data processing device, including:

[0015] An acquisition module is used to acquire a task flow to be executed, where the task flow includes task features;

[0016] A determination module, used to determine, according to the task characteristics, the scheduling information of at least two operator nodes that execute the logic operation instruction on the task flow and execute the logic operation instruction, wherein the at least two operator nodes include a first operator node and a second operator node, and the operator types of the at least two operator nodes are different;

[0017] The processing module is used to call the first operator node and the second operator node to execute the logic instruction according to the scheduling information through the preset mixed operator node to obtain the target task result of the task flow.

[0018] According to a fourth aspect of an embodiment of the present application, there is provided an operator node generating device, including:

[0019] A receiving module, used to receive access requests from at least two operator nodes, where the operator types of the at least two operator nodes are different;

[0020] An acquisition module, used to acquire node characteristics of each operator node of at least two operator nodes;

[0021] A construction module is used to construct a target mixed operator node according to the node characteristics of each operator node and the preset integration conditions; wherein,

[0022] The target mixing operator node is used to convert the task data and result data of the task flow according to the node characteristics of each target operator node in at least two target operator nodes that execute the task flow during the execution of the task flow, so that the task data and result data are transmitted between at least two target operator nodes, and the result data is the data obtained after each target operator node processes the task data.

[0023] According to a fifth aspect of an embodiment of the present application, there is provided a computer device, including: a memory and a processor;

[0024] A memory for storing a computer program;

[0025] A processor is used to execute a computer program stored in a memory, and when the computer program is run, the processor executes the steps of the data processing method shown in the first aspect, or executes the steps of the operator node generation method shown in the second aspect.

[0026] According to the sixth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a computer device, the computer device executes the steps of the data processing method shown in the first aspect, or executes the steps of the operator node generation method shown in the second aspect.

[0027] According to the seventh aspect of the embodiment of the present application, a computer program product is provided, including a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the data processing method shown in the first aspect, or executes the steps of the operator node generation method shown in the second aspect.

[0028] According to the data processing method, apparatus, device and storage medium in the embodiments of the present application, through the task flow to be executed, various geographic information systems and computing functions that execute the task flow are abstracted into at least two operator nodes, and on this basis, at least two operator nodes of different sources and different types are expanded and integrated. Then, a preset mixed operator node that can realize the conversion of data transmitted between at least two operator nodes is defined, so that data between various operator nodes can be transmitted to each other. Then, according to the logic of the execution task flow, the scheduling information of at least two operator nodes that execute the operation instructions is quickly followed, and various operator nodes are called through the preset mixed operator node to complete the task flow to be executed. Thus, the compatible calculation of various operator nodes is met, and the orchestration, scheduling and execution of operator nodes across platforms, languages ​​and types is realized. It can effectively provide users with geographic information services through the mixed orchestration of nodes from different sources and different operator types in complex service scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present application may be better understood from the following description of specific embodiments of the present application in conjunction with the accompanying drawings, wherein the same or similar figure numbers represent the same or similar features.

[0030] Figure 1 is a schematic diagram showing a data processing architecture according to an embodiment;

[0031] Figure 2is a schematic diagram showing the structure of an operator editor according to an embodiment;

[0032] Figure 3 is a flow chart showing a data processing method according to one embodiment;

[0033] Figure 4 is a flow chart showing a method for generating an operator node according to an embodiment;

[0034] Figure 5 is a flow chart showing a method of determining a target task result according to one embodiment;

[0035] Figure 6 is a schematic diagram showing the structure of a data processing device according to an embodiment;

[0036] Figure 7 is a schematic structural diagram showing an operator node generation device according to an embodiment;

[0037] Figure 8 FIG. 4 is a schematic diagram showing the hardware structure of a computer device according to an embodiment. DETAILED DESCRIPTION

[0038] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0039] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0040] With the continuous development of GIS and big data technology, the sources and types of GIS and the computing functions provided by GIS are becoming more and more abundant. For example, GIS includes ArcGIS platform and SuperMap platform. The computing functions provided by GIS can include various spatial operators provided by GIS, spatial computing functions in spatial databases, algorithm applications, etc. The spatial operators involved in the embodiments of the present application are function models that are based on surveying and mapping, geography and mathematical formulas, and realize two-dimensional and three-dimensional calculation and analysis of data with geographical features through computer programming. Spatial operators include but are not limited to function models of geographic coding, spatial analysis, three-dimensional calculation, projection transformation, topology inspection, etc.

[0041] In related technologies, geographic information services can be provided to users through a single GIS and the computing functions provided by GIS. For example, the ArcGIS platform can provide spatial operators such as geocoding, spatial analysis, three-dimensional calculation, projection transformation, and topology checking, and provide users with functions such as data synchronization, data exploration, data quality inspection, and data services.

[0042] However, as the complexity of service scenarios increases, in complex service scenarios, due to the inability of multiple geographic information systems and the computing functions they provide to communicate with each other, a single GIS platform and the functions of a single spatial operator provided by the platform cannot meet the complex service logic, making it impossible to provide geographic information services to users in complex service scenarios. For example, the ArcGIS platform and SuperMap platform only support the arrangement and execution of spatial operators built into the ArcGIS platform, and do not support mixed arrangement with other types of spatial operators.

[0043] Therefore, how to support data interoperability between multiple geographic information systems and the computing functions they provide, and provide geographic information services to users in complex service scenarios has become an issue that requires urgent attention.

[0044] Based on this, an embodiment of the present application provides a data processing method that supports mixed computing of spatial operators. Through the task flow to be executed, various geographic information systems and computing functions that execute the task flow are abstracted into at least two operator nodes, and on this basis, at least two operator nodes of any type such as different languages, different sources, different standards and different purposes (spatial / non-spatial) are expanded and integrated. Then, a preset mixed operator node that can realize the conversion of data transmitted between at least two operator nodes is defined, so that data between various operator nodes can be transmitted to each other. Then, according to the logic of executing the task flow, the scheduling information of at least two operator nodes that execute the operation instructions is quickly followed, and various operator nodes are called through the preset mixed operator node to complete the task flow to be executed.

[0045] Therefore, the compatible computing of various operator nodes is met, and the orchestration, scheduling and execution of operator nodes across platforms, languages ​​and types are realized. In complex service scenarios, the mixed orchestration of nodes from different sources and operator types can be effectively realized to describe and analyze complex service scenarios, so as to provide users with geographic information services. Figure 1 The data processing architecture in the embodiment of the present application shown in the figure describes in detail the data processing method provided in the embodiment of the present application.

[0046] In one or more possible embodiments, Figure 1 As shown, the data processing architecture proposed in the embodiment of the present application may include a data resource platform, and the data resource platform may be a data resource platform in at least one of the following clouds: proprietary cloud, private cloud, public cloud, and hybrid cloud.

[0047] In order to meet the hot-swap and scalability requirements of operator nodes in a data resource platform, an operator editor 10 is provided in the data resource platform in an embodiment of the present application. Scheduling and execution of operator nodes in the operator editor adopt a layered design approach, that is, the operator editor includes a console 101 and a server 102.

[0048] The console 101 is used to provide an operating framework for the overall task flow so as to schedule multiple operator nodes provided by the data resource platform 10. In one example, Figure 2 As shown, the console 101 may include a unit service module (UnitService) 1011 , a data source service module (Data Source Service) 1012 , a task flow service module (Workflow Service) 1013 and an interface service module (Api Service) 1014 .

[0049] Specifically, the unit service class module 1011 is used to read the operator information of at least two operator nodes through the configuration file (Unit Mapper). The data source service class module 1012 is used to manage the database structure, the structured query language (SQL) execution, such as the field information of the logical operation instructions, the scheduling information, and other data required by the data processing method provided by the embodiment of the present application. The task flow service module 1013 is used to arrange the task flow and schedule the preset mixed operator node (Node). Further, the arrangement of the task flow may include determining the logical operation instructions for the task flow and the scheduling information of at least two operator nodes that execute the logical operation instructions when the task flow to be executed is obtained. The scheduling Node may include constructing the Node based on the operator information of at least two operator nodes of the access data resource platform 10 read by the unit service class module 1011, or constructing the Node according to the at least two operator nodes that execute the logical operation instructions. The interface service module 1014 is used to communicate with the service station 102, that is, to send the logical operation instructions, scheduling information, operator information of at least two operator nodes and operator information of Node output by the task flow service module 1013 to the service station 102, so as to implement the calling of specific operator instance methods in the embodiment of the present application. It should be noted that in the embodiment of the present application, in order to enable data between various types of operator nodes to be transmitted to each other, in one example, the operator node can be encapsulated in the form of an application programming interface (API) through the interface service module 1014, and the cross-platform access and calling of various types of operator nodes can be implemented in the form of API, which can effectively provide users with geographic information services through the mixed arrangement of nodes from different sources and different types of operators in complex service scenarios.

[0050] The service station 102 is used to call at least two operator node instances of the task flow to execute the task flow (or workflow). Figure 2 The service station 102 may include a task flow execution service class module (Work flowExcute Service) 1021, which is used to communicate with the interface service module 1014. In an example, the communication content may include logical operation instructions for executing the task flow, scheduling information of at least two operator nodes for executing the logical operation instructions, operator information of at least two operator nodes, and operator information of the Node.

[0051] Based on this, the task flow execution service class module 1021 can also be used to determine the first operator node and the second operator node that execute the logic operation instruction according to the operator information of at least two operator nodes, and determine the Node according to the operator information of the Node. Then, according to the scheduling information, the first operator node and the second operator node are called by the Node to execute the logic instruction to obtain the target task result of the task flow.

[0052] Furthermore, the task flow execution service class module 1021 can be specifically used to call the first operator node to execute logical instructions according to the scheduling information to obtain the first task result, convert the first task result through the Node to obtain the second task result, and use the second task result as the input information of the next operator node, that is, based on the second task result, call the second operator node to execute logical instructions to obtain the target task result.

[0053] Therefore, the embodiment of the present application abstracts the characteristics of various GIS platform operators, defines universal preset mixed operator nodes, satisfies the compatible calculations of various operators, and provides an operator editor of a console-service desk architecture to enable data between different geographic information systems and the computing functions they provide to be transmitted to each other, thereby realizing the directed acyclic graph (DAG) orchestration, scheduling and execution of cross-platform, cross-language and cross-type operators, and solving the problem of the need for mixed orchestration and execution of operator nodes from different sources and with different definitions in complex service scenarios.

[0054] Among them, the operator editor abstracts and integrates various types of geographic information systems and computing functions into operator nodes, and abstracts preset mixed operator nodes of these operator nodes with different languages, different sources, different standards and different purposes (spatial / non-spatial).

[0055] Based on this, when the task flow to be executed is obtained, at least two operator nodes that execute the task flow are orchestrated and mobilized by pre-setting mixed operator nodes. In this way, the operator nodes are mixed and calculated by pre-setting mixed operator nodes, so that data between various operator nodes can be transmitted to each other, and cross-platform access and calling of various operator nodes are realized. It can effectively provide users with geographic information services through mixed orchestration of nodes from different sources and different operator types in complex service scenarios, and support the flexibility and scalability of operator nodes through loose coupling.

[0056] It should be noted that the hybrid computing in the embodiments of the present application can be a computing method for describing and analyzing complex scenes by hybrid arrangement of any type of operator nodes with different languages, different sources, different standards, and different purposes (spatial / non-spatial).

[0057] According to the above architecture and application scenarios, the following are combined Figures 3 to 5 The data processing method provided in the embodiment of the present application is described in detail.

[0058] Figure 3 is a flow chart showing a data processing method according to an embodiment.

[0059] like Figure 3 As shown, the data processing method can be applied to Figure 1 Or Figure 2 As shown, it may specifically include:

[0060] Step 310, obtaining a task flow to be executed, the task flow including task characteristics; Step 320, determining, based on the task characteristics, the scheduling information of at least two operator nodes that execute the logical operation instructions on the task flow and execute the logical operation instructions, the at least two operator nodes including a first operator node and a second operator node, and the operator types of the at least two operator nodes are different; Step 330, according to the scheduling information, calling the first operator node and the second operator node through a preset mixed operator node to execute the logical instructions, to obtain the target task result of the task flow.

[0061] The above steps are described in detail below, as shown below.

[0062] The embodiments of the present application provide the following two methods to trigger the execution of the data processing method provided in the embodiments of the present application.

[0063] The first method: First, receive requests for multiple geographic information systems and the computing functions they provide to access the data resource platform; then, based on these requests, abstract these multiple geographic information systems with different languages, different sources, different standards, and different purposes (spatial / non-spatial) and the computing functions they provide as operator nodes through the data resource platform; then, based on these operator nodes, build the target mixed operator node of the data resource platform, so that through the target mixed operator node, the data between the abstracted multiple operator nodes can be transmitted to each other, realizing cross-platform access and calling of various operator nodes, and can effectively provide users with geographic information services through mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0064] The second method: the data resource platform can obtain the task flow to be executed, and according to the task flow, the data resource platform abstracts the geographic information system and computing functions that execute the task flow into operator nodes, wherein the geographic information system and computing functions that execute the task flow may belong to different languages, different sources, different standards, and different purposes (spatial / non-spatial); then, based on the operator node that executes the task flow, a dedicated preset mixed operator node is constructed with the operator node that executes the task flow, so that through the preset mixed operator node, the data between at least two operator nodes that execute the task flow can be transmitted to each other, thereby realizing cross-platform access and calling of various operator nodes, and can effectively provide users with geographic information services through mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0065] Based on the above two methods, the methods of determining at least two operator nodes and determining a preset mixing operator node provided in the embodiment of the present application are different, as shown below.

[0066] Based on the first method, Figure 4 As shown, a flowchart of an operator node generation method is provided in an embodiment of the present application. The method may include: steps 410 to 430.

[0067] Step 410: Receive access requests from at least two operator nodes, where the operator types of the at least two operator nodes are different.

[0068] Step 420: Obtain node features of each operator node in at least two operator nodes.

[0069] Step 430 : construct a target hybrid operator node according to the node characteristics of each operator node and the preset integration conditions.

[0070] Among them, the target mixing operator node is used to convert the task data and result data of the task flow according to the node characteristics of each target operator node in at least two target operator nodes that execute the task flow during the execution of the task flow, so that the task data and result data are transmitted between at least two target operator nodes, and the result data is the data obtained after each target operator node processes the task data.

[0071] Specifically, with respect to step 430, the embodiment of the present application provides the following four methods for integrating at least two operator nodes to obtain a target hybrid operator node.

[0072] Method 1: According to the node characteristics of at least two operator nodes of different sources and different operator types accessing the data resource platform, an operator application image package of each operator node in at least two operator nodes is constructed, wherein the operator application image package includes operator operating system information and editing environment information.

[0073] Then, based on the operator application mirror package of each operator node, a target hybrid operator node is generated.

[0074] Exemplarily, the at least two operator nodes include a first operator node and a second operator node. If the first operator node is a SuperMap operator, the operator operating system information of the SuperMap operator is a centOS system, and the editing environment information may be a Java compilation environment. If the second operator node is a QGIS operator, the operator operating system information of the QGIS operator is an Ubuntu system, and the editing environment information may be a Python compilation environment.

[0075] Then, based on the operator running system information and editing environment information of each operator node, a target hybrid operator node is generated. Since the target hybrid operator node records the operator running system information and editing environment information of each operator node, the input data and output data of each operator node can be converted through the target hybrid operator node, so that the data between various operator nodes can be transmitted to each other, so as to realize cross-platform access and calling of various operator nodes. It can effectively provide users with geographic information services through mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0076] Method 2: Based on the node characteristics of at least two operator nodes of different sources and different operator types connected to the data resource platform, the common attributes of each operator node in at least two operator nodes are abstracted, where the common attributes may include the operator input parameter attributes, calculation parameter attributes and operator output parameter attributes of the operator node.

[0077] Then, based on the operator input parameter attributes, calculation parameter attributes, and operator output parameter attributes of each operator node, a target hybrid operator node is generated.

[0078] Furthermore, the operator input parameter attributes may specifically include two parameters, namely input name and input type, which are used to characterize the input parameters of the operator node; the calculation parameter attributes are used to characterize the execution logic of the operator node; the operator output parameter attributes are used to set the output mode of the operator node, such as binary stream or database.

[0079] Based on this, the target hybrid operator node records the common properties of each operator node. Therefore, the target hybrid operator node can be used to convert the data to be transmitted to the operator node according to the operator input parameters and calculation parameter properties of the operator node, so that the operator node can recognize and process the converted data. Then, the operator node can set the operator output parameters so that the target hybrid operator node can recognize the data output by the operator node and forward it to other operator nodes, so that data between various operator nodes can be transmitted to each other, so as to realize cross-platform access and calling of various operator nodes, and can effectively provide users with geographic information services through mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0080] Method three: by using a preset parameter parsing algorithm, the operator types of at least two operator nodes of different sources and different operator types connected to the data resource platform are parsed to obtain the operator data format of each operator node.

[0081] Generate a target hybrid operator node based on the operator data format of each operator node.

[0082] Exemplarily, the preset parameter parsing algorithm may be Data Set Convert ToWKB or Data Set Convert To WKT, and the at least two operator nodes may be ArcGIS, SuperMap and QGIS, so that the operator data format of each operator node may be obtained.

[0083] Then, the target hybrid operator node stores the operator data format of each operator node converted into an open source GIS data format, so that the target hybrid operator node can convert the data output by each operator node into a unified data format, so that the data between various operator nodes can be transmitted to each other, so as to realize cross-platform access and calling of various operator nodes, and effectively provide users with geographic information services through mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0084] Method 4: Based on at least two operator nodes of different sources and different operator types connected to the data resource platform, a target hybrid operator node is constructed, and the target hybrid operator node provides an open source API (open API). The open API is used to convert the task data and result data of the task flow during the execution of the task flow, so that the task data and result data are transmitted between at least two target operator nodes, and the result data is the data obtained after each target operator node processes the task data. Alternatively, a directed acyclic graph DAG is constructed and executed according to the task flow.

[0085] Based on this, the open API may include a Save model, a Run model, and a getStatus model, wherein the Save model is used to save the operator parameters of each operator node in the DAG, the Run model is used to access and call the logical operation instructions of each operator node to execute the task flow, and the get Status model is used to monitor the execution results. For example, when get status returns a preset value such as 200, it indicates that the target task result of the task flow is obtained. Here, in some embodiments, the execution result of the task flow can be queried from the database.

[0086] Therefore, by building DAG through open API and monitoring the execution results of DAG task flow, data between various operator nodes can be transmitted to each other to achieve cross-platform access and calling of various operator nodes. It can effectively provide users with geographic information services through mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0087] The second method involves step 320. The embodiment of the present application provides at least two methods for determining at least two operator nodes and scheduling information for executing logical operation instructions, as shown below.

[0088] In a possible embodiment, step 320 may specifically include:

[0089] Identify whether the task feature includes information of at least two designated operator nodes designated by the user to execute the task flow;

[0090] When it is identified that the task characteristics include information of at least two designated operator nodes specified by the user for executing the task flow, the logical operation instructions for executing the task flow are determined according to the task flow, and the at least two designated operator nodes are determined to be at least two operator nodes according to the information of the designated operator nodes.

[0091] In another possible embodiment, step 320 may specifically include: constructing a directed acyclic graph of the task flow by presetting a mixed operator node according to the task characteristics, wherein the directed acyclic graph includes a path and at least two spatial operators on the path;

[0092] At least two spatial operators are determined as at least two operator nodes, and scheduling information of the at least two operator nodes is generated according to the path.

[0093] Based on the preset mixing operator node and the target mixing operator node determined by the first and second methods, step 330 may be executed. It should be noted that the preset mixing operator node in step 330 is equivalent to the target mixing operator node.

[0094] In one or more possible embodiments, step 330 is involved. Figure 5 As shown, the step 330 may specifically include steps 3301 to 3303, as shown below.

[0095] Step 3301: According to the scheduling information, call the first operator node to execute the logic instruction to obtain the first task result.

[0096] Step 3302: Convert the first task result through a preset hybrid operator node to obtain a second task result.

[0097] Furthermore, step 3302 may specifically include:

[0098] According to the node attribute of the second operator node, the first task result is converted through the preset hybrid operator node to obtain the second task result.

[0099] Step 3303: Based on the second task result, call the second operator node to execute the logic instruction to obtain the target task result.

[0100] The following describes step 3302 and step 3303 in detail in combination with different scenarios.

[0101] In one example, step 3302 is involved, and in the embodiment of the present application, at least two node attributes are provided as follows to illustrate step 3302 respectively.

[0102] (1) When the node attribute includes an operator application image package, and the operator application image package includes operator operating system information and editing environment information, the first task result is converted through a preset hybrid operator node according to the operator operating system information and editing environment information of the second operator node to obtain a second task result.

[0103] (2) When the node attributes include operator input parameter attributes, the first task result is converted through a preset hybrid operator node according to the input parameter attributes to obtain the second task result.

[0104] Based on possibility (2), step 3303 in the embodiment of the present application may specifically include:

[0105] According to the calculation parameter attributes and based on the second task result, the second operator node is called to execute the logic instruction to obtain the stage task result;

[0106] According to the output parameter attributes of the operator, the stage task results are converted to obtain the target task results.

[0107] In another example, before step 3302, the following may also be included:

[0108] Determining a preset parameter conversion algorithm for each operator node according to an operator type of each operator node in at least two operator nodes;

[0109] By using a preset parameter conversion algorithm, operator parameters in an operator node corresponding to the preset parameter conversion algorithm are converted to obtain an open source data format for data interaction between at least two operator nodes.

[0110] Based on this, step 3302 may specifically include:

[0111] According to the open source data format, the first task result is converted through the preset hybrid operator node to obtain the second task result.

[0112] In yet another example, before step 330, the method may further include:

[0113] A plurality of operator nodes are obtained, and based on each of the plurality of operator nodes, an open source program programming interface of each operator node is constructed by presetting a mixed operator node.

[0114] Based on this, step 330 is involved. In the case where the first operator node corresponds to the first open source program programming interface and the second operator node corresponds to the second open source program programming interface, step 330 may specifically include:

[0115] According to the scheduling information, the first open source program programming interface is called through the preset hybrid operator node to execute the logic instruction to obtain the third task result;

[0116] Convert the third task result through the preset hybrid operator node to obtain the fourth task result;

[0117] Based on the fourth task result, the second open source program programming interface is called to execute the logic instruction to obtain the target task result.

[0118] Therefore, the embodiment of the present application provides a data processing method that supports the mixing of operator nodes. By abstracting the characteristics of various types of operator nodes, a universal preset mixed operator node is defined, and on this basis, operator nodes from various sources are expanded and integrated to achieve cross-platform, cross-language, and cross-type operator node orchestration, scheduling, and execution, solving the problem of the need for mixed orchestration and execution of operator nodes from different sources and with different definitions in complex service scenarios.

[0119] It should be clear that the present application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the convenience and brevity of description, a detailed description of the known methods is omitted here, and the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, which will not be repeated here.

[0120] Based on the same inventive concept, the present application provides a data processing device corresponding to the above-mentioned data processing method. Figure 6Provide detailed explanation.

[0121] Figure 6 It is a schematic diagram showing the structure of a data processing device according to an embodiment.

[0122] like Figure 6 As shown, the data processing device 60 is used as follows Figure 1 or Figure 2 The operator editor 10 shown, the data processing device 60 may specifically include:

[0123] An acquisition module 601 is used to acquire a task flow to be executed, where the task flow includes task features;

[0124] A determination module 602 is used to determine, according to the task characteristics, the scheduling information of at least two operator nodes that execute the logic operation instruction on the task flow and execute the logic operation instruction, wherein the at least two operator nodes include a first operator node and a second operator node, and the operator types of the at least two operator nodes are different;

[0125] The processing module 603 is used to call the first operator node and the second operator node to execute the logic instruction through the preset hybrid operator node according to the scheduling information to obtain the target task result of the task flow.

[0126] Based on this, the data processing device 60 provided in the embodiment of the present application is described in detail below:

[0127] In one or more possible embodiments, the processing module 603 may be specifically configured to, according to the scheduling information, call the first operator node to execute the logic instruction to obtain the first task result;

[0128] Convert the first task result through the preset hybrid operator node to obtain the second task result;

[0129] Based on the second task result, the second operator node is called to execute the logic instruction to obtain the target task result.

[0130] In another or more possible embodiments, the processing module 603 may be specifically configured to convert the first task result through a preset hybrid operator node according to the node attribute of the second operator node to obtain the second task result.

[0131] In another one or more possible embodiments, the processing module 603 can be specifically used to, when the node attributes include an operator application image package, and the operator application image package includes operator operating system information and editing environment information, convert the first task result through a preset mixed operator node according to the operator operating system information and editing environment information of the second operator node to obtain the second task result.

[0132] In one or more possible embodiments, the processing module 603 may be specifically configured to, when the node attributes include operator input parameter attributes, convert the first task result through a preset hybrid operator node according to the input parameter attributes to obtain the second task result.

[0133] In one or more possible embodiments, the processing module 603 may be specifically configured to, when the node attributes further include operator output parameter attributes and calculation parameter attributes, call the second operator node to execute the logic instruction based on the second task result according to the calculation parameter attributes to obtain the stage task result;

[0134] According to the output parameter attributes of the operator, the stage task results are converted to obtain the target task results.

[0135] In one or more possible embodiments, the data processing device 60 provided in the embodiment of the present application may further include a conversion module; wherein:

[0136] The determination module 602 may also be used to determine a preset parameter conversion algorithm for each operator node according to the operator type of each operator node in the at least two operator nodes;

[0137] The conversion module is used to convert the operator parameters in the operator nodes corresponding to the preset parameter conversion algorithm through the preset parameter conversion algorithm to obtain an open source data format for data interaction between at least two operator nodes.

[0138] In yet another one or more possible embodiments, the processing module 603 may be specifically configured to convert the first task result through a preset hybrid operator node according to an open source data format to obtain a second task result.

[0139] In one or more possible embodiments, the data processing device 60 provided in the embodiment of the present application may also include a construction module for obtaining multiple operator nodes, and based on each operator node in the multiple operator nodes, constructing an open source program programming interface for each operator node by presetting a mixed operator node.

[0140] In one or more possible embodiments, the processing module 603 may be specifically configured to, when the first operator node corresponds to the first open source program programming interface and the second operator node corresponds to the second open source program programming interface, call the first open source program programming interface to execute logic instructions through a preset hybrid operator node according to the scheduling information to obtain a third task result;

[0141] Convert the third task result through the preset hybrid operator node to obtain the fourth task result;

[0142] Based on the fourth task result, the second open source program programming interface is called to execute the logic instruction to obtain the target task result.

[0143] In yet another one or more possible embodiments, the determination module 602 may be specifically configured to construct a directed acyclic graph of the task flow by presetting a mixed operator node according to the task characteristics, wherein the directed acyclic graph includes a path and at least two spatial operators on the path;

[0144] At least two spatial operators are determined as at least two operator nodes, and scheduling information of the at least two operator nodes is generated according to the path.

[0145] Therefore, through the task flow to be executed, various geographic information systems and computing functions that execute the task flow are abstracted into at least two operator nodes, and on this basis, at least two operator nodes of different sources and different types are expanded and integrated. Then, a preset mixed operator node that can realize the conversion of data transmitted between at least two operator nodes is defined, so that data between various operator nodes can be transmitted to each other. Then, according to the logic of the execution task flow, the scheduling information of at least two operator nodes that execute the operation instructions is quickly followed, and various operator nodes are called through the preset mixed operator node to complete the task flow to be executed. In this way, the compatible calculation of various operator nodes is met, and the orchestration, scheduling and execution of operator nodes across platforms, languages ​​and types is realized. It can effectively provide users with geographic information services through the mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0146] Based on the same inventive concept, the present application provides an operator node generation device corresponding to the operator node generation method involved above. Figure 7 Provide detailed explanation.

[0147] Figure 7 It is a schematic diagram showing the structure of an operator node generating device according to an embodiment.

[0148] like Figure 7 As shown, the operator node generating device 70 is applied to Figure 1 or Figure 2 The operator editor 10 shown, the operator node generating device 70 may specifically include:

[0149] A receiving module 701 is used to receive access requests from at least two operator nodes, where the operator types of the at least two operator nodes are different;

[0150] An acquisition module 702 is used to acquire node features of each operator node of at least two operator nodes;

[0151] The construction module 703 is used to construct the target mixed operator node according to the node characteristics of each operator node and the preset integration conditions; wherein,

[0152] The target mixing operator node is used to convert the task data and result data of the task flow according to the node characteristics of each target operator node in at least two target operator nodes that execute the task flow during the execution of the task flow, so that the task data and result data are transmitted between at least two target operator nodes, and the result data is the data obtained after each target operator node processes the task data.

[0153] Therefore, the target hybrid operator node is constructed to identify the data output by the operator node and forward it to other operator nodes, so that the data between various operator nodes can be transmitted to each other, so as to realize cross-platform access and calling of various operator nodes. It can effectively provide users with geographic information services through the mixed orchestration of nodes from different sources and different operator types in complex service scenarios.

[0154] Figure 8 FIG. 4 is a schematic diagram showing the hardware structure of a computer device according to an embodiment.

[0155] like Figure 8 As shown, the computer device 800 includes an input device 801 , an input interface 802 , a processor 803 , a memory 804 , an output interface 805 , and an output device 806 .

[0156] The input interface 802, the processor 803, the memory 804, and the output interface 805 are interconnected via the bus 810, and the input device 801 and the output device 806 are connected to the bus 810 via the input interface 802 and the output interface 805, respectively, and are further connected to other components of the computer device 800. Specifically, the input device 801 receives input information from the outside, and transmits the input information to the processor 803 via the input interface 802; the processor 803 processes the input information based on the computer executable instructions stored in the memory 804 to generate output information, temporarily or permanently stores the output information in the memory 804, and then transmits the output information to the output device 806 via the output interface 805; the output device 806 outputs the output information to the outside of the computer device 800 for use by the user.

[0157] In one embodiment, Figure 8 The computer device 800 shown can be implemented as a data processing device, which may include: a memory configured to store a program; a processor configured to run the program stored in the memory to execute the data processing method described in the above embodiment. Or, Figure 8 The computer device 800 shown may be implemented as an operator node generation device, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the operator node generation method described in the above embodiment.

[0158] In one embodiment, the memory can also be used to store the target task results of the task flow and the target task results of the task flow. Figures 1 to 5 The calculation result of each step in the data processing and operator node generation process described herein. As an example, the calculation result at least includes: a logical operation instruction for executing the task flow and scheduling information of at least two operator nodes that execute the logical operation instruction.

[0159] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer-readable storage medium. For example, an embodiment of the present application includes a computer-readable storage medium, which includes storing a program or instruction on the computer-readable storage medium, and when the program or instruction is executed by a computer device, the computer device performs the steps of the above method.

[0160] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions, and when it is run on a computer, the computer executes the method described in each of the above embodiments. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid-state drives), etc.

[0162] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, comprising: Acquire a task flow to be executed, wherein the task flow includes task features; According to the task characteristics, determining the logical operation instruction for executing the task flow and the scheduling information of at least two operator nodes for executing the logical operation instruction, wherein the at least two operator nodes include a first operator node and a second operator node, and the operator types of the at least two operator nodes are different; wherein the operator node is a spatial operator, and the spatial operator includes geographic coding, spatial analysis, three-dimensional calculation, projection transformation, and topological inspection, and the spatial operator is a function model based on surveying and mapping, geography, and mathematical formulas, and is implemented through computer programming to perform two-dimensional and three-dimensional calculation and analysis on data with geographic features; According to the scheduling information, calling the first operator node and the second operator node by a preset hybrid operator node to execute the logic operation instruction, so as to obtain the target task result of the task flow; The preset mixing operator node is obtained by the following method: Receiving access requests from at least two operator nodes, where the at least two operator nodes are of different operator types; Obtaining node characteristics of each operator node of the at least two operator nodes; Abstracting common attributes of each of the at least two operator nodes according to node features of the at least two operator nodes of different sources and different operator types accessing the data resource platform, wherein the common attributes include operator input parameter attributes, calculation parameter attributes, and operator output parameter attributes of the operator nodes; Generate the preset hybrid operator node based on the operator input parameter attributes, calculation parameter attributes and operator output parameter attributes of each operator node; or, Parsing the operator types of the at least two operator nodes of different sources and different operator types accessing the data resource platform through a preset parameter parsing algorithm to obtain the operator data format of each operator node; The preset hybrid operator node is generated based on the operator data format of each operator node.

2. The method according to claim 1, wherein: The step of calling the first operator node and the second operator node to execute the logic operation instruction according to the scheduling information by using a preset hybrid operator node to obtain a target task result of the task flow includes: According to the scheduling information, calling the first operator node to execute the logic operation instruction to obtain a first task result; Converting the first task result through the preset hybrid operator node to obtain a second task result; Based on the second task result, the second operator node is called to execute the logical operation instruction to obtain the target task result.

3. The method according to claim 2, wherein: The converting the first task result by using the preset mixing operator node to obtain the second task result includes: According to the node attribute of the second operator node, the first task result is converted through the preset hybrid operator node to obtain the second task result.

4. The method according to claim 3, wherein: The node attributes include an operator application image package, and the operator application image package includes operator operating system information and editing environment information; The converting the first task result through the preset hybrid operator node according to the node attribute of the second operator node to obtain the second task result includes: According to the operator running system information and the editing environment information of the second operator node, the first task result is converted through the preset hybrid operator node to obtain the second task result.

5. The method according to claim 2, wherein: Before the method further comprises: calling the first operator node and the second operator node to execute the logic operation instruction according to the scheduling information by a preset mixed operator node to obtain the target task result of the task, the method further comprises: Determining a preset parameter conversion algorithm for each operator node according to an operator type of each operator node in the at least two operator nodes; By using the preset parameter conversion algorithm, the operator parameters in the operator node corresponding to the preset parameter conversion algorithm are converted to obtain an open source data format for data interaction between the at least two operator nodes; According to the open source data format, the first task result is converted through the preset hybrid operator node to obtain the second task result.

6. The method according to claim 1, wherein: Before the method further comprises: calling the first operator node and the second operator node to execute the logic operation instruction according to the scheduling information by a preset mixed operator node to obtain the target task result of the task flow; A plurality of operator nodes are obtained, and based on each of the plurality of operator nodes, an open source program programming interface of each operator node is constructed through the preset mixed operator node.

7. The method according to claim 1, wherein: The determining, according to the task characteristics, the logical operation instruction for executing the task flow and the scheduling information of at least two operator nodes for executing the logical operation instruction comprises: According to the task characteristics, constructing a directed acyclic graph of the task flow through the preset hybrid operator node, wherein the directed acyclic graph includes a path and at least two spatial operators on the path; The at least two spatial operators are determined as the at least two operator nodes, and scheduling information of the at least two operator nodes is generated according to the path.

8. A data processing device, comprising: An acquisition module, used for acquiring a task flow to be executed, wherein the task flow includes task features; A determination module, used to determine, according to the task characteristics, the execution of the logic operation instruction on the task flow and the scheduling information of at least two operator nodes that execute the logic operation instruction, wherein the at least two operator nodes include a first operator node and a second operator node, and the operator types of the at least two operator nodes are different; wherein the operator node is a spatial operator, and the spatial operator includes geographic coding, spatial analysis, three-dimensional calculation, projection transformation, and topological inspection, and the spatial operator is a function model based on surveying and mapping, geography, and mathematical formulas, and realizes two-dimensional and three-dimensional calculation and analysis of data with geographic features through computer programming; A processing module, configured to call the first operator node and the second operator node to execute the logic operation instruction according to the scheduling information through a preset hybrid operator node, so as to obtain a target task result of the task flow; The preset hybrid operator node is obtained by the operator node generation device: The operator node generation device includes a receiving module, an acquisition module and a construction module; The receiving module is used to receive access requests from at least two operator nodes, where the operator types of the at least two operator nodes are different; The acquisition module is used to acquire the node characteristics of each operator node of the at least two operator nodes; The construction module is used to abstract the common attributes of each operator node in the at least two operator nodes according to the node characteristics of the at least two operator nodes of different sources and different operator types accessing the data resource platform, wherein the common attributes include the operator input parameter attributes, calculation parameter attributes and operator output parameter attributes of the operator node; Generate the preset hybrid operator node based on the operator input parameter attributes, calculation parameter attributes and operator output parameter attributes of each operator node; or, Parsing the operator types of the at least two operator nodes of different sources and different operator types accessing the data resource platform through a preset parameter parsing algorithm to obtain the operator data format of each operator node; The preset hybrid operator node is generated based on the operator data format of each operator node.

9. A computer device comprising: Memory and processor, The memory is used to store a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the steps of the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program or instruction, wherein when the program or instruction is executed by a computer device, the computer device executes the steps of the data processing method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, which enables a computer device to execute the steps of the data processing method according to any one of claims 1 to 7 when the computer program is executed by the computer device.

Images