Scientific data workflow system, method and equipment based on data networking
Patent Information
- Application Number
- CN202411127668.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-03
AI Technical Summary
The difficulty in reproducibility in the field of scientific data is mainly due to the lack of openness of data and the lack of clear and structured description of data processing methods, which makes it difficult for research results to be reproduced by others.
Using a scientific data workflow system based on digital networking, the scientific data processing method is transformed into a formal workflow through controllers, data object managers, workflow execution engines and execution instance schedulers, and the scientific data processing method is transformed into a formal workflow using digital networking infrastructure. The execution example is constructed to realize the automatic reproduction of the data processing method.
It improves the clarity and accuracy of the description of data processing methods, realizes the automated reproduction of scientific data processing processes, makes originally unopen data sets discoverable and accessible, and enhances the availability of data.
Smart Images

Figure CN119067425A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of scientific data processing, and in particular to a scientific data workflow system, method and device based on digital networking. Background Art
[0002] The reproducibility of scientific research refers to the property that research results can be repeated by repeating experiments under the same conditions. Reproducibility has always been the foundation and core of scientific research. It is not only the basic guarantee for verifying research conclusions, but also the cornerstone of scientific progress. However, as experiments become more complex and research methods become more diverse, the results of scientific research are often difficult to be reproduced by others, and the problem of reproducibility difficulties has become increasingly prominent.
[0003] With the advent of the big data era, scientific data has become an increasingly important scientific research paradigm. Like traditional experimental science, research in the field of scientific data often encounters the problem of poor reproducibility. In addition, due to the complexity and diversity of the research process in the field of scientific data, the data relied on is often not publicly accessible due to copyright, privacy or other reasons, making the reproducibility of data processing in the field of scientific data more difficult. Therefore, it is necessary to develop a scientific data workflow system, method and equipment based on the Internet of Things to accurately and efficiently reproduce the data processing process in the field of scientific data. Summary of the invention
[0004] In view of the above problems, the embodiments of the present application provide a scientific data workflow system, method and device based on the Internet of Things to overcome the above problems or at least partially solve the above problems.
[0005] In a first aspect of an embodiment of the present application, a scientific data workflow system based on a data network is provided, the system comprising:
[0006] The controller is used to receive an execution request sent by a client, wherein the execution request is used to request re-execution of a first workflow; wherein the first workflow is composed of a set of function objects, data objects, execution instances, and final output results of the first workflow, and the first workflow is used to describe the execution process of a first data processing method, wherein the first data processing method represents a method of obtaining first output data by performing function mapping of one or more stages using first scientific data;
[0007] A data object manager, configured to obtain a target digital object from a data network infrastructure according to the execution request, and store the target digital object locally; the target digital object includes: the first scientific data and function used by the first data processing method;
[0008] A workflow execution engine, configured to obtain an execution instance of a second workflow isomorphic to the first workflow according to the execution request and the target digital object;
[0009] An execution instance scheduler, used to execute the execution instance of the second workflow using the target digital object to obtain a final output result of the second workflow;
[0010] The controller is further configured to return the final output result of the second workflow as the execution result of the execution request to the client.
[0011] The second aspect of the embodiment of the present application further provides a scientific data workflow method based on a data network, the method comprising:
[0012] Receive an execution request sent by a client, wherein the execution request is used to request re-execution of a first workflow; wherein the first workflow is composed of a set of function objects, data objects, execution instances, and final output results of the first workflow, and the first workflow is used to describe the execution process of a first data processing method, wherein the first data processing method represents a method of performing function mapping of one or more stages using first scientific data to obtain first output data;
[0013] According to the execution request, a target digital object is obtained from the Internet infrastructure, and the target digital object is stored locally; the target digital object includes: the first scientific data and function used by the first data processing method;
[0014] Acquire, according to the execution request and the target digital object, an execution instance of a second workflow isomorphic to the first workflow;
[0015] Utilizing the target digital object, executing the execution instance of the second workflow to obtain a final output result of the second workflow;
[0016] The final output result of the second workflow is used as the execution result of the execution request and returned to the client.
[0017] The third aspect of the embodiment of the present application also provides an electronic device, including a memory, a processor and a computer program stored on the memory, and the processor executes the computer program to implement the steps in the method for creating a scientific data workflow based on the Internet of Things described in the second aspect of the embodiment of the present application.
[0018] The fourth aspect of the embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the method for creating a scientific data workflow based on the Internet of Things described in the second aspect of the embodiment of the present application are implemented.
[0019] The fifth aspect of the embodiments of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor is enabled to implement the steps in the method for creating a scientific data workflow based on the Internet of Things as described in the second aspect of the embodiments of the present application.
[0020] An embodiment of the present application provides a scientific data workflow system based on a digital network, the system comprising: a controller, for receiving an execution request sent by a client, the execution request being used to request re-execution of a first workflow; wherein the first workflow is composed of a set of function objects, data objects, execution instances and final output results of the first workflow, the first workflow being used to describe the execution process of a first data processing method, the first data processing method representing a method of obtaining first output data by performing function mapping of one or more stages using first scientific data; a data object manager, for obtaining a target digital object from a digital network infrastructure according to the execution request, and storing the target digital object locally; the target digital object comprising: the first scientific data and function used by the first data processing method; a workflow execution engine, for obtaining an execution instance of a second workflow isomorphic to the first workflow according to the execution request and the target digital object; an execution instance scheduler, for executing the execution instance of the second workflow using the target digital object to obtain the final output result of the second workflow; the controller is also used to return the final output result of the second workflow to the client as the execution result of the execution request.
[0021] The specific beneficial effects are:
[0022] On the one hand, the present application uses the formal definition of scientific data in the workflow (the first workflow, consisting of a set of function objects, data objects, execution instances, and the final output results of the first workflow), and converts the data processing method (a method of using the first scientific data to perform function mapping in one or more stages to obtain first output data) into the form of a workflow, thereby improving the clarity and accuracy of the method description, and realizing the automated reproduction of the data processing method by re-executing the isomorphic workflow (the second workflow).
[0023] On the other hand, the present application obtains the digital objects (target digital objects) of the digital network through the interface provided by the basic settings of the digital network, and uses them as input data for executing workflows, making originally closed data sets discoverable and accessible, thereby enhancing the availability of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0025] Figure 1 It is a structural diagram of a scientific data workflow system based on a data network provided in an embodiment of the present application;
[0026] Figure 2 This is a scientific data workflow system architecture diagram provided by an embodiment of the present application;
[0027] Figure 3 It is a flow chart of a scientific data workflow provided by an embodiment of the present application;
[0028] Figure 4 It is a flowchart of the steps of a scientific data workflow method based on the Internet of Things provided in an embodiment of the present application;
[0029] Figure 5 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.
[0031] Reproducibility has always been the foundation and core of scientific research. It is not only the basic guarantee for verifying research conclusions, but also the cornerstone of scientific progress. Reproducibility means that the results published by the author can be verified by repeating the experiment under the same conditions. However, with the continuous development of scientific research, the problem of reproducibility has become increasingly prominent. As experiments become more complex and research methods become more diverse, the results of scientific research are often difficult to be reproduced by others, which has caused widespread concern in the scientific community about the quality and credibility of research.
[0032] With the advent of the big data era, scientific data has become an increasingly important scientific research paradigm. Like traditional experimental science, research in the field of scientific data often encounters the problem of poor reproducibility. At present, there are two key problems in the field of scientific data: one is that the data relied on by the research is not open, and the other is that the description of the data processing methods is not clear and structured enough. First, a large amount of scientific data research relies on specific data sets. However, these data sets are often not publicly accessible due to copyright, privacy or other reasons. Secondly, researchers often describe their data analysis and processing methods in natural language in papers or technical documents. This description lacks the necessary structure and standardization, making it difficult to accurately reproduce the original research.
[0033] In view of the above problems, the present application embodiment proposes a scientific data workflow system based on a data network to solve the above problems such as the difficulty in reproducing the data processing process in the field of scientific data. The following is a detailed description of the scientific data workflow system based on a data network provided by the present application embodiment through some embodiments and their application scenarios in conjunction with the accompanying drawings.
[0034] The first aspect of the present application embodiment provides a scientific data workflow system based on a data network, referring to Figure 1 , Figure 1 A schematic diagram of a scientific data workflow system based on a data network is provided in an embodiment of the present application, such as Figure 1 As shown, the system comprises:
[0035] The controller is used to receive an execution request sent by a client, wherein the execution request is used to request re-execution of a first workflow; wherein the first workflow is composed of a set of function objects, data objects, execution instances, and final output results of the first workflow, and the first workflow is used to describe the execution process of a first data processing method, wherein the first data processing method represents a method of obtaining first output data by performing function mapping of one or more stages using first scientific data;
[0036] A data object manager, configured to obtain a target digital object from a data network infrastructure according to the execution request, and store the target digital object locally; the target digital object includes: the first scientific data and function used by the first data processing method;
[0037] A workflow execution engine, configured to obtain an execution instance of a second workflow isomorphic to the first workflow according to the execution request and the target digital object;
[0038] An execution instance scheduler, used to execute the execution instance of the second workflow using the target digital object to obtain a final output result of the second workflow;
[0039] The controller is further configured to return the final output result of the second workflow as the execution result of the execution request to the client.
[0040] In the related art, the analysis and processing methods of data are often described in natural language. This description lacks the necessary structuring and standardization, and it is difficult to accurately reproduce the original research process (i.e., the original data processing process). The present application converts a complete data processing method into a corresponding workflow through a scientific data workflow system. Each time a workflow is executed, it means that the data processing method has been executed once. Therefore, when a research process (e.g., a first data processing method for scientific data) needs to be reproduced, an execution request can be sent through the client to request the re-execution of the workflow corresponding to the data processing method (e.g., the first workflow). The scientific data workflow system executes an isomorphic second workflow according to the first workflow, and obtains the final output result of the second workflow (i.e., the final result obtained by re-executing the first data processing method for scientific data), thereby realizing the reproduction of the entire process of the first data processing method.
[0041] Specifically, in order to structurally represent the workflow, this application converts a specific data processing process (a method process of using the first scientific data to perform one or more stages of function mapping to obtain the first output data, that is, the required scientific data) into a workflow with a clear and accurate structure, and formally defines the following basic concepts. The first workflow and the second workflow in the embodiment of this application both conform to the following definitions. Among them, the first workflow and the second workflow are scientific data-related workflows, which are used to characterize scientific data-related information (for example, specific data values of scientific data, specific data processing processes, and other information).
[0042] Definition 1: Function object f represents a function that maps input data to output data. Define each function object f = (c, p - , p + ).in:
[0043] c represents the execution logic of the function object f, and its structure and properties are determined by the specific application scenario.
[0044] P - is a set of input parameters, in Represents the input parameter label of the i-th input parameter (used to refer to the label of the specific input parameter in the function object). For each function object f, the input parameter labels between every two input parameters are different, satisfying the condition:
[0045] P +is a set of output parameters, in Represents the output parameter label of the i-th output parameter (used to refer to the label of the specific output parameter in the function object). For each function object f, the output parameter labels between every two output parameters are different, satisfying the condition:
[0046] Definition 2: Data object d is a set of one or more data items, where the data item labels of every two data items in the set are different. in:
[0047] k i Represents the label of the i-th data item, satisfying the conditions:
[0048] v i Represents the data content of the i-th data item.
[0049] Definition 3: An execution instance is the smallest computational unit in a workflow, which represents the relationship between input data, output data, and function object f in a function execution. We define an execution instance as
[0050] in:
[0051] f is a function object, representing the function executed in the execution instance e.
[0052] D - is the set of input data (i.e., input data set) for executing instance e, It means that when the execution instance e is executed, there are a total of n input data, among which the input label of the i-th input data is The corresponding data object (input data) is
[0053] D + is the set of output data (i.e., output data set) of executing instance e, It means that when the execution instance e is executed, there are a total of m output data, among which the output label of the i-th output data is The corresponding data object (output data) is
[0054] The execution of a function object is realized by executing one or more execution instances in a certain order. A function object can have multiple input parameters p - and output parameter p + , each input / output parameter has a label different from other parameters (i.e., the input parameter label in Definition 1 and output parameter labels ). In an execution instance e, input data Input tags used Indicates which parameter input (i.e. input label) it is used as during the execution of the function (i.e. during the execution of the execution instance e) The input parameter label of one of the input parameters of the function object f Same), output data Using Output Tags Indicates which parameter is output as a result of the function execution (i.e. output label The output parameter label of an output parameter of the function object f For each input / output parameter of a function, there is only one input / output data (i.e., data object d) with the same label. That is, let f = (c, p - , p + )(Definition 1), satisfies the following conditions:
[0055] a)
[0056] b)
[0057] c)
[0058] d)
[0059] Based on this, the mapping of execution instances to their input and output datasets can be defined as follows:
[0060] Definition 3.1: Define the mapping of execution instance e to its input data set as input(e). For e = (f, D - , D + ), d - is the data object corresponding to the input data, k - For the data object d - Labels of the input data objects in the input dataset.
[0061] Definition 3.2: The mapping from an execution instance e to its output dataset is defined as output(e). For e = (f, D - , D + ), d + is the data object corresponding to the output data, k + For the data object d + Labels for output data objects in the output dataset.
[0062] Definition 4: A workflow R consists of a set of function objects, a set of data objects, a set of execution instances that describe the relationship between the function objects and the data objects (input data and output data), and a set of final output results that describe the output of the workflow. A workflow describes a complete process (i.e., the execution process of a data processing method) of using one or more input data to perform multi-stage function mapping and finally obtain one or more result data objects. Define workflow R = (F, D, E, D * ),in:
[0063] F is the function set of workflow R, where f i is the i-th function object, and n is the number of function objects in workflow R.
[0064] D is the data object set of workflow R, where d i is the i-th data object (input data or output data), and m is the number of data objects in workflow R.
[0065] E is the set of execution instances of workflow R, where e i is the i-th execution instance, and N is the number of execution instances in the workflow R. For an execution instance in a workflow, its function object f must belong to the workflow function set F, and the data object corresponding to its input / output data (i.e., d - and d + ) must belong to the data object set D of workflow R, and no two execution instances have the same output data object (that is, the data object d corresponding to the output data). + ).Right now, The following conditions are met:
[0066] a)f i ∈F;
[0067] b)
[0068] c)
[0069] d)
[0070] D* represents the set of output data of the workflow as a whole, which is a subset of the output data of all execution instances in the workflow. Specifically, part of the output data of the execution instance belongs to the intermediate result data, which needs to be reused as input data when executing the subsequent execution instance; the other part belongs to the output data of the workflow as a whole, which is part of the final output result of the workflow. in An output data output by a certain execution instance in the workflow. is the label of this output data, and M is the total number of output data of the workflow. D* satisfies the following conditions:
[0071] The output data of the workflow output has two different labels.
[0072] The output data of a workflow output must be the output data of an execution instance in the workflow.
[0073] The data objects in the workflow can be divided into two categories: external input data of the workflow and intermediate result data, according to whether they are output data of an execution instance. * ), has the following definition:
[0074] Definition 4.1: For a data object d∈D, if there does not exist e∈E such that d∈output(e), then d is called an external input data of workflow R. The input data set of workflow R is defined as INPUT(R).
[0075] Definition 4.2: For a data object d∈D, if there exists e∈E such that d∈output(e), then d is called an intermediate result data of workflow R. Define the intermediate result set (the set of intermediate result data) of workflow R as INTERMEDIATE(R),
[0076] Corollary 4.3:
[0077]
[0078] In the embodiment of the present application, in order to achieve the reproduction of the data processing method, the scientific data workflow system executes the isomorphic second workflow according to the first workflow indicated by the execution request, obtains the final output result of the second workflow (that is, the final result obtained by re-executing the first data processing method), and realizes the reproduction of the entire process of the first data processing method. The workflow execution engine obtains the execution instance of the second workflow isomorphic to the first workflow according to the execution request and the target digital object (data stored in the form of digital objects belonging to the field of scientific data); then the execution instance scheduler uses the target digital object to execute the execution instance of the second workflow to obtain the final output result of the second workflow.
[0079] In a possible implementation, the first workflow is isomorphic to the second workflow, indicating that the input data of each execution instance of the first workflow and the second workflow are the same, the calculation logic is the same, and the output data are equivalent; the output data equivalence indicates that the difference between the output data output by the corresponding execution instances of the first workflow and the second workflow is within a preset error range.
[0080] In the present embodiment, the isomorphic workflows have the same input data and calculation logic (the function objects of the two workflows correspond one to one, and the execution logic c is equivalent), so the two isomorphic workflows should have "equivalent" output results. For two isomorphic workflows, when the input data is the same and the execution logic of the function is the same, the output data may be different but equivalent because of the randomness of the parameters or calculations in the function (for example, parameter 1 in the function is a random number between 0 and 1, and the value of parameter 1 in the previous execution process is 0.2, and the value of parameter 1 in another execution process is 0.5.). The actual size of the error range can be determined by the specific application and is not limited in the embodiments of the present application.
[0081] In a possible implementation, the workflow execution engine obtains an execution instance of the second workflow that is isomorphic to the first workflow under the following constraints:
[0082] The relationship function between the function object set of the first workflow and the function object set of the second workflow is a bijective function;
[0083] The relationship function between the data object set of the first workflow and the data object set of the second workflow is a bijective function; the data object set includes: external input data and intermediate result data, the external input data represents data objects that do not belong to output data of any execution instance in the workflow, and the intermediate result data represents data objects that belong to output data of any execution instance in the workflow;
[0084] The relationship function between the set of execution instances of the first workflow and the set of execution instances of the second workflow is a bijective function;
[0085] The relationship function between the output data set of the first workflow and the output data set of the second workflow is a bijective function; the output data represents the result data output by the execution instance in the workflow, and each output data carries a corresponding output label;
[0086] For any two one-to-one corresponding function objects in the first workflow and the second workflow, the execution logic is the same, the input parameter labels are the same, and the output parameter labels are the same;
[0087] For any two one-to-one corresponding data objects in the first workflow and the second workflow, when both data objects are the external input data, the two are the same;
[0088] For any two one-to-one corresponding data objects in the first workflow and the second workflow, when the data objects are both the intermediate result data, the execution instances that output the intermediate result data correspond one-to-one, and the output tags carried by the intermediate result data are the same;
[0089] For any two one-to-one corresponding execution instances in the first workflow and the second workflow, the function objects described by the execution instances correspond one-to-one, the input data of the execution instances correspond one-to-one, and the output data of the execution instances correspond one-to-one;
[0090] For the first workflow and the second workflow, any two one-to-one corresponding input data carry the same input label, and any two one-to-one corresponding output data carry the same output label.
[0091] In order to express the recurrence relationship between two workflows (such as the first workflow and the second workflow), the embodiment of the present application provides the following definition 5 for the isomorphism of the two workflows. With the following definition 5 as a constraint condition, the workflow execution engine obtains an execution instance of the second workflow that is isomorphic to the first workflow under the following constraint condition.
[0092] Definition 5: For workflows R1 = (F1, D1, E1, D1*) and R2 = (F2, D2, E2, D2*), R1 and R2 are isomorphic if and only if there exists a function h F :F1→F2,h D :D1→D2,h E :E1→E2 and h D *: D1*→D2*, and the following conditions are met:
[0093] Definition 5.1, h F is a bijective function. That is, there exists satisfy:
[0094] and The relationship function h between the function object set F1 of the first workflow and the function object set F2 of the second workflow Fis a bijective function. A bijective function is a mapping relationship between two sets that satisfies both single injection and surjectiveness, that is, a bijective relationship, also known as a one-to-one mapping. For example, a mapping relationship from set X to set Y is called a bijective function, which satisfies: for any element y in set Y, there is only one element x in set X such that y=f(x).
[0095] Definition 5.2, h D is a bijective function. The relationship function h between the data object set D1 of the first workflow and the data object set D2 of the second workflow D is a bijective function;
[0096] Definition 5.3, h E is a bijective function. The relationship function h between the execution instance set E1 of the first workflow and the execution instance set E2 of the second workflow E is a bijective function.
[0097] Definition 5.4, h D* is a bijective function. The relationship function h between the output data set D1* of the first workflow and the output data set D2* of the second workflow D* is a bijective function.
[0098] Definition 5.5: For all f1∈F1, f2∈F2, if h F (f1) = f2, Then we have:
[0099] a) c1 and c2 are equivalent;
[0100] b)
[0101] c)
[0102] For any two one-to-one corresponding function objects f1 and f2 in the first workflow and the second workflow, the execution logic c1 of the function object f1 is equivalent to the execution logic c2 of the function object f2 (the two execution logics c1 and c2 are equivalent, indicating that the difference between the two execution logics is within a preset difference range); the input parameter label of the input parameter of the function object f1 Input argument labels corresponding to the input arguments of the function object f2 Same as the output parameter labels of the output parameters of the function object f1 Output parameter labels corresponding to the output parameters of function object f2 same.
[0103] Definition 5.6: For all d1∈D1,d2∈D2, if h D(d1) = d2, then the following conditions are met:
[0104] a) If d1∈INPUT(R1), then d2∈INPUT(R2), and d1=d2. For any two one-to-one corresponding data objects d1 and d2 in the first workflow and the second workflow, when one of the data objects d1 and d2 is external input data (a data object that does not belong to the output data of any execution instance in the workflow, corresponding to the specific content of definition 4.1 above), the other one also belongs to external input data, and the two d1 and d2 are the same.
[0105] b) If d1∈INTERMEDIATE(R1), then d2∈INTERMEDIATE(R2), and h E (e1) = e2, let There is a label l that satisfies the conditions: For any two one-to-one corresponding data objects d1 and d2 in the first workflow and the second workflow, when one of the data objects d1 and d2 is intermediate result data (a data object belonging to the output data of any execution instance in the workflow, corresponding to the specific content of definition 4.2 above), the other one also belongs to the intermediate result data, and the execution instances e1 and e2 of the outputs d1 and d2 correspond one-to-one, and the output labels l carried by d1 and d2 as the output data of the execution examples are the same (the output labels correspond to the specific content of definition 3 above).
[0106] Definition 5.7: For all e1∈E1, e2∈E2, let If h E (e1)=e2, then the following conditions are met:
[0107] a)h F (f1)=f2; for any two one-to-one corresponding execution instances e1 and e2 in the first workflow and the second workflow, the function objects f1 and f2 described by the execution instances e1 and e2 respectively correspond to each other one-to-one.
[0108] b) like but satisfy and
[0109] c) like but satisfy and
[0110] For any two one-to-one corresponding execution instances e1 and e2 in the first workflow and the second workflow, the input data of the execution instances e1 and e2 and One-to-one correspondence; any two one-to-one corresponding input data and The input tag l carried - same.
[0111] d) like but satisfy and
[0112] e) like but satisfy and
[0113] For any two one-to-one corresponding execution instances e1 and e2 in the first workflow and the second workflow, the output data of the execution instances e1 and e2 and One-to-one correspondence; any two one-to-one corresponding output data and The output label l carried + same.
[0114] Definition 5.8: For all like The following conditions are met:
[0115] a) like but satisfy and
[0116] b) like but satisfy and
[0117] A set of output data of the first workflow and the second workflow and The output data in corresponds one to one, Corresponds to And the two output data are one-to-one corresponding and The tag carried * same.
[0118] According to the above constraint conditions (Definition 5), the workflow execution engine executes the second workflow to re-execute the first data processing method and obtain the execution result after re-execution (the final output result of the second workflow). In a possible implementation, the workflow execution engine is further used to determine a candidate execution instance from the execution instance of the second workflow, the candidate execution instance represents an unexecuted execution instance whose corresponding input data has been stored locally; put the candidate execution instance into a work queue;
[0119] The execution instance scheduler selects a computing service, executes the candidate execution instance in the work queue, and obtains candidate output data;
[0120] After receiving the candidate output data, the workflow execution engine marks the completed candidate execution instance as a completed execution instance, removes it from the work queue, and stores the candidate output data locally; repeats the above steps until the last execution instance of the second workflow is completed, thereby obtaining the final output result of the second workflow.
[0121] In this embodiment, the workflow execution engine supports two operations: creating a new execution instance and reproducing the original workflow. For the former, the workflow execution engine directly sends the required input data request to the data object manager, and sends the execution request to the execution instance scheduler, which selects the appropriate computing service to complete the execution and obtain the execution result. For the latter, the workflow execution engine and the execution instance scheduler need to follow the above steps to complete the execution scheduling of each execution instance of the second workflow. The specific process is as follows:
[0122] Reference Figure 2 , Figure 2 A scientific data workflow system architecture diagram is shown, Figure 2 As shown, after the target digital object is obtained through the digital object cache module by the data object manager, the target digital object is preprocessed in different ways according to the type of input data, and the appropriate storage service is selected through collaborative scheduling with the execution instance scheduler to store the target digital object as an input data set locally.
[0123] The workflow executor determines whether the input data set (all input data) of the execution instance in the current second workflow has been stored locally. The execution instance with the prepared input data set is determined as a candidate execution instance and put into the work queue.
[0124] The execution instance scheduler selects a suitable computing service (e.g., a computing service with sufficient computing resources) according to the user-defined scheduling strategy, executes the candidate execution instance in the work queue, that is, obtains the input data set corresponding to the execution instance stored locally, executes the calculation of the function corresponding to the execution instance, and obtains the candidate output data. In this process, the execution instance scheduler monitors the execution of the function throughout the life cycle of the function execution (i.e., the entire execution process of the second workflow).
[0125] After each execution instance is completed, the execution instance scheduler sends the candidate output data (execution results) returned by the computing service to the workflow executor. After receiving the candidate output data, the workflow executor stores the candidate output data locally, marks the completed candidate execution instance as a completed execution instance, removes it from the work queue, and marks the output data of this execution instance (candidate output data) as completed.
[0126] For each execution instance in the workflow, the input data set of some execution instances comes entirely from external data. Therefore, after the data object manager acquires the target digital object and stores it locally, the input data set of this part of the execution instance is ready and can be executed. The input data set of another part of the execution instance may not come entirely from external data, but depends on the output data of some previously completed execution instances. Exemplarily, the input data of execution instance B is the output data of execution instance A, so execution instance A needs to be completed first, and after obtaining output data a, output data a is used as the input data of execution instance B.
[0127] Therefore, after each execution instance is completed, the above operation needs to be repeated, that is, the workflow executor determines whether there is a candidate execution instance whose input data set is ready and has not been executed, adds it to the work queue, and schedules it for execution by the execution instance scheduler to obtain candidate output data and store it locally. Until the last execution instance of the second workflow is completed, the output data obtained by the completion of the last execution instance is the final output result of the second workflow.
[0128] A workflow represents the complete process of a scientific data calculation, which includes data objects, function objects, execution instances and output data sets (corresponding to the content of definition 4 above). The scientific data workflow system proposed in the embodiment of the present application supports the reproduction of an original workflow that has been executed. Based on the above formal definition, the reproduction process for a workflow is the process of creating a workflow isomorphic thereto. In the process of reproducing the original workflow (first workflow), the system calls the function equivalent to the execution logic of the execution instance of the original workflow with equivalent input data as a parameter, thereby continuously creating equivalent execution instances, and finally creating a new workflow (second workflow) isomorphic to the original workflow (first workflow). By checking whether the final output result of the new workflow is equivalent to the final output result of the original workflow, it is possible to check whether the reproduction is successful.
[0129] In a possible implementation, the target digital object at least includes: a data view object or a function digital object;
[0130] The data view object is used to provide a unified access interface for the functions in the workflow to obtain the processing logic of the digital object;
[0131] The function digital object includes at least one of the following:
[0132] A first list whose element type is a string, wherein the elements in the first list are labels of input data of the function digital object;
[0133] A second list whose element type is a string, wherein the elements in the second list are labels of output data of the function digital object;
[0134] Metadata of type string, indicating the type of execution logic of the function digital object.
[0135] In this embodiment, the scientific data workflow system is implemented based on the digital object architecture, that is, it is built on the Internet of Things infrastructure. The ultimate sources of data (data objects, such as input data) and code (such as function objects) used in the system are other digital object repositories in the Internet of Things. Figure 2As shown, the data network infrastructure at least includes: a scientific research code warehouse, an industrial data warehouse and a public data warehouse, which are used to provide the relevant data required for the execution request. The data network infrastructure also includes an identification resolution system for resolving the identification information carried in the data request to retrieve the required target digital object. In the scientific data workflow, the digital objects (target digital objects) from the data network infrastructure mainly have two roles: data digital objects and code digital objects. Among them, the code digital object is the source of the execution logic of the function object in the workflow (corresponding to the content of definition 1 above), and the code digital object contains a program, and its data entity can be a link to a git (open source distributed version control system) warehouse, a compiled binary file, a source code compressed package and other forms. The data digital object is the source of the data object in the workflow (corresponding to the content of definition 2 above), and the data entity of the data digital object can be in various formats such as text, pictures, videos, and data streams.
[0136] like Figure 3 As shown, Figure 3 A schematic diagram of a scientific data workflow is shown. Figure 3 The proposed workflow represents the process of training and testing a deep learning model and finally publishing the test results. Figure 3 As shown, first obtain external digital objects, including data digital objects and code digital objects, where the data digital objects are used as training data and test data for the deep learning model, and the code digital objects are used as training functions and test functions. The model training and testing process belongs to the execution process of the corresponding execution instance. Figure 3 As shown, the deep learning model is first trained using training data in the form of digital objects and training functions in the form of digital code objects to obtain trained model parameters (which may be model parameters in the form of digital data objects). Then, the trained deep learning model is tested using test data in the form of digital data objects and test functions in the form of digital code objects, and the test results are finally output as the final output results of the workflow.
[0137] In the related art, the digital objects in the digital network have complex structures, diverse sources, no unified data format, and often require various authentication mechanisms to achieve the acquisition of digital objects. The digital network supports a flexible data ownership mechanism, which also makes it difficult to access digital objects through simple logic automation programs, and the results calculated by these digital objects also need to consider copyright issues when publishing. These characteristics make it difficult for data digital objects and code digital objects in the digital network to be directly used by workflows. In response to the above problems, the embodiments of the present application propose to use data view objects and function digital objects to allow heterogeneous digital objects in the digital network to be used in the system in the same way without saving copies of digital objects.
[0138] In the embodiment of the present application, the data object is an object used as a function input / output in the workflow. In the scientific data workflow system of the present application, all data objects are represented as a set of key-value pairs, and each key-value pair represents a data item v and its corresponding label k in the data object d (corresponding to the content of definition 2 above). Such a modeling method makes it possible for data objects to easily support data in various formats, such as image data sets, single compressed files, directory trees, etc. At the same time, modeling the data with data items as granularity also makes it possible to express data transformations with data items as granularity in the workflow.
[0139] A data view (i.e., a data view object) is a special data object. A data view is based on an external digital data object, processes digital objects of different formats in a corresponding manner, and provides a unified access interface with other data objects to functions in the workflow. A data view can be seen as an abstraction of digital objects obtained from the Internet of Things. The data view does not save the data entity of the digital object, but only the processing logic of the digital object. During calculation (i.e., during the execution of the workflow execution instance), the system calculates the intermediate results on demand according to the processing logic in the data view, and handles the data ownership issue at the same time.
[0140] As a digital object, a data view object contains type metadata, which is a metadata of type string, indicating the type of the underlying storage of the data object. Each data item in the data object is saved as an element in the data entity. The id of this element is the key of this data item, and the content of this element is the value of this data item.
[0141] The target digital object may also include a function digital object, which is used as a function object in the workflow. The function object processes the data (input data) in the workflow and outputs the processed new data (output data). A function object contains three parts: execution logic, input parameters and output parameters (corresponding to the content of definition 1 above). The execution logic of the function object can be based on an external code digital object, or it can be a processing logic built into a system. Exemplarily, the function object can be based on an external git repository, and the execution environment of the code in the git repository is defined in the metadata of the function object in the form of a Docker image, the input parameters and output parameters of the function are defined in the form of input and output mount points, and the execution logic of the function is defined in the form of a shell command that executes the code in the git repository in the data entity. Alternatively, the function object can be based on a processing logic built into the system. For example, the system provides a built-in function for subsetting data objects, and its input parameter is the label of all data items in the target subset. Then the function object can set the built-in function as a subset function in the metadata, and save the label of the data item in the target subset in the data entity.
[0142] As a number object, a function object (i.e. a function number object) contains the following metadata:
[0143] A first list Input Parameters whose element type is a string, wherein the elements in the first list are labels of input data of the function digital object;
[0144] a second list Output Parameters whose element type is a string, wherein the elements in the second list are labels of output data of the function digital object;
[0145] The metadata Type of the string type indicates the type of the execution logic of the function digital object. The execution logic of the function digital object is stored in its data entity.
[0146] An execution instance is the smallest unit of computation in a workflow, describing the execution process of a function. An execution instance contains a function, a set of input data sets and their labels, and a set of output data sets and their labels (corresponding to the content of Definition 3 above). When creating an execution instance in the system, given a function and several input data sets, the system selects an execution method based on the type of function, passes the input data to the corresponding parameters of the function according to the label, and creates the output as the corresponding output data after the function execution is completed.
[0147] As a digital object, an execution instance contains the following metadata:
[0148] Function metadata Function, that is, a metadata of the string type, indicates the identifier of the function executed in the execution instance.
[0149] Input metadata Input Datasets, that is, a dictionary with both keys and values of string type, represents the input data of the execution instance. The key in the dictionary is the label of the input data, and the value is the identifier of the corresponding input data object.
[0150] Output metadata Output Datasets, that is, a dictionary with both keys and values of string type, represents the output data of the execution instance. The key in the dictionary is the label of the output data, and the value is the identifier of the corresponding output data object.
[0151] In a possible implementation, the data object manager is further configured to set a corresponding cache expiration time according to the type of the target digital object; and to mark the stored target digital object as invalid when the actual cache time of the target digital object exceeds the cache expiration time.
[0152] In this embodiment, since the use of digital objects in the execution process of the scientific data workflow has time locality, the data object manager saves the most recently used digital objects (i.e., target digital objects) locally through the digital object cache module, thereby reducing the communication traffic between the data network infrastructure and the data network infrastructure. The digital object cache module supports read cache and write cache. The data object manager uses the digital object interface protocol (Digital Object Interface Protocol, DOIP) for a period of time and saves the target digital objects obtained from the data network infrastructure locally through the digital object cache module. Before each DOIP Retrieve operation, it checks whether the acquired target digital object has been saved locally. If there is already such an object in the cache, the object in the cache is directly returned. For write operations on digital objects, such as DOIP create operations (DOIP Create), DOIP delete operations (DOIP Delete), and DOIP update operations (DOIPUpdate), the digital object cache module first saves the digital object to be written locally (for example, the output data obtained after the execution instance of the second workflow is completed is saved locally), and when the written data reaches a certain amount, it is sent to the warehouse together, and the batch operation is used for the data in the same warehouse to reduce additional overhead.
[0153] Since different types of digital objects have different timeliness requirements, the data object manager supports setting different cache expiration times for different types of digital objects. If the difference between the cached timestamp and the current timestamp (i.e., the actual cache duration of the target digital object) exceeds the cache expiration time, the digital object in the cache will be marked as invalid, and when a new DOIP.Retrieve request arrives, it will be retrieved from the data network infrastructure again. Through the above method, while ensuring consistency to a certain extent, it greatly improves the system's read and write performance for digital objects.
[0154] In a possible implementation, the system further includes a scientific research workflow warehouse, which is used to store the function objects, data objects, execution instances and final output results of the second workflow in the form of digital objects;
[0155] The scientific research workflow warehouse is also used to obtain the digital object requested by the user according to the data acquisition request sent by the client.
[0156] In this embodiment, the system also includes a scientific research workflow warehouse, which is equivalent to a digital object warehouse. The data objects, function objects, execution instances, workflows, etc. in the system are all stored in the scientific research workflow warehouse. Exemplarily, after the system completes the execution of the second workflow and obtains the final output result of the second workflow, the function objects, data objects, execution instances and final output results of the second workflow are stored in the form of digital objects in the scientific research workflow warehouse. Therefore, when the user sends an execution request through the client to request to re-execute the first workflow, the relevant information of the first workflow can be obtained through the scientific research workflow warehouse to create an isomorphic second workflow. The scientific research workflow warehouse uses the MongoDB document database as the underlying storage and provides various operations specified by the DOIP protocol to the upper layer. Figure 2 As shown, the user can send a data acquisition request through the DOIP protocol of the client, access the scientific research workflow warehouse through the controller, and thus obtain the digital objects in the system.
[0157] In a possible implementation, the system further includes an object retrieval engine, which is used to perform a relationship-based retrieval of stored digital objects based on a digital object relationship index, or to perform a keyword-based retrieval of stored digital objects based on a digital object keyword index.
[0158] In this embodiment, the object retrieval engine is equivalent to a digital object registry, supporting relationship-based retrieval and keyword-based retrieval of various digital objects in the system. The system indexes the keywords in the metadata of various digital objects in the system through the object retrieval engine (such as the lucene engine), thereby realizing keyword-based search; the system stores the relationship between various digital objects in the workflow through a graph database (such as the Neo4j graph database), and realizes a variety of graph database search methods including path pattern matching through the object retrieval engine, thereby realizing relationship-based search.
[0159] The embodiment of the present application proposes a scientific data workflow system based on the Internet of Things. The system makes the originally closed data sets discoverable and accessible through the interface provided by the Internet of Things infrastructure, and obtains the target digital objects required for executing the workflow, thereby greatly enhancing the availability of the data. At the same time, through the formal definition of the scientific data workflow (the first workflow, which is composed of a set of function objects, data objects, execution instances and the final output results of the first workflow), the data processing method (a method of using the first scientific data to perform one or more stages of function mapping to obtain the first output data) is converted into the form of a workflow. The system can describe the data processing method in the research in a structured manner, which not only improves the clarity and accuracy of the method description, but also makes automated reproduction possible.
[0160] The second aspect of the embodiment of the present application also provides a scientific data workflow method based on the Internet of Things, referring to Figure 4 , Figure 4 A flowchart of a scientific data workflow method based on the Internet of Things is shown. Figure 4 As shown, the method includes:
[0161] Step S401, receiving an execution request sent by a client, wherein the execution request is used to request re-execution of a first workflow; wherein the first workflow is composed of a set of function objects, data objects, execution instances, and final output results of the first workflow, and the first workflow is used to describe the execution process of a first data processing method, wherein the first data processing method represents a method of performing function mapping of one or more stages using first scientific data to obtain first output data;
[0162] Step S402, according to the execution request, obtaining a target digital object from a digital network infrastructure, and storing the target digital object locally; the target digital object includes: the first scientific data and function used by the first data processing method;
[0163] Step S403, acquiring an execution instance of a second workflow isomorphic to the first workflow according to the execution request and the target digital object;
[0164] Step S404, using the target digital object to execute the execution instance of the second workflow to obtain the final output result of the second workflow;
[0165] Step S405: Return the final output result of the second workflow as the execution result of the execution request to the client.
[0166] In a possible implementation, acquiring an execution instance of a second workflow isomorphic to the first workflow according to the execution request and the target digital object, executing the execution instance of the second workflow using the target digital object, and obtaining a final output result of the second workflow includes:
[0167] Determine a candidate execution instance from the execution instances of the second workflow, the candidate execution instance representing an unexecuted execution instance whose corresponding input data has been stored locally;
[0168] Putting the candidate execution instance into a work queue;
[0169] Select a computing service, execute the candidate execution instance in the work queue, and obtain candidate output data;
[0170] After receiving the candidate output data, marking the completed candidate execution instance as a completed execution instance, removing it from the work queue, and storing the candidate output data locally;
[0171] The above steps are repeated until the last execution instance of the second workflow is completed, and the final output result of the second workflow is obtained.
[0172] In a possible implementation, the first workflow is isomorphic to the second workflow, indicating that the input data of each execution instance of the first workflow and the second workflow are the same, the calculation logic is the same, and the output data are equivalent; the output data equivalence indicates that the difference between the output data output by the corresponding execution instances of the first workflow and the second workflow is within a preset error range.
[0173] In a possible implementation, under the following constraints, an execution instance of the second workflow is obtained that is isomorphic to the first workflow:
[0174] The relationship function between the function object set of the first workflow and the function object set of the second workflow is a bijective function;
[0175] The relationship function between the data object set of the first workflow and the data object set of the second workflow is a bijective function; the data object set includes: external input data and intermediate result data, the external input data represents data objects that do not belong to output data of any execution instance in the workflow, and the intermediate result data represents data objects that belong to output data of any execution instance in the workflow;
[0176] The relationship function between the set of execution instances of the first workflow and the set of execution instances of the second workflow is a bijective function;
[0177] The relationship function between the output data set of the first workflow and the output data set of the second workflow is a bijective function; the output data represents the result data output by the execution instance in the workflow, and each output data carries a corresponding output label;
[0178] For any two one-to-one corresponding function objects in the first workflow and the second workflow, the execution logic is the same, the input parameter labels are the same, and the output parameter labels are the same;
[0179] For any two one-to-one corresponding data objects in the first workflow and the second workflow, when both data objects are the external input data, the two are the same;
[0180] For any two one-to-one corresponding data objects in the first workflow and the second workflow, when the data objects are both the intermediate result data, the execution instances that output the intermediate result data correspond one-to-one, and the output tags carried by the intermediate result data are the same;
[0181] For any two one-to-one corresponding execution instances in the first workflow and the second workflow, the function objects described by the execution instances correspond one-to-one, the input data of the execution instances correspond one-to-one, and the output data of the execution instances correspond one-to-one;
[0182] For the first workflow and the second workflow, any two one-to-one corresponding input data carry the same input label, and any two one-to-one corresponding output data carry the same output label.
[0183] In a possible implementation, the target digital object at least includes: a data view object or a function digital object;
[0184] The data view object is used to provide a unified access interface for the functions in the workflow to obtain the processing logic of the digital object;
[0185] The function digital object includes at least one of the following:
[0186] A first list whose element type is a string, wherein the elements in the first list are labels of input data of the function digital object;
[0187] A second list whose element type is a string, wherein the elements in the second list are labels of output data of the function digital object;
[0188] Metadata of type string, indicating the type of execution logic of the function digital object.
[0189] In a possible implementation, the method further includes:
[0190] A corresponding cache expiration time is set according to the type of the target digital object; and when the actual cache time of the target digital object exceeds the cache expiration time, the stored target digital object is marked as invalid.
[0191] In a possible implementation, the method further includes:
[0192] storing the function objects, data objects, execution instances and final output results of the second workflow in the form of digital objects;
[0193] According to the data acquisition request sent by the client, the digital object requested by the user is acquired.
[0194] In a possible implementation, the method further includes:
[0195] The stored digital objects are searched based on the relationship based on the digital object relationship index, or the stored digital objects are searched based on the keyword based on the digital object keyword index.
[0196] The present application also provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of an electronic device proposed in an embodiment of the present application. Figure 5 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the scientific data workflow method based on the Internet of Things disclosed in the embodiment of the present application.
[0197] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the scientific data workflow method based on the Internet of Things as disclosed in the embodiment of the present application are implemented.
[0198] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor is enabled to implement the steps of the scientific data workflow method based on the Internet of Things as disclosed in the embodiment of the present application.
[0199] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0200] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0201] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0203] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0204] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0205] The above is a detailed introduction to a scientific data workflow device, method and equipment based on digital networking provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A scientific data workflow system based on the Internet of Things, characterized in that: include: The controller is used to receive an execution request sent by a client, wherein the execution request is used to request re-execution of a first workflow; wherein the first workflow is composed of a set of function objects, data objects, execution instances, and final output results of the first workflow, and the first workflow is used to describe the execution process of a first data processing method, wherein the first data processing method represents a method of obtaining first output data by performing function mapping of one or more stages using first scientific data; A data object manager, configured to obtain a target digital object from a data network infrastructure according to the execution request, and store the target digital object locally; the target digital object includes: the first scientific data and function used by the first data processing method; A workflow execution engine, configured to obtain an execution instance of a second workflow isomorphic to the first workflow according to the execution request and the target digital object; An execution instance scheduler, used to execute the execution instance of the second workflow using the target digital object to obtain a final output result of the second workflow; The controller is further configured to return the final output result of the second workflow as the execution result of the execution request to the client.
2. The scientific data workflow system based on the Internet of Things according to claim 1, characterized in that: The workflow execution engine is further used to determine a candidate execution instance from the execution instance of the second workflow, wherein the candidate execution instance represents an unexecuted execution instance whose corresponding input data has been stored locally; and put the candidate execution instance into a work queue; The execution instance scheduler selects a computing service, executes the candidate execution instance in the work queue, and obtains candidate output data; After receiving the candidate output data, the workflow execution engine marks the completed candidate execution instance as a completed execution instance, removes it from the work queue, and stores the candidate output data locally; repeats the above steps until the last execution instance of the second workflow is completed, thereby obtaining the final output result of the second workflow.
3. The scientific data workflow system based on the Internet of Things according to claim 1, characterized in that: The first workflow is isomorphic to the second workflow, which means that each execution instance of the first workflow and the second workflow has the same input data, the same calculation logic, and equivalent output data; Output data equivalence indicates that a difference between output data outputted by two corresponding execution instances of the first workflow and the second workflow is within a preset error range.
4. The scientific data workflow system based on the Internet of Things according to claim 1, characterized in that: The workflow execution engine obtains an execution instance of the second workflow that is isomorphic to the first workflow under the following constraints: The relationship function between the function object set of the first workflow and the function object set of the second workflow is a bijective function; The relationship function between the data object set of the first workflow and the data object set of the second workflow is a bijective function; The data object set includes: external input data and intermediate result data, wherein the external input data represents a data object that does not belong to the output data of any execution instance in the workflow, and the intermediate result data represents a data object that belongs to the output data of any execution instance in the workflow; The relationship function between the set of execution instances of the first workflow and the set of execution instances of the second workflow is a bijective function; The relationship function between the output data set of the first workflow and the output data set of the second workflow is a bijective function; the output data represents the result data output by the execution instance in the workflow, and each output data carries a corresponding output label; For any two one-to-one corresponding function objects in the first workflow and the second workflow, the execution logic is the same, the input parameter labels are the same, and the output parameter labels are the same; For any two one-to-one corresponding data objects in the first workflow and the second workflow, when both data objects are the external input data, the two are the same; For any two one-to-one corresponding data objects in the first workflow and the second workflow, when the data objects are both the intermediate result data, the execution instances that output the intermediate result data correspond one-to-one, and the output tags carried by the intermediate result data are the same; For any two one-to-one corresponding execution instances in the first workflow and the second workflow, the function objects described by the execution instances correspond one-to-one, the input data of the execution instances correspond one-to-one, and the output data of the execution instances correspond one-to-one; For the first workflow and the second workflow, any two one-to-one corresponding input data carry the same input label, and any two one-to-one corresponding output data carry the same output label.
5. The scientific data workflow system based on the Internet of Things according to claim 1, characterized in that: The target digital object at least includes: a data view object or a function digital object; The data view object is used to provide a unified access interface for the functions in the workflow to obtain the processing logic of the digital object; The function digital object includes at least one of the following: A first list whose element type is a string, wherein the elements in the first list are labels of input data of the function digital object; A second list whose element type is a string, wherein the elements in the second list are labels of output data of the function digital object; Metadata of type string, indicating the type of execution logic of the function digital object.
6. The scientific data workflow system based on the Internet of Things according to claim 1, characterized in that: The data object manager is further configured to set a corresponding cache expiration time according to the type of the target digital object; and mark the stored target digital object as invalid when the actual cache expiration time of the target digital object exceeds the cache expiration time.
7. The scientific data workflow system based on the Internet of Things according to claim 1, characterized in that: The system further comprises a scientific research workflow warehouse, which is used to store the function objects, data objects, execution instances and final output results of the second workflow in the form of digital objects; The scientific research workflow warehouse is also used to obtain the digital object requested by the user according to the data acquisition request sent by the client.
8. The scientific data workflow system based on the Internet of Things according to claim 1, characterized in that: The system further comprises an object retrieval engine, which is used for performing a relationship-based retrieval on the stored digital objects based on the digital object relationship index, or performing a keyword-based retrieval on the stored digital objects based on the digital object keyword index.
9. A method for creating a scientific data workflow based on a data network, characterized in that: The method comprises: Receive an execution request sent by a client, wherein the execution request is used to request re-execution of a first workflow; wherein the first workflow is composed of a set of function objects, data objects, execution instances, and final output results of the first workflow, and the first workflow is used to describe the execution process of a first data processing method, wherein the first data processing method represents a method of performing function mapping of one or more stages using first scientific data to obtain first output data; According to the execution request, a target digital object is obtained from the Internet infrastructure, and the target digital object is stored locally; the target digital object includes: the first scientific data and function used by the first data processing method; Acquire, according to the execution request and the target digital object, an execution instance of a second workflow isomorphic to the first workflow; Utilizing the target digital object, executing the execution instance of the second workflow to obtain a final output result of the second workflow; The final output result of the second workflow is used as the execution result of the execution request and returned to the client.
10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the method for creating a scientific data workflow based on the Internet of Things as claimed in claim 9.
Citation Information
Cited By
Digital networking data space access system based on DOIP (Document Object Internet Protocol) caching mechanism
CN120455542A
