Operator distributed management sharing system and method based on digital object

Abstract

The invention relates to an operator distributed management sharing system and method based on a digital object, belongs to the technical field of big data computing, and is applied to a public security environment. The method comprises the following steps: 1, defining a metadata structure of a data pipeline operator, and creating an operator digital object type; 2, operator source code development is carried out according to a cross-language development interface of a data pipeline operator; 3, compiling a data pipeline operator source code to obtain a data pipeline operator executable module, and packaging the data pipeline operator executable module, an operator metadata definition file, a configuration file and the like; 4, registering the data pipeline operator as a digital object, and setting an access right for the digital object for operator sharing; 5, loading and operating a data pipeline operator; according to the definition operator developing and packaging method, the operator interoperation problem is solved, and cross-system multiplexing sharing is achieved; the operators are stored in a digital object library in a distributed manner and are endowed with globally unique identifiers, so that distributed management of the data pipeline operators is realized.

Description

Technical Field

[0001] The present invention relates to a digital object-based operator distributed management sharing system and method, belonging to the technical field of big data computing and applied in a public safety environment. Background Art

[0002] 1. In the public safety environment, the current status of the data pipeline operator system is:

[0003] With the rapid development of science and technology, public safety data processing plays an increasingly important role in protecting people's lives and property. Public safety management faces complex and ever-changing challenges. The acceleration of urbanization has led to an increase in population density, and problems such as traffic, environment, and public security have emerged one after another. The rapid development of big data technology has brought revolutionary changes to the field of public safety. As one of the important tools for big data processing, the data pipeline system can provide comprehensive data support for public safety management departments by integrating various data sources, such as monitoring data, traffic data, and environmental monitoring data, helping them to better cope with various challenges and improve management efficiency.

[0004] Data pipeline is a high-level abstraction of the big data processing process. It abstracts the data processing process into a series of operators and their connections. By shielding the underlying data processing details of the operators, general users without programming knowledge can construct data pipelines and run them on the cloud platform by splicing and configuring operators in a visual interface, thereby lowering the threshold for system use while making full use of the data processing capabilities of the underlying computing architecture. Data pipeline operators are components that perform specific operations on data, and operators can be reused in multiple data pipelines.

[0005] Digital Object Architecture (DOA) is a new computing architecture proposed by Robert E. Kahn, the winner of the Illustrated Prize and the inventor of the Internet, to achieve interoperability of heterogeneous information systems. It is a data governance infrastructure that identifies, resolves, manages and securely controls digital resources in a network environment. In the digital object architecture, information resources are abstractly represented as digital objects (DO), including data, resources, services, processes, etc. The digital object architecture provides a unique identifier (DOI) for each digital object to ensure that each object can be uniquely identified and referenced, and establishes a complete governance system to achieve secure, efficient and controllable interconnection and interoperability between digital objects, and has the ability to provide global unique identification, information resolution, information management and security control services for various physical entities and digital objects. The digital object architecture is mainly composed of an identification resolution system and a digital object library, providing metadata management, storage, version management, access control, search and discovery functions for digital objects. The digital object system includes two basic protocols: the Identifier / Resolution Protocol (IRP) and the Digital Object Interface Protocol (DOIP). There are three basic components in the digital object system, namely the Identifier / Resolution System, which is responsible for the identification and resolution of digital objects, the Repository System, which is responsible for the storage and access of digital objects, and the Registry System, which is responsible for the registration of digital object metadata and the search of digital objects. The Identifier Resolution System is the core part of the digital object system, which is mainly used for the registration, resolution and management of digital object identifiers. It has a global resolution platform and a segment management mechanism, and supports the provision of external services through the Digital Object Identifier Resolution Protocol (IRP). Currently, the Digital Object Identifier Resolution System on the Internet is the Handle system. In the digital object system, all digital objects are stored in the Digital Object Repository (Repository System), and support the provision of external services through the Digital Object Interface Protocol (DOIP). Cordra is an open source implementation project of the Digital Object Repository.

[0006] 2. In the public security environment, the data pipeline operator system has the following defects:

[0007] Data pipelines are widely used in various data processing systems in the field of public security. Each system defines its own data pipeline operator specifications according to application requirements and technical architecture. Operators developed by each system cannot interoperate due to different development specifications, so cross-system data pipeline operator sharing and reuse cannot be achieved. In addition, data pipeline operators are usually built into the operator library within each data processing system. There is no unified data pipeline operator library for distributed storage management and publishing sharing of operators.

[0008] 3. Therefore, how to achieve distributed management and cross-system sharing of data pipeline operators in the public security field has become an urgent problem to be solved. Summary of the invention

[0009] The purpose of the present invention is to address the technical issues of distributed management and cross-system sharing of data pipeline operators in the field of public security, and to propose a digital object-based operator distributed management and sharing system and method.

[0010] The objective of the present invention is achieved through the following technical solutions:

[0011] The present invention discloses a method for distributed management and sharing of operators based on digital objects, comprising the following steps:

[0012] Step 1: Define the metadata structure of the data pipeline operator and create the operator digital object type;

[0013] Step 1.1: Define the attribute name, attribute semantics, and attribute data type of the metadata structure of the data pipeline operator;

[0014] Step 1.2: Create the operator digital object type in the digital object library according to the data pipeline operator metadata structure and the digital object type creation process;

[0015] Step 2: Based on the operator's cross-language development interface, develop the data pipeline operator source code including the initialization interface, message processing interface, and destruction interface; the initialization interface is called when the data pipeline is created and initializes variables and resources; the message processing interface is called once when the previous operator in the data pipeline sends a message, which is used to process the message data; the destruction interface is called when the data pipeline stops to release resources;

[0016] Step 3: Compile the data pipeline operator source code to obtain the data pipeline operator executable module and the operator to be packaged file for packaging;

[0017] Step 3.1: Compile the data pipeline operator source code to obtain the data pipeline operator executable module;

[0018] Step 3.2: Fill the metadata information of the data pipeline operator into the metadata definition file template to obtain the operator metadata definition file;

[0019] Step 3.3: Encapsulate and package the files to be encapsulated of the data pipeline operator to obtain a data pipeline operator encapsulation package;

[0020] Step 4: Register the data pipeline operator as a digital object and set access permissions for the digital object for sharing of the operator;

[0021] Step 4.1: Use the metadata information of the data pipeline operator as the digital object attribute; use the data pipeline operator package described in step 3 as the digital object attachment; form the digital object attribute and the digital object attachment into a digital object through the digital object registration process and store it in the digital object library, and assign a unique identifier;

[0022] Step 4.2: Set access permissions for digital objects for operator sharing;

[0023] Step 5: Load and run the data pipeline operator;

[0024] Step 5.1: Based on the unique identifier of the operator digital object, obtain the metadata structure and data pipeline operator package of the data pipeline operator from the digital object library through the digital object access interface, disassemble the operator package and load the operator executable module, thereby completing the loading of the data pipeline operator;

[0025] Step 5.2: Call the initialization interface and message processing interface of the loaded data pipeline operator to implement the call of the operator message processing logic;

[0026] The present invention discloses a digital object-based operator distributed management and sharing system, which is used to implement the above method. The present invention discloses a digital object-based operator distributed management and sharing system, which includes an operator library module, an interface component module, an operator management module, an operator library management interface component module, and an operator execution engine module.

[0027] The operator library module is used for storage, identification and parsing, version management, and permission control of data pipeline operators; and serves as the input of the operator management module;

[0028] The interface component module is used to define the initialization interface, message processing interface and destruction interface that need to be implemented in the process of developing the data pipeline operator. The initialization interface is used to initialize variables and resources; the message processing interface is used for processing the message data received by the operator; the destruction interface is used to release resources and close the data pipeline;

[0029] The operator management module is used for cross-system operator library module management and serves as the input of the operator library management interface component and the operator execution engine module;

[0030] The operator library management interface component module is used for registration, deletion, update, version management, and permission control of data pipeline operators, and interacts with the operator management module;

[0031] The operator execution engine module is used to load and execute data pipeline operators.

[0032] Beneficial effects:

[0033] Data pipeline operators are defined in digital object form, and unified operator interface functions and packaging specifications are defined for different development languages ​​to solve the interoperability problem of heterogeneous operators and realize cross-system reuse and sharing of data pipeline operators. By distributing and storing operators in the digital object library and assigning globally unique identifiers, distributed registration, publishing, and search and discovery of operators can be performed based on the digital object access protocol, and access control can be performed on operators according to digital object access rights, realizing distributed and unified management of data pipeline operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the process of the present invention;

[0035] Figure 2 Compile and package the flow chart for the data pipeline Java operator of the present invention;

[0036] Figure 3 Packaging flow chart for the data pipeline Python operator of the present invention;

[0037] Figure 4 This is a visualization interface diagram of operator library management of the present invention;

[0038] Figure 5 This is a visualization interface diagram of operator registration of the present invention;

[0039] Figure 6 This is a visualization interface diagram of operator version management of the present invention. DETAILED DESCRIPTION

[0040] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0041] Example

[0042] like Figure 1 As shown, the operator distributed management and sharing method based on digital objects of the present invention has the following specific implementation steps:

[0043] Step 1: Define the metadata structure of the data pipeline operator and create the operator digital object type;

[0044] Step 1.1: Define the attribute name, attribute semantics, and attribute data type of the metadata structure of the data pipeline operator;

[0045] Step 1.2: Create the operator digital object type in the digital object library according to the data pipeline operator metadata structure and the digital object type creation process;

[0046] In the embodiment, the attribute name, attribute semantics, and attribute data type of the metadata structure of the data pipeline operator are first defined. The data pipeline operator metadata includes the following attributes:

[0047] (1) name: operator name, which is a UTF8-encoded string with a maximum length of 255. It is used to distinguish different operators. Different versions of the same operator should have the same name.

[0048] (2) version: operator version, which is a utf8-encoded string with a maximum length of 255. It is used to identify different versions of the same operator. Versions are usually defined in ABC format, where A is the major version number, B is the minor version number, and C is the revision number.

[0049] (3) language: operator development language, which is a utf8-encoded string with a maximum length of 255. It is used to identify the operator development language, including Java and Python.

[0050] (4) languageVersion: Operator development language version: The type is a utf8-encoded string with a maximum length of 255. It is used to identify the operator development language version, for example: Jdk8, Python3.7.

[0051] (5) organization: operator development unit, the type is a UTF8-encoded string with a maximum length of 65535. It is used to store the name of the operator development unit.

[0052] (6) description: operator description, which is a utf8-encoded string with a maximum length of 65535. It is used to store a text description of the operator's purpose, parameter semantics, data processing flow, etc.

[0053] (7)entry: operator entry, the type is a utf8-encoded string with a maximum length of 65535. It is used to specify the loading of the operator executable module. For Java language operators, the full path of the operator class name is saved in the entry; for Python language operators, the Python source code file name containing the operator class definition is saved in the entry.

[0054] (8) initParam: operator initialization parameter list definition, the type is a utf8-encoded string with a maximum length of 65535. The operator initialization parameters are passed to the operator when the operator is initialized, and can be configured in the data pipeline editing interface. The string contains a Json array, and each item in the array stores an initialization parameter definition of the operator, including: initialization parameter name, initialization parameter data type, and initialization parameter default value.

[0055] (9) inputParam: operator input parameter list definition, the type is a utf8-encoded string with a maximum length of 65535. The operator input parameter is the input message field during the operation of the operator. The input message is a Json object, and each attribute of the object corresponds to a parameter in the operator input parameter list. The string contains a Json array, and each item in the array stores an input parameter definition of the operator, including: input parameter name, input parameter data type, and input parameter default value.

[0056] (10)outputParam: operator output parameter list definition, the type is a utf8-encoded string with a maximum length of 65535. The operator output parameter is the output message field during the operation of the operator. The output message is a Json object, and each attribute of the object corresponds to a parameter in the operator output parameter list. The string contains a Json array, and each item in the array stores an output parameter definition of the operator, including: output parameter name, output parameter data type, and output parameter default value.

[0057] Secondly, based on the data pipeline operator metadata structure definition, the data pipeline operator digital object type is created in the digital object library according to the digital object type creation process of the digital object library. The digital object type of the digital object library is defined by a Json object, and the property list of the digital object type is set in the properties of the Json object. The definition of the data pipeline operator digital object type is as follows:

[0058]

[0059]

[0060] Step 2: Based on the operator's cross-language development interface, develop the data pipeline operator source code including the initialization interface, message processing interface, and destruction interface; the initialization interface is called when the data pipeline is created and initializes variables and resources; the message processing interface is called once when the previous operator in the data pipeline sends a message, which is used to process the message data; the destruction interface is called when the data pipeline stops to release resources;

[0061] In the embodiment, the operator development interface is illustrated by using Java language and Python language.

[0062] (1) Java language data pipeline operator development interface: When an operator is developed in Java language, it is necessary to inherit the AbstractProcessor class and implement the interface functions of this class, including the initialization function, destruction function, and message processing function. The definition of the AbstractProcessor class is as follows.

[0063]

[0064]

[0065] The function description is as follows:

[0066] 1) The prepare function is the initialization function of the operator. It is called once when the data pipeline is created. Variables and resources can be initialized in this function. The initialization parameters are passed in through the function parameter initParams. The initParams parameter is a list of key-value pairs, each of which stores the name and value of an initialization parameter.

[0067] 2) The processRow function is the message processing function of the operator. It is called once when the previous operator in the data pipeline sends a message. Its parameter msg is the input message of the operator, and the return value is the output message of the operator. The generic types T and R represent the input message class and output message class of the operator respectively, which are defined by the operator developer and need to correspond to the input and output parameters in the operator metadata. If the operator is the first operator in the data pipeline, the function will be called continuously, and the input parameter of the function will be empty until the data pipeline is closed.

[0068] 3) The close function is the destruction function of the operator and is called once when the data pipeline is closed. Operations such as resource release can be performed in this function.

[0069] (2) Python language data pipeline operator development interface: When an operator is developed in Python, it is necessary to implement the Component class. The class needs to implement three function interfaces. The Component class is defined as follows:

[0070]

[0071] The functions are described as follows:

[0072] 1) The _init_ function is the initialization function of the operator. It is called once when the data pipeline is created. Variables and resources can be initialized in this function. The initialization parameters are passed in through the function parameter initParams. The initParams parameter is a key-value pair dictionary. Each key-value pair stores the name and value of an initialization parameter.

[0073] 2) The processRow function is the message processing function of the operator. It is called once when the previous operator in the data pipeline sends a message. Its parameter msg is the input message of the operator, and the return value is the output message of the operator. The input and output message types are both Json object strings, and the properties of the Json object need to correspond to the input and output parameters in the operator metadata. If the operator is the first operator in the data pipeline, the function will be called continuously, and the input parameters of the function will be empty until the data pipeline is closed.

[0074] 3) The close function is the destruction function of the operator and is called once when the data pipeline is closed. Operations such as resource release can be performed in this function.

[0075] Step 3: Compile the data pipeline operator source code to obtain the data pipeline operator executable module and the operator to be packaged file for packaging;

[0076] Step 3.1: Compile the data pipeline operator source code to obtain the data pipeline operator executable module;

[0077] Step 3.2: Fill the metadata information of the data pipeline operator into the metadata definition file template to obtain the operator metadata definition file;

[0078] Step 3.3: Encapsulate and package the files to be encapsulated of the data pipeline operator to obtain a data pipeline operator encapsulation package;

[0079] In the embodiment, the operator encapsulation and packaging process is as follows:

[0080] (1) Compile the data pipeline operator source code to obtain the data pipeline operator executable module. For operators developed in Java, the executable module is a jar package. For operators developed in Python, code compilation is not required.

[0081] (2) Write an operator metadata definition file according to the operator metadata definition file format. The operator metadata file is a Json format text file. The Json object contains the data pipeline operator metadata definition, including operator name, operator development language, operator development unit, operator entry class / file and other information. The example is as follows:

[0082]

[0083] (3) Figure 2 and Figure 3 As shown, create a data pipeline operator package. Create a blank directory, put the operator metadata definition file in the root directory, and name it schema.json. Create a subdirectory named component, and put the operator executable module and dependent library files in the component directory. For operators developed in Java, its executable module is a jar package, named component.jar, and placed in the component directory; for operators developed in Python, its executable module is a series of py source files, which are placed in the component directory. In addition, the site-packages directory in the virtual environment is also placed in the component directory, which stores the operator's third-party dependent library files. Put the configuration files and data files required for the operator to run in the component directory. Compress the above directories into a zip format compressed package using compression software to obtain the operator package.

[0084] Step 4: Register the data pipeline operator as a digital object and set access permissions for the digital object for sharing of the operator;

[0085] Step 4.1: Use the metadata information of the data pipeline operator as the digital object attribute; use the data pipeline operator package described in step 3 as the digital object attachment; form the digital object attribute and the digital object attachment into a digital object through the digital object registration process and store it in the digital object library, and assign a unique identifier;

[0086] Step 4.2: Set access permissions for digital objects for operator sharing;

[0087] In the embodiment, the registration of the data pipeline operator can be performed in the digital object library management interface, or in the operator registration interface of the data processing system, which is divided into the following situations:

[0088] (1) When registering an operator in the operator registration interface of the data processing system, the data processing system needs to implement the operator registration interface and implement the operator registration by calling the digital object creation interface of the digital object library. In the operator registration interface, the user enters information such as the operator package path, the target digital object library, and the user role that can access the operator. The system reads the operator metadata from the package and fills in the attribute data of the operator digital object based on the data pipeline operator metadata structure definition; calls the digital object creation interface of the digital object library to create a new data pipeline operator digital object with the operator digital object type as the base class, and the digital object library automatically assigns a unique identifier for the operator digital object; places the data pipeline operator package as an attachment in the operator digital object; and sets the access rights of the operator digital object based on the user role information of the operator that can be accessed entered by the user to implement the registration and release of the operator.

[0089] (2) When registering an operator in the digital object library management interface, the user needs to manually enter the operator metadata and upload the operator package in the digital object creation interface of the digital object library to realize the registration and release of the operator. The user logs in to the target digital object library management interface, opens the digital object creation interface of the operator digital object type, enters the operator metadata information in the interface, including name, version, development language and other information, and then uploads the operator package as an attachment to the newly created digital object to create a data pipeline operator digital object. The digital object library will automatically assign a unique identifier for the operator digital object. The user configures the access rights of the newly created digital object through the digital object permission configuration interface of the digital object library to realize the registration and release of the operator digital object.

[0090] Step 5: Load and run the data pipeline operator;

[0091] Step 5.1: Based on the unique identifier of the operator digital object, obtain the metadata structure and data pipeline operator package of the data pipeline operator from the digital object library through the digital object access interface, disassemble the operator package and load the operator executable module, thereby completing the loading of the data pipeline operator;

[0092] Step 5.2: Call the initialization interface and message processing interface of the loaded data pipeline operator to implement the call of the operator message processing logic;

[0093] In the embodiment, during the initialization of the data pipeline, the data processing system obtains the operator encapsulation package from the digital object library according to the digital object unique identifier of the operator in the data pipeline, loads the operator executable module during the operation of the data pipeline, and calls the interface functions of the operator as needed, thereby realizing the loading, operation and interoperation of the operator, which is specifically divided into the following stages:

[0094] (1) In the data pipeline loading phase, first, for the operator in the data pipeline, the digital object library to which the operator belongs is located according to the operator's digital object unique identifier; then, based on the digital object access interface, the operator metadata and the operator package in the attachment are read from the digital object library; finally, the operator executable module and running dependency library and other related files are extracted from the operator package and saved to the local file system.

[0095] (2) In the data pipeline initialization phase, for each operator in the data pipeline, the operator's executable module is loaded and the operator's initialization function is called. Due to the heterogeneity of operator development languages ​​and operating architectures, it is necessary to develop specific operator loading logic for each development language; in addition, in order to achieve isolation of the operator operating environment, the operator needs to be run in a separate container.

[0096] During the operation of the data pipeline, starting from the first operator, the operator's message processing function is called. When the operator's message processing result is obtained, the message processing function of each operator connected to the operator is called according to the data pipeline operator connection relationship. When the data pipeline is finished, the operator's destruction function is called for each operator in the data pipeline.

[0097] The present invention discloses a digital object-based operator distributed management and sharing system, which is used to implement the above method. The present invention discloses a digital object-based operator distributed management and sharing system, which includes an operator library module, an interface component module, an operator management module, an operator library management interface component module, and an operator execution engine module.

[0098] The operator library module is used for storage, identification and parsing, version management, and permission control of data pipeline operators; and serves as the input of the operator management module;

[0099] The interface component module is used to define the initialization interface, message processing interface and destruction interface that need to be implemented in the process of developing the data pipeline operator. The initialization interface is used to initialize variables and resources; the message processing interface is used for processing the message data received by the operator; the destruction interface is used to release resources and close the data pipeline;

[0100] The operator management module is used for cross-system operator library module management and serves as the input of the operator library management interface component and the operator execution engine module;

[0101] The operator library management interface component module is used for registration, deletion, update, version management, and permission control of data pipeline operators, and interacts with the operator management module;

[0102] The operator execution engine module is used to load and execute data pipeline operators.

[0103] In the embodiment, the operator distributed management sharing system adopts a B / S architecture with front-end and back-end separation, and the front-end and back-end interact using HTTP interface and Websocket protocol. The system front-end interface mainly includes the operator library management interface component module, the back-end mainly includes the operator management module and the operator running engine module, the operator library module is used for operator storage, and the interface component module is required during the operator development process. The description of each module is as follows:

[0104] (1) Operator library module, which implements distributed storage and publishing management of operators by distributing operators to the Cordra digital object library in the digital object system. The operation of the digital object library is mainly completed through the digital object interface protocol DOIP (Digital Object Interface Protocol). In the Cordra system, two digital object operation interfaces based on DOIP are provided: the DOIP protocol implemented based on the HTTP interface; client libraries provided for various programming languages, and the official client libraries in Java and Javascript languages ​​are provided. The digital object operation interface provides basic operation interfaces, including: obtaining service information, identity authentication, adding, deleting, modifying and querying digital object types, adding, deleting, modifying and querying digital objects, enumerating and searching digital objects, setting access rights for digital objects, and version management of digital objects.

[0105] (2) The interface component module is provided to the developers of operators for the development of new operators, including the definition of the base class AbstractProcessor of Java operators and the base class Component of Python operators. Taking the JAVA operator as an example, during the operator development process, it is necessary to implement an operator class based on the base class AbstractProcessor and implement the three interface functions defined in the base class AbstractProcessor in the operator class: initialize variables and resources in the prepare function; process message data in the processRow function, and use the message processing result as the function return value; and release resources in the close function.

[0106] (3) The operator management module is a bridge between the operator library management interface and the operator library. On the one hand, it provides the operator management-related HTTP interface to the front-end operator library management interface; on the other hand, the Java client library based on Cordra interacts with Cordra and converts the operator-related management operations into operations on the operator digital objects in Cordra. Taking the operator registration operation as an example, the operator management module provides the operator registration interface to the front-end operator library management interface. When the front-end interface calls the interface, it needs to provide the basic metadata information of the operator and the operator package file; when the operator management module receives the operator registration interface call, it creates a new digital object in Cordra with the operator digital object type as the object type based on the operator metadata information provided by the interface call and the information in the operator package, fills in the operator metadata information, and adds the operator package to the attachment of the new digital object, thereby realizing the creation of the operator digital object. The interfaces provided by the operator management module to the front-end operator library management interface include: operator registration, operator enumeration, operator search, operator deletion, operator modification, operator version data acquisition and setting, operator access rights acquisition and setting, etc.

[0107] (4) The operator library management interface component module provides users with a visual interface for operator management. Operator management operations are implemented by calling the HTTP interface provided by the operator management module. The operator library management interface mainly includes the following parts: the operator list interface, which can display the existing operator list information in the operator library in pages, search the operator name, description, etc. based on keywords, and delete and modify the selected operators; the operator registration interface, in which users fill in the operator metadata information in the interface and upload the operator package to register a new operator; the operator version management interface, in which users can view the historical version information of the operator and set the current version of the operator; the operator permission management interface, in which users can view and set the list of user roles that can access the operator.

[0108] (5) The operator execution engine module is used to load and execute the data pipeline operators. In the data pipeline loading phase, for each operator in the data pipeline, the operator metadata and operator encapsulation package are obtained from the operator management module according to the operator's digital object ID, and the operator executable module and operation dependency library and other related files are extracted from the operator encapsulation package and saved to the local file system. In the data pipeline initialization phase, for each operator in the data pipeline, the operator's executable module is loaded and the operator's initialization function is called. During the operation of the data pipeline, starting from the first operator, the operator's message processing function is called. When the operator's message processing result is obtained, the message processing function of each operator connected to the operator is called according to the data pipeline operator connection relationship. When the data pipeline is completed, the operator's destruction function is called for each operator in the data pipeline.

[0109] In order to further illustrate the practicability of the present invention in the field of public security, the following description is given in the form of a visual interface.

[0110] like Figure 4 The figure shows the operator library management visualization interface of the present invention. In this interface, the basic information of all operators in the operator library that the current user has the right to view is displayed in a list format, where the digital object ID is the unique identifier of the operator digital object. Click the "Register" button to enter the operator registration interface, click the "Edit" button to enter the corresponding operator editing interface, click "Version Management" to enter the corresponding operator version management interface, and click the "Delete" button to delete the corresponding operator from the operator library.

[0111] like Figure 5 The figure shows the operator registration visualization interface of the present invention. In this interface, the user enters the basic metadata information of the operator, including: operator name, digital object repository, read / write role, icon and other information, and clicks the "Select File" button. After selecting the operator package file in the pop-up file selection dialog box, click the "OK" button to complete the operator registration process. The operator will be stored in the selected digital object library in the form of a digital object.

[0112] like Figure 6 The figure shows the operator version management visualization interface of the present invention. In this interface, the basic information of the current operator historical version is displayed in a list. Clicking the "Apply" button will use the selected historical version as the current version of the current operator. Clicking the "View" button will display the operator detailed information of the selected historical version in a pop-up dialog box.

[0113] The results show that the present invention realizes a digital object-based operator distributed management and sharing system and method.

Claims

1. A method for distributed management and sharing of operators based on digital objects, characterized in that: The following steps are included: Step 1: Define the metadata structure of the data pipeline operator and create the operator digital object type; Step 1.1: Define the attribute name, attribute semantics, and attribute data type of the metadata structure of the data pipeline operator; Step 1.2: Create the operator digital object type in the digital object library according to the data pipeline operator metadata structure and the digital object type creation process; Step 2: Based on the operator's cross-language development interface, develop the data pipeline operator source code including the initialization interface, message processing interface, and destruction interface; the initialization interface is called when the data pipeline is created and initializes variables and resources; the message processing interface is called once when the previous operator in the data pipeline sends a message, which is used to process the message data; the destruction interface is called when the data pipeline stops to release resources; Step 3: Compile the data pipeline operator source code to obtain the data pipeline operator executable module and the operator to be packaged file for packaging; Step 3.1: Compile the data pipeline operator source code to obtain the data pipeline operator executable module; Step 3.2: Fill the metadata information of the data pipeline operator into the metadata definition file template to obtain the operator metadata definition file; Step 3.3: Encapsulate and package the files to be encapsulated of the data pipeline operator to obtain a data pipeline operator encapsulation package; Step 4: Register the data pipeline operator as a digital object and set access permissions for the digital object for sharing of the operator; Step 4.1: Use the metadata information of the data pipeline operator as the digital object attribute; use the data pipeline operator package described in step 3 as the digital object attachment; form the digital object attribute and the digital object attachment into the digital object library through the digital object registration process, and assign a unique identifier; Step 4.2: Set access permissions for digital objects for operator sharing; Step 5: Load and run the data pipeline operator; Step 5.1: Based on the unique identifier of the operator digital object, obtain the metadata structure and data pipeline operator package of the data pipeline operator from the digital object library through the digital object access interface, disassemble the operator package and load the operator executable module, thereby completing the loading of the data pipeline operator; Step 5.2: Call the initialization interface and message processing interface of the loaded data pipeline operator to implement the call of the operator message processing logic.

2. A digital object-based operator distributed management and sharing system implementing the method of claim 1, characterized in that: It includes operator library module, interface component module, operator management module, operator library management interface component module, and operator running engine module; The operator library module is used for storage, identification and parsing, version management, and permission control of data pipeline operators; and serves as the input of the operator management module; The interface component module is used to define the initialization interface, message processing interface and destruction interface that need to be implemented in the development process of the data pipeline operator. The initialization interface is used to initialize variables and resources; the message processing interface is used for the processing of the operator receiving message data; The destruction interface is used to release resources and close the data pipeline; The operator management module is used for cross-system operator library module management and serves as the input of the operator library management interface component and the operator execution engine module; The operator library management interface component module is used for registration, deletion, update, version management, and permission control of data pipeline operators, and interacts with the operator management module; The operator execution engine module is used to load and execute data pipeline operators.

Images