Multi-source industrial big data management method and system

Abstract

The application provides a multi-source industrial big data management method and system, comprising the following steps: S1, different types of data sources and related detailed information are configured in a unified management page to realize unified management of multi-source data and access through a unified interface; S2, data of different types of data sources are combined and encapsulated, and a unified access interface is provided externally. The application supports configuration of specific field mapping relationship of multi-source data, strengthens the expandability of the data source, expands the application range of the data source, and increases the use scenarios of the data source without changing the original data source structure.

Description

Technical Field

[0001] This invention relates to the field of big data management technology, specifically to a method and system for managing multi-source industrial big data, and more specifically to a design method and implementation method for managing multi-source industrial big data. Background Technology

[0002] Patent document CN110266533A (application number: 201910527509.4) discloses a big data platform management system, including a big data collection and processing module, an application support service module, a cloud infrastructure module, and a platform management module. It aggregates, models, analyzes, and presents data from various data centers, and provides a comprehensive evaluation of the big data analysis and processing platform, customizing various views to intuitively display relevant indicators for users. This enables interconnection and information exchange among data centers, achieving automated aggregation, modeling, analysis, and presentation of big data, fully showcasing big data information. While this master-slave switching application service guarantee method can support continuous business operations after a switchover, it cannot guarantee that business requests will not be lost during the switchover process.

[0003] Currently, many real-time business applications involve very high transaction volumes and speeds. When a failure occurs, during the application server switchover process, continuous business transaction requests within this time window often cannot be processed. In other words, such dual-machine hot standby or other redundancy methods inevitably lose some business processing during the switchover process, leading to business failures and the application service backend's compensatory handling. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a multi-source industrial big data management method and system.

[0005] A multi-source industrial big data management method provided by the present invention includes:

[0006] Step S1: Configure different types of data sources and related details in a unified management page to achieve unified management of multi-source data and access it through a unified interface;

[0007] Step S2: Combine and encapsulate data from different types of data sources, and provide a unified access interface to the outside world.

[0008] Preferably, the management page includes: a unit information configuration page and a field configuration page;

[0009] Configure different types of data sources, including time-series data and relational data, on the unit information configuration page; and perform CRUD operations on different types of data sources on the unified management page.

[0010] The field configuration page allows for the configuration of preset fields in the data source, enabling more refined configuration.

[0011] Preferably, both time-series data and relational data require configuration of the unit number, data source connection address, username, and password; at the same time, time-series data and relational data have different configuration items depending on the characteristics of each type of data source.

[0012] The unit number indicates that the data source is used under the current unit and is regarded as the identifier of the data source;

[0013] Preferably, the data source is filtered by the unit number through the unit information configuration page.

[0014] Preferably, the field configuration page adopts:

[0015] Step S3: Select and lock the specific data source by unit number;

[0016] Step S4: Configure the mapping relationship of the data source fields according to the locked data source.

[0017] Preferably, in the field mapping relationship: it supports importing Excel files from external sources to generate the mapping relationship, and also supports exporting the configured field mapping relationship to an Excel file, and supports reuse in the client.

[0018] Preferably, in step S2, a data model capable of integrating data from different types of data sources is constructed to achieve multi-source data modeling;

[0019] Multi-source data modeling employs the following configurations: template directory, templates, template attributes, object directory, objects, object attributes, and object data source.

[0020] The object is an instantiation of a template; the template is a collection of common features extracted from a class of objects; the template directory is a collection of templates.

[0021] Preferably, step S2 employs:

[0022] Step S2.1: Configure the template directory to archive templates;

[0023] Step S2.2: Configure the template under the current node according to the nodes in the template directory. After the template is configured, configure the template attributes under the current template.

[0024] Step S2.3: Configure the object directory, select the directory node, and create the object based on the template selected by the directory node;

[0025] Step S2.4: Configure multiple types of data sources;

[0026] Step S2.5: Generate corresponding object properties based on the template properties of the template to which the object is bound. The object property name and object property point type are the same as the template property name and template property field type of the corresponding template property. The object property point name, data source name, and query parameters are configured by the user.

[0027] The template attributes include the template attribute name, the template attribute description, and the template attribute field type.

[0028] Preferably, step S2.4 involves configuring a data source that includes real-time data, relational data, and time-series data.

[0029] When configuring object properties, if the data source bound to the property is a real-time data source, you need to select the tag point in the object property; if the data source is a relational data source, you need to configure the SQL statement in the object property to determine the fields of the relational data table to which the current property is bound; if the data source is a time-series data source, you need to select the fields of the time-series data table to which the current property is bound in the object property.

[0030] A multi-source industrial big data management system provided by the present invention includes:

[0031] Module M1: Enables unified management of multi-source data by configuring different types of data sources and related details in a unified management page and accessing it through a unified interface;

[0032] Module M2: Combines and encapsulates data from different types of data sources and provides a unified access interface to the outside world.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. This invention solves the problem of low configuration efficiency of different types of data sources by extracting the common characteristics of different source data and integrating their configuration interfaces.

[0035] 2. By supporting the configuration of specific field mapping relationships for multi-source data, this invention enhances the scalability of the data source, expands its applicability, and increases its usage scenarios without changing the original data source structure.

[0036] 3. In real industrial production environments, the data sources at the edge are numerous and of different types. However, when actually using this data, what is often needed is not data of a single type or source, but data used in combination. By supporting the binding of multiple different types of industrial data in the same model, this invention solves the problem of the single application scenario and narrow applicability of the data model. Attached Figure Description

[0037] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 Flowchart for configuring multi-source data.

[0039] Figure 2 Flowchart for configuring field mapping relationships.

[0040] Figure 3 Flowchart for modeling multi-source data. Detailed Implementation

[0041] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0042] Example 1

[0043] According to the present invention, a multi-source industrial big data management method is provided, such as... Figures 1 to 3 As shown, it includes:

[0044] Step S1: Configure different types of data sources and related details in a unified management page to achieve unified management of multi-source data and access it through a unified interface;

[0045] Step S2: Combine and encapsulate data from different types of data sources, and provide a unified access interface to the outside world.

[0046] Specifically, the management page includes: a unit information configuration page and a field configuration page;

[0047] Configure different types of data sources, including time-series data and relational data, on the unit information configuration page; and perform CRUD operations on different types of data sources on the unified management page.

[0048] The field configuration page allows for the configuration of preset fields in the data source, enabling more refined configuration.

[0049] Specifically, both time-series and relational data require configuration of the unit number, data source connection address, username, and password; at the same time, time-series and relational data have different configuration items depending on the characteristics of each type of data source.

[0050] The unit number indicates that the data source is used under the current unit and is regarded as the identifier of the data source;

[0051] Specifically, the data source is filtered by the unit number through the unit information configuration page.

[0052] Specifically, the field configuration page adopts the following approach:

[0053] Step S3: Filter and lock the specific data source by unit number;

[0054] Step S4: Configure the mapping relationship of the data source fields according to the locked data source.

[0055] Specifically, in the field mapping relationship: it supports importing Excel files from external sources to generate the mapping relationship, and it also supports exporting the configured field mapping relationship to an Excel file, and it supports reuse in the client.

[0056] Specifically, in step S2, a data model capable of integrating data from different types of data sources is constructed to achieve multi-source data modeling;

[0057] Multi-source data modeling employs the following configurations: template directory, templates, template attributes, object directory, objects, object attributes, and object data source.

[0058] The object is an instantiation of a template; the template is a collection of common features extracted from a class of objects; the template directory is a collection of templates.

[0059] Specifically, step S2 employs the following:

[0060] Step S2.1: Configure the template directory to archive templates;

[0061] Step S2.2: Configure the template under the current node according to the nodes in the template directory. After the template is configured, configure the template attributes under the current template.

[0062] Step S2.3: Configure the object directory, select the directory node, and create the object based on the template selected by the directory node;

[0063] Step S2.4: Configure multiple types of data sources;

[0064] Step S2.5: Generate corresponding object properties based on the template properties of the template to which the object is bound. The object property name and object property point type are the same as the template property name and template property field type of the corresponding template property. The object property point name, data source name, and query parameters are configured by the user.

[0065] The template attributes include the template attribute name, the template attribute description, and the template attribute field type.

[0066] Specifically, step S2.4 involves configuring a data source that includes real-time data, relational data, and time-series data.

[0067] When configuring object properties, if the data source bound to the property is a real-time data source, you need to select the tag point in the object property; if the data source is a relational data source, you need to configure the SQL statement in the object property to determine the fields of the relational data table to which the current property is bound; if the data source is a time-series data source, you need to select the fields of the time-series data table to which the current property is bound in the object property.

[0068] A multi-source industrial big data management system provided by the present invention includes:

[0069] Module M1: Enables unified management of multi-source data by configuring different types of data sources and related details in a unified management page and accessing it through a unified interface;

[0070] Module M2: Combines and encapsulates data from different types of data sources and provides a unified access interface to the outside world.

[0071] Example 2

[0072] Example 2 is a preferred example of Example 1.

[0073] This invention provides a management method for multi-source industrial big data and its specific implementation. By configuring relevant information of multiple different types of data sources on a unified page, users can manage multi-source data in a unified manner on the same platform and access it through a unified interface. At the same time, by constructing a model, the data from these different types of data sources are combined and encapsulated, and a unified access interface is provided to the outside world.

[0074] This invention provides a design system for multi-source industrial big data management, including a multi-source data configuration module and a multi-source data modeling module. The implementation of the module functions mainly relies on a self-developed industrial internet platform, which is implemented based on a self-encapsulated Java framework.

[0075] Multi-source data configuration module:

[0076] The main feature is the ability to configure different types of data sources and their related details in a unified management page.

[0077] The management page includes a unit information configuration page and a field configuration page;

[0078] The unit information configuration page allows users to configure two different types of data sources: time-series data (InfluxDB) and relational data (DB2). CRUD operations are available for both types of data sources. Both time-series and relational data sources require configuration of the unit number, data source connection address, username, and password, but different configuration items exist due to the characteristics of each data source type. The unit number here indicates that the data source is used under the current unit and can be considered as the data source identifier.

[0079] This configuration page also supports filtering data sources by unit number. Generally speaking, different data source types differ significantly in structure and required configuration items, so they have their own configuration pages. However, this module extracts the common characteristics of different source data and merges their configuration interfaces together, improving the configuration efficiency of data sources. This is especially noticeable when there are many industrial data sources in the system with significant differences in type.

[0080] The field configuration page allows users to configure specific fields in the data source, enabling more granular configuration for each data source. On this page, users can first filter and lock onto a specific data source by unit number, and then configure the mapping relationships of fields within that data source (down to the specific data table). Besides the usual manual addition of field mappings, users can also import Excel files to generate them. Furthermore, configured field mappings can be exported as Excel files for reuse in other clients, significantly improving the efficiency of field mapping configuration. The main purpose of configuring field mappings is to ensure that when accessing data source data through a unified data access interface, the field parameters requested by the user do not need to match the actual fields in the data source table. Without changing the original data table structure, by pre-configuring the mapping relationships, users can access configuration information from multiple data sources using a unified data source interface. This feature enhances the scalability of the data source, enabling it to be accessed in more scenarios.

[0081] Multi-source data modeling module:

[0082] The main implementation is a data model that can integrate data from different types of data sources, and provides a unified model access interface.

[0083] Multi-source data modeling requires users to configure seven components: template directory, templates, template attributes, object directory, objects, object attributes, and object data source. A template is an object template that integrates the common characteristics of a class of objects; objects using the same template possess all the attributes contained within the template. An object can be viewed as an instantiation of a template; a template extracts the common characteristics of a class of objects. First, the template directory is configured. The main function of the template directory is to archive templates. After selecting a specific node in the template directory, the template under that node can be configured. After configuring the template, the template attributes under that template can be configured. The configuration items for template attributes include the template attribute name, template attribute description, and template attribute field type. Although an object is an instantiation of a template, it is not the same as the template itself. Therefore, the object directory cannot use the same template directory; objects require their own archiving method. After configuring the object directory, it is necessary to select a directory node and create an object. Because an object is an instantiation of a specific template, a template must be selected when creating an object. Once an object is created, corresponding object properties will be generated based on the template properties of the template to which the object is bound. The object property name and object property point type will be identical to the template property name and template property field type of the corresponding template property. The object property point name, data source name, and query parameters need to be configured manually. Before configuring object properties, the data source must be configured. This supports three data types: real-time data, relational data, and time-series data. When configuring object properties, if the data source bound to the property is a real-time data source, a tag point needs to be selected in the object property point name field. If the data source is a relational data source, an SQL statement needs to be configured in the object property point name field to determine the fields of the relational data table to which the property is bound (configuring an SQL statement instead of directly selecting field names is because SQL statements have better extensibility, effectively adding filtering conditions to the bound data). If the data source is a time-series data source, the fields of the time-series data table to which the property is bound need to be selected in the object property point name field.

[0084] The query parameters serve to add conditions for binding data to object properties, thus narrowing the scope of the bound data and making the data more accurate. To improve the efficiency of object property configuration, in addition to manual configuration, it also supports importing Excel files from external sources to generate the properties. Furthermore, it supports exporting the configured object properties as Excel files for reuse in other clients. These features significantly improve the efficiency of object property configuration.

[0085] Data modeling gives data real-world meaning, making it applicable to various scenarios in the industrial field. Compared to ordinary data modeling, multi-source data modeling supports binding multiple different types of data within a single model. In real industrial production environments, edge data sources are numerous and of varying types. However, when actually using this data, what is often needed is not a single type or source of data, but rather a combination of them. Multi-source data modeling supports this, making the model more meaningful in actual production and broadening its applicability.

[0086] The client-side uses a B / S architecture, with configuration pages implemented using JSP and JavaScript. The server-side uses Java. Communication between the client and server is achieved by the client sending HTTP requests to the server, and both use a unified data exchange format. Configuration data generated by the module is stored in a specified relational database. This relational database is configurable, and its information is stored in a configuration file on the server.

[0087] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0088] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A multi-source industrial big data management method, characterized in that, include: Step S1: Configure different types of data sources and related details in a unified management page to achieve unified management of multi-source data and access it through a unified interface; Step S2: Combine and encapsulate data from different types of data sources and provide a unified access interface; construct a data model that can integrate data from different types of data sources to achieve multi-source data modeling; Step S3: Filter and lock the specific data source by unit number; Step S4: Configure the mapping relationship of the data source fields according to the locked data source; Step S2 employs the following: Step S2.1: Configure the template directory to archive templates; Step S2.2: Configure the template under the current node according to the nodes in the template directory. After the template is configured, configure the template attributes under the current template. Step S2.3: Configure the object directory, select the directory node, and create the object based on the template selected by the directory node; Step S2.4: Configure multiple types of data sources; Step S2.5: Generate corresponding object properties based on the template properties of the template to which the object is bound. The object property name and object property point type are the same as the template property name and template property field type of the corresponding template property. The object property point name, data source name, and query parameters are configured by the user. The object is an instantiation of a template; the template extracts common features of a class of objects; the template directory archives the templates. The template attributes include the template attribute name, the template attribute description, and the template attribute field type.

2. The multi-source industrial big data management method of claim 1, wherein, The management page includes: a unit information configuration page and a field configuration page; Configure different types of data sources, including time-series data and relational data, on the unit information configuration page; and perform CRUD operations on different types of data sources on the unified management page. The field configuration page allows for the configuration of preset fields in the data source, enabling more refined configuration.

3. The multi-source industrial big data management method of claim 2, wherein, Both time-series and relational data require configuration of the unit number, data source connection address, username, and password; furthermore, time-series and relational data have different configuration items depending on the characteristics of each type of data source. The unit number indicates that the data source is used under the current unit and is regarded as the identifier of the data source.

4. The multi-source industrial big data management method of claim 2, wherein, The data source can be filtered by the unit number through the unit information configuration page.

5. The multi-source industrial big data management method of claim 1, wherein, In the mapping relationship of the fields: it supports importing Excel files from external sources to generate the mapping relationship, and also supports exporting the configured field mapping relationship to an Excel file, and supports reuse in the client.

6. The multi-source industrial big data management method of claim 1, wherein, Step S2.4 involves configuring a data source that includes real-time data, relational data, and time-series data. When configuring object properties, if the data source bound to the property is a real-time data source, you need to select a tag point in the object property. When the data source is relational data, you need to configure the SQL statement in the object property to determine the fields of the relational data table to which the current property is bound; When the data source is time-series data, you need to select the field of the time-series data table that the current property is bound to in the object properties.

7. A system for implementing the multi-source industrial big data management method of claim 1, characterized in that, include: Module M1: Enables unified management of multi-source data by configuring different types of data sources and related details in a unified management page and accessing it through a unified interface; Module M2: Combines and encapsulates data from different types of data sources and provides a unified access interface to the outside world.

Images