Heterogeneous resource integration system for data element processing

Abstract

The invention provides a heterogeneous resource integration system for data element processing, and the system comprises a resource unit which fuses data resources, algorithm resources, model resources and application resources, calls corresponding resources according to a resource list, and gives a task result; the resource classification unit comprises an association network structure of association relationships among data resources, algorithm resources, model resources and application resources in a plurality of typical application scenarios; the resource recommendation unit is used for generating a resource list according to the task data and the associated network structure; and the resource application unit receives the task data and outputs a task result. According to the method, different data resources, algorithm resources, model resources and application resources can be independently and flexibly called, loose coupling between the data resources and the algorithm resources and loose coupling between the model resources and the application resources are achieved, the data resources and application scenes are flexibly expanded, and rapid construction and timely response of the application scenes during sudden tasks are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of heterogeneous resource management, and in particular to a heterogeneous resource integration system for data element processing. Background Art

[0002] With the vigorous development of big data technology, various social networking sites have emerged. People usually post their life experiences and personal opinions on social networks, making the value of network data increasingly greater. The types and styles of public opinion production factors have also undergone more changes. The application of public opinion production factors is not only at the data level. In more scenarios, it is necessary to conduct joint analysis with the algorithms, models and applications of artificial intelligence technology to obtain more accurate and detailed data factor analysis results.

[0003] As data analysis capabilities gradually improve and data resource production channels become more flexible and diverse, the requirements for the management and integration capabilities of public opinion production factors are also gradually increasing. From the perspective of big data technology, the integration of heterogeneous data is currently a widely used data integration solution, but most of the integration content on the market is still at the literal data level, based on the business data generated by various business systems, through the data middle platform or ETL tools, to achieve centralized data storage and processing. This requires the ability to synchronously process business data generated by different systems, rather than simply centralized storage.

[0004] In the prior art, existing data integration technologies are usually applied in specific application scenarios. If new business application scenarios are provided externally, it is necessary to build applications separately based on the business scenarios, which will make the framework structure of the entire data integration technology increasingly large and bloated. As business needs increase, the applications on the data platform become strongly coupled, and flexibility is limited. Summary of the invention

[0005] In view of this, the problem to be solved by the present invention is to provide a heterogeneous resource integration system for data element processing, which can independently and flexibly call different data resources, algorithm resources, model resources and application resources, realize loose coupling between various data resources and algorithm resources, model resources and application resources, flexibly expand data resources and application scenarios, and realize rapid construction and timely response of application scenarios in case of sudden tasks.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A heterogeneous resource integration system for data element processing, including resource units: integrating data resources, algorithm resources, model resources and application resources, calling corresponding resources according to a resource list and giving task results;

[0008] Resource classification unit: contains the association network structure of the association relationship between data resources, algorithm resources, model resources and application resources in several typical application scenarios;

[0009] Resource recommendation unit: Generates a resource list based on task data and associated network structure;

[0010] Resource application unit: receives task data and outputs task results.

[0011] Furthermore, the data resources include structured data, and the structured data includes several databases and data calling modules;

[0012] The data calling module includes several calling interfaces, and the calling interfaces are connected to the database through adapters.

[0013] Furthermore, the database includes a DAMO database, an Oracle database, and a MySQL database.

[0014] Further, the data resource includes unstructured data, and the unstructured data includes a data file, a data parsing module and a format conversion module;

[0015] The data parsing module is used to parse the data file to obtain the text segment;

[0016] The format conversion module is used to convert text segments into a unified format and store them.

[0017] Furthermore, the data files include excel files, word files, xml files and json files.

[0018] Further, the algorithm resources include a program unit, an algorithm parameter parsing module, an input and output unification module and an algorithm calling module;

[0019] The program unit includes several algorithm programs that implement different functions;

[0020] The algorithm parameter parsing module is used to determine the parameters required by the algorithm program to generate parameter information;

[0021] The input and output unification module is used to convert the input and output information of the algorithm program into a file in a unified format;

[0022] The algorithm calling module is used to generate control algorithm program calling parameters and run execution calling commands.

[0023] Furthermore, the algorithm program may be developed in a language such as Java, Python, or Scala.

[0024] Furthermore, the model resource includes a model unit, a model parameter parsing module and a model calling module;

[0025] The model unit includes several models that realize different functions;

[0026] The model parameter parsing module is used to generate a unified calling interface based on the execution parameters of the model;

[0027] The model calling module is used to generate the entry address for accessing the model.

[0028] Furthermore, the application resources include an application unit, an application parameter analysis module, an application integration module and an application calling module;

[0029] The application unit includes several applications with analytical functions;

[0030] The application parameter analysis module is used to generate a parameter list;

[0031] The application integration module is used to integrate all applications to generate an integration interface or integration page;

[0032] The application calling module configures the calling information of the data to be analyzed according to the application and parameter list.

[0033] Furthermore, the calling information includes an access application address, a calling method of the application or calling parameters to be analyzed, and an analysis result output.

[0034] The advantages and positive effects of the present invention are:

[0035] By setting resource units, resource classification units, resource recommendation units and resource application units, the resource unit integrates data resources, algorithm resources, model resources and application resources. The data resources include structured data composed of several databases and unstructured data composed of several data files. The algorithm resources include several algorithm programs, the model resources include several trained models, and the application resources include several applications with different functions. The resource recommendation unit generates a resource list based on the set application scenario. The resource list associates structured data, unstructured data, algorithm programs, models and applications, and calls structured data, unstructured data, algorithm programs, models and applications according to the association relationship to cooperate with each other to complete the application scenario tasks and output the task results, thereby improving the flexibility of operation in different application scenarios.

[0036] By setting up resource classification units and resource recommendation units, the resource classification unit combines various typical application scenarios to build horizontal correlations between various resources within the resource unit, forming a heterogeneous resource classification system with a spatial network structure. The resource recommendation unit analyzes the application scenarios (task data) and generates a resource list based on the analysis results and the classification system, thereby improving the flexibility of application scenario construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 It is an overall system diagram of a heterogeneous resource integration system for data element processing of the present invention;

[0039] Figure 2 It is a structured data system diagram of a heterogeneous resource integration system for data element processing of the present invention;

[0040] Figure 3 It is an unstructured data system diagram of a heterogeneous resource integration system for data element processing of the present invention;

[0041] Figure 4 It is an algorithm resource system diagram of a heterogeneous resource integration system for data element processing of the present invention;

[0042] Figure 5 It is a model resource system diagram of a heterogeneous resource integration system for data element processing of the present invention;

[0043] Figure 6 It is an application resource system diagram of a heterogeneous resource integration system for data element processing of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0046] The present invention provides a heterogeneous resource integration system for data element processing, such as Figure 1As shown, it includes resource units, resource classification units, resource recommendation units and resource application units. Resource units include data resources, algorithm resources, model resources and application resources. The resource classification unit obtains a large number of typical application scenarios and determines the association network structure of the association relationship between data resources, algorithm resources, model resources and application resources in the same scenario of the typical application scenario. The association network structures corresponding to various application scenarios together constitute the resource system.

[0047] The operation process of the system is as follows: the resource application unit receives the task data or application scenarios provided by the user, the resource recommendation unit identifies the task data to determine its category, generates a resource list to determine the resource operation status based on the identification result of the task data and the resource system of the same category, and the resource unit calls the corresponding resources based on the resource list, completes data collection, data extraction, data classification, data calculation and other processing, and then feeds back the task results, and outputs the task results through the resource application unit.

[0048] By integrating data resources, algorithm resources, model resources and application resources through resource fusion technology, unified resource management is achieved, and the service capability of heterogeneous resources is provided to the outside world to support the task construction of typical application scenarios.

[0049] Data resources: data information generated by different business scenarios and different business systems in their work; algorithm resources: a number of program units that follow certain steps and rules to solve specific problems through analysis, design, optimization, etc.; model resources: refers to training on large-scale data sets and pre-learning certain tasks in order to complete training faster and generalize better in subsequent tasks; application resources: an application system or a WEB service component that completes a specific business capability.

[0050] like Figure 2 As shown, data resources include structured data and unstructured data. Structured data includes several databases and data calling modules. The data calling module includes several calling interfaces. The calling interface is connected to the database through an adapter, which facilitates the data calling module to randomly call any one or more databases.

[0051] One embodiment of the present application is: the database includes a DAMO database, an Oracle database, a MySQL database, etc. The data calling unit instantiates different databases through a factory mode, configures several interfaces in the factory mode, sets a corresponding relationship between the interface and the adapter, so that the interface can communicate with the database data through the adapter.

[0052] When the data call module accesses a specified database, it determines the configured database category based on the adapter information, obtains the database connection information based on the database category, connects to the database based on the connection information to achieve access, and randomly calls information in different databases through the adapter factory to facilitate runtime calls of algorithm resources, model resources, and application resources.

[0053] like Figure 3 As shown, unstructured data includes data files, data parsing modules and format conversion modules. The data parsing module parses the data files according to the file format, and the format conversion module converts the parsed data files into a unified format and stores them to facilitate subsequent calls by other resources.

[0054] An example of the present application is: data files include excel files, word files, xml files and json files, etc. The format conversion module converts all unstructured data into CSV format files and stores them to facilitate the calling of algorithm resources, model resources and application resources.

[0055] When unstructured data is running, the staff provides the collected or sorted data files. The data parsing module parses the data files into several text segments according to the format of the data files. The format conversion module separates all the text segments with commas to form a CSV format file, which is convenient for calling algorithm resources, model resources and application resources at runtime.

[0056] like Figure 4 As shown, the algorithm resource includes a program unit, an algorithm parameter parsing module, an input and output unification module and an algorithm calling module. The program unit includes several algorithm programs that implement different functions. An embodiment of the present application is that the algorithm program can be developed in Java, Python, or Scala.

[0057] The algorithm parameter parsing module is used to determine the parameters required by the algorithm program to generate parameter information, and to determine the calling information of the parameters required when the algorithm program is running by parsing the algorithm program to generate the parameter information of the algorithm program.

[0058] The input and output unification module is used to convert the input and output information of the algorithm program into a unified format file, and convert the input and output information of the algorithm program into a unified format file through data conversion, so as to facilitate the intercommunication between the algorithm program and external resource data. An embodiment of the present application is that the format file of the input and output information can be in CSV or Json format, which can be conveniently stored in unstructured data or conveniently received by other algorithm resources or model resources.

[0059] The algorithm calling unit is used to generate control algorithm program calling parameters and run execution calling commands, and controls the algorithm program to call parameters from data resources and run by executing the calling commands.

[0060] An operating embodiment of the algorithm resource is: receiving algorithm resource information in a resource list, determining the algorithm program to be called and the algorithm data to be called based on the algorithm resource information, the algorithm calling module generating an execution calling command based on the parameter information of the algorithm program, calling the algorithm data from the data resource based on the execution calling command, the input and output unification module converting the algorithm data into a unified format, controlling the algorithm program to process the algorithm data and outputting the calculation results based on the execution calling command, the input and output unification module converting the calculation results into a unified format, and storing them in unstructured data.

[0061] The algorithm resources are turned into a black box as a whole, and various algorithm resources are abstracted. The most basic command execution algorithm calling method is used to start each algorithm program to execute tasks and submit results. At the same time, the algorithm data required by the algorithm is passed in as parameters. The upper-level application's use of algorithm resources is isolated from the algorithm calling in the background and the development language of the algorithm program, which improves the flexibility of the algorithm program use and realizes loose coupling between each algorithm program and the system.

[0062] like Figure 5 As shown, the model resources include a model unit, a model parameter parsing module, and a model calling module. The model unit includes several models that implement different functions, and commonly used pre-trained models or business models. An example of this application is: the application can be a BERT model based on text processing.

[0063] The model parameter parsing module is used to generate a unified calling interface according to the execution parameters of the model, determine the execution parameters of the model by parsing the model, and convert all the execution parameters of the model into a unified calling interface.

[0064] The model calling module is used to generate the entry address for accessing the model. The entry address of the model is generated according to the model function and the calling interface, so as to call the program according to the model function and control its operation.

[0065] An operating embodiment of the model resource is: receiving model resource information in a resource list, determining the model to be called and the corresponding model data based on the model resource information, the model calling module generates an entry address based on the model, the calling interface and the corresponding model data, connecting the model with the data resource through the entry address, allowing the model to call the data normally and run, and storing the results of the model in unstructured data.

[0066] Model resources form entry addresses based on the model's functions, calling interfaces, and model data, making it easier for the model to provide external services. The use of model resources is isolated from the underlying model framework, making it easier to call application models randomly, improving the flexibility of model use, and achieving loose coupling between models and systems.

[0067] like Figure 6As shown, the application resources include an application unit, an application parameter analysis module, an application integration module and an application call module. The application unit includes several applications that implement different functions, usually including several applications with analysis capabilities.

[0068] The application parameter analysis module is used to generate a parameter list and determine the application parameter list by analyzing the corresponding application parameters.

[0069] The application integration module is used to integrate all applications to generate an integrated interface or integrated page. Based on the capabilities provided by the application system to the outside, the application system is integrated in the form of interface integration or page integration for easy access.

[0070] The application calling module configures the calling information of the data to be analyzed according to the application and the parameter list. The application to be called and the parameters to be analyzed are determined according to the application resource information in the resource list. The application calling module automatically configures the access application address, the calling method of the application or the calling parameter to be analyzed, and the analysis result output according to the application, the parameter list and the parameters to be analyzed. One embodiment of the present application is: the application analysis result is stored in the unstructured data.

[0071] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of this patent.

Claims

1. A heterogeneous resource integration system for data element processing, characterized in that: include, Resource unit: integrates data resources, algorithm resources, model resources and application resources, calls corresponding resources according to the resource list and gives task results; Resource classification unit: contains the association network structure of the association relationship between data resources, algorithm resources, model resources and application resources in several typical application scenarios; Resource recommendation unit: Generates a resource list based on task data and associated network structure; Resource application unit: receives task data and outputs task results.

2. A heterogeneous resource integration system for data element processing according to claim 1, characterized in that: The data resources include structured data, and the structured data includes several databases and data calling modules; The data calling module includes several calling interfaces, and the calling interfaces are connected to the database through adapters.

3. A heterogeneous resource integration system for data element processing according to claim 2, characterized in that: The databases include DAMO database, oracle database and MySQL database.

4. A heterogeneous resource integration system for data element processing according to claim 1, characterized in that: The data resources include unstructured data, and the unstructured data includes data files, data parsing modules and format conversion modules; The data parsing module is used to parse the data file to obtain the text segment; The format conversion module is used to convert text segments into a unified format and store them.

5. A heterogeneous resource integration system for data element processing according to claim 4, characterized in that: The data files include excel files, word files, xml files and json files.

6. A heterogeneous resource integration system for data element processing according to claim 1, characterized in that: The algorithm resources include a program unit, an algorithm parameter parsing module, an input and output unification module and an algorithm calling module; The program unit includes several algorithm programs that implement different functions; The algorithm parameter parsing module is used to determine the parameters required by the algorithm program to generate parameter information; The input and output unification module is used to convert the input and output information of the algorithm program into a file in a unified format; The algorithm calling module is used to generate control algorithm program calling parameters and run execution calling commands.

7. A heterogeneous resource integration system for data element processing according to claim 1, characterized in that: The algorithm program may be developed in Java, Python, or Scala.

8. A heterogeneous resource integration system for data element processing according to claim 1, characterized in that: The model resources include a model unit, a model parameter parsing module and a model calling module; The model unit includes several models that realize different functions; The model parameter parsing module is used to generate a unified calling interface based on the execution parameters of the model; The model calling module is used to generate the entry address for accessing the model.

9. A heterogeneous resource integration system for data element processing according to claim 1, characterized in that: The application resources include an application unit, an application parameter analysis module, an application integration module and an application calling module; The application unit includes several applications with analytical functions; The application parameter analysis module is used to generate a parameter list; The application integration module is used to integrate all applications to generate an integration interface or integration page; The application calling module configures the calling information of the data to be analyzed according to the application and parameter list.

10. A heterogeneous resource integration system for data element processing according to claim 9, characterized in that: The calling information includes the access application address, the calling method of the application or the calling parameter to be analyzed, and the output of the analysis result.

Images