Data resource map generation method and system based on big data mining technology

Generating a global data resource map through big data mining technology solves the problem that enterprises cannot form a full data resource map, and realizes comprehensive management and security control of data resources.

CN119066071AActive Publication Date: 2024-12-03DONGGUAN MENGDA INDAL INVESTMENT
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411200898.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The existing technology cannot form a complete data resource map for enterprises, resulting in high costs of data governance, data security and data traceability.

Method used

Using the data resource map generation method based on big data mining technology, we can form a modular data resource map by obtaining the front-end page and back-end data interface information in the development environment, and connect to the server environment based on identification and resolution technology, collect data tables and data processing Rules to form a global data resource map.

Benefits of technology

It realizes the generation of enterprise data resource maps, solves the problem of incomplete data resource management, and reduces the cost of data governance and security management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066071A_ABST
    Figure CN119066071A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a data resource map generation method and system based on a big data mining technology, and the method comprises the steps: obtaining a component B1 in a front-end page A1 in a development environment, obtaining a rear-end data interface or a data interaction calling information request C1 in operation butt joint with the component B1, and obtaining server log data based on the data interaction calling information request C1; acquiring a front-end page A1, a component B1, a data interaction calling information request C1, a data set, a data table, a data field and a data processing rule which are called by a back-end triggering task and a triggering mechanism under different request conditions of server log data; and associating relationships among the data set, the data table, the data field and the data processing rule to form a first modular data resource map and a second modular data resource map, and finally forming a global data resource map M. The problem that a data resource map of the whole enterprise cannot be formed in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data resource map generation system, method, electronic device and storage medium based on big data mining technology. Background Art

[0002] Data classification and grading is to formulate classification and grading standards suitable for data based on the natural attributes of data assets in combination with national and industry standards, and to classify and grade customer data according to the classification and grading standards to determine the security level of data assets.

[0003] Currently, many enterprises do not systematically manage their own systems, especially cross-system data resources, and are unable to form a comprehensive data resource map for the enterprise. The costs incurred in data governance, data security, and data traceability are relatively high.

[0004] Therefore, there is an urgent need for a method that can generate a comprehensive data resource map of an enterprise based on big data mining technology. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a data resource map generation system, method, electronic device and storage medium based on big data mining technology to solve the problem that the existing technology cannot form a comprehensive data resource map of the enterprise.

[0006] To achieve the above purpose, an embodiment of the present invention provides a data resource map generation method based on big data mining technology, and the data resource map generation method based on big data mining technology specifically includes:

[0007] Obtain component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and obtain server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more;

[0008] Obtaining data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions;

[0009] Associating the relationships among the front-end page A1, the component B1, the data interaction call information request C1, the data set, the data table, the data field and the data processing rules to form a first modular data resource map;

[0010] Obtain the data set, data table, data field and data processing rules called by the back-end trigger task and the trigger mechanism, and associate the relationship between the front-end page A1, component B1, data interaction call information request C1, data set, data table, data field and data processing rules to form a second modular data resource map;

[0011] A global data resource map M is formed based on the first modular data resource map and the second modular map, wherein the number of the first modular data resource map and the second modular map is one or more.

[0012] On the basis of the above technical solution, the present invention can also be improved as follows:

[0013] Further, the obtaining of the data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions, and the obtaining of the data sets, data tables, data fields and data processing rules called by the backend triggering tasks and triggering mechanisms, include:

[0014] Generate a unique A for the front-end page A1, and generate a unique D for each data entry element component D1 in A;

[0015] Generate a unique B for each component B1, generate a unique C for each data interaction call information request C1, generate a unique T based on different request conditions of C, and generate a unique TT based on T.

[0016] Furthermore, the data resource map generation method based on big data mining technology also includes:

[0017] Associate the relationships between A, D, B, C, T, TT, and M;

[0018] Unique identification information is generated for A, D, B, C, T, TT, M and each association relationship between A, D, B, C, T, TT, M, and the identification information is stored.

[0019] Furthermore, the data resource map generation method based on big data mining technology also includes:

[0020] Determine whether A, D, B, C, T, TT, M and the association relationship between A, D, B, C, T, TT, M have been stored. If so, do not store them.

[0021] Furthermore, associating the relationships among data sets, data tables, data fields, and data processing rules to form a first modular data resource map includes:

[0022] Obtain the front data table FT, the front data set FA, the front data field FT-FT, the front data field FA-FT, the data processing rule PR, the back data table BT, the back data field BT-FT, the back data set BA and the back data field BA-FT corresponding to TT;

[0023] Based on the data processing rule decomposition of the data processing rule PR, the front data table FT, the front data set FA, the front data field FT-FT, the front data field FA-FT are logically associated with the post data table BT, the post data field BT-FT, the post data set BA, and the post data field BA-FT to form a first modular data resource map.

[0024] Furthermore, the process of associating the relationships among the data sets, data tables, data fields, and data processing rules to form a first modular data resource map further includes:

[0025] When the previous data table FT and the previous data set FA do not exist, it is determined that the data comes from the data entry element combination D in A.

[0026] Furthermore, the data resource map generation method based on big data mining technology also includes:

[0027] Forming an association relationship between at least one upstream data resource and one downstream data resource for each data table;

[0028] Based on the association relationship between the upstream data resources and downstream data resources corresponding to each data table, they are integrated and linked to finally form a global data resource map M.

[0029] A data resource map generation system based on big data mining technology, comprising:

[0030] An acquisition module is used to acquire component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and acquire server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more;

[0031] Obtaining data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions;

[0032] A map generation module is used to associate the relationship between the front-end page A1, the component B1, the data interaction call information request C1, the data set, the data table, the data field and the data processing rule to form a first modular data resource map;

[0033] Obtain the data set, data table, data field and data processing rules called by the back-end trigger task and trigger mechanism, associate the relationship between the front-end page A1, component B1, data interaction call information request C1, data set, data table, data field and data processing rules, and form a second modular data resource map;

[0034] A global data resource map M is formed based on the first modular data resource map and the second modular map, wherein the number of the first modular data resource map and the second modular map is one or more.

[0035] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0036] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0037] The embodiments of the present invention have the following advantages:

[0038] The data resource map generation method based on big data mining technology in the present invention is based on identification resolution technology, connects to various server environments, collects information including but not limited to server logs, forms an association relationship between at least one upstream data resource and downstream data resource for each data table information, and then forms a comprehensive data resource map for the enterprise, solving the problem that the prior art cannot form a comprehensive data resource map for the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0040] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0041] Figure 1 It is a flow chart of the data resource map generation method based on big data mining technology of the present invention;

[0042] Figure 2 This is a first architecture diagram of a data resource map generation system based on big data mining technology of the present invention;

[0043] Figure 3 This is a second architecture diagram of the data resource map generation system based on big data mining technology of the present invention;

[0044] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided by the present invention.

[0045] The accompanying drawings are denoted as follows:

[0046] Acquisition module 10 , map generation module 20 , storage module 30 , association module 40 , electronic device 50 , processor 501 , memory 502 , bus 503 . DETAILED DESCRIPTION

[0047] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Figure 1 This is a flow chart of an embodiment of a method for generating a data resource map based on big data mining technology of the present invention. Figure 1 As shown, a method for generating a data resource map based on big data mining technology provided by an embodiment of the present invention includes the following steps:

[0049] S101, obtaining component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and obtaining server log data based on the data interaction call information request C1;

[0050] Specifically: the number of components B1 is one or more, and the number of data interaction call information requests C1 is one or more.

[0051] S102, obtaining data sets, data tables, data fields and data processing rules corresponding to server log data under different request conditions;

[0052] S103, associating the relationship between the front-end page A1, the component B1, the data interaction call information request C1, the data set, the data table, the data field and the data processing rule to form a first modular data resource map;

[0053] S104, obtaining the data set, data table, data field and data processing rules called by the backend trigger task and the trigger mechanism, and associating the relationship between the front-end page A1, component B1, data interaction call information request C1, data set, data table, data field and data processing rules to form a second modular data resource map;

[0054] Specifically, collect which fields of which data tables are called by the operation functions of each system front end, and which components are used for display on the front end;

[0055] Collect the data of which front-end components are stored by the front-end operation functions of each system, and what rules are used to transform the stored data based on the front-end data, and in which field of the data table the data is stored;

[0056] How does the collection backend process the field data of the data table to obtain the field data of the new data table and data set?

[0057] Generate a unique A for the front-end page A1, and generate a unique D (D1, D2, D3...) for each data entry element component D1 in A, such as the front-end text box, radio button and other front-end data acquisition elements representing the front-end business attributes, which have the front-end business attributes;

[0058] Generate a unique B for each component B1, generate a unique C for each data interaction call information request C1, generate a unique T based on different request conditions of C, and generate a unique TT based on T;

[0059] Associate the relationships between A, D, B, C, T, TT, and M;

[0060] Unique identification information is generated for A, D, B, C, T, TT and each association relationship between A, D, B, C, T, TT, and the identification information is stored.

[0061] Determine whether the association relationship between A, D, B, C, T, TT and A, D, B, C, T, TT, M has been stored. If so, do not store it.

[0062] Based on the backend program TT, disassemble the data table / processing rules called by the backend program / write the data of the new data table based on the data table data and processing rules (a, add or modify operation) or disassemble the data table / processing rules called by the backend program / and then obtain the data set for displaying (b, query operation) or disassemble the front-end data parameters / processing rules received by the backend program / write the new data table (c, add operation). Based on the data processing words corresponding to the pre-configured data processing syntax of different backend development languages ​​(such as addition, subtraction, multiplication, division, rounding up to two decimal places, etc.); obtain the front data table FT, front data set FA, front data field FT-FT, front data field FA-FT, data processing rule PR, post data table BT, post data field BT-FT, post data set BA and post data field BA-FT corresponding to TT;

[0063] Based on the data processing rule decomposition of the data processing rule PR, the front data table FT, the front data set FA, the front data field FT-FT, the front data field FA-FT are logically associated with the post data table BT, the post data field BT-FT, the post data set BA, and the post data field BA-FT to form a first modular data resource map.

[0064] When the previous data table FT and the previous data set FA do not exist, it is determined that the data comes from the data entry element combination D in A.

[0065] For example:

[0066] a. Obtain server log data, analyze the current front-end business system module, page and operation function name, and mark the operation function name with a unique ID1. Based on the data set called by the operating system, obtain the data table and data processing rules called by the data set;

[0067] If it is a query operation, obtain the component of the data indicator displayed on the front-end page. For the above unique ID1, if the server log data has differences in data sets, data tables, data processing rules, data displays, etc. under different conditions, generate different conditions to generate ID2 and the corresponding business system, system module, page, operation function name, called data table or data set, data processing rules, data indicators on the front-end display page and components;

[0068] If it is an addition, deletion or modification operation, the current front-end operation business system module, page and operation function name are analyzed in the same way, and the operation function name is marked with a unique ID1, then the business system, system module, page and component corresponding to the data processed by the addition, deletion or modification, and the data processing rules are obtained. If the server log data has differences in the corresponding components of the data processed under different conditions, the different conditions generate ID2 and the corresponding business system, system module, page and component, as well as the data processing rules;

[0069] For example, if the data processing is generated by a mechanism such as a task trigger on the backend, a unique ID1 is generated corresponding to the task trigger or mechanism. Similarly, ID2 is generated based on different conditions, as well as the data table and data rules called under the corresponding conditions, as well as the data table stored after processing or the generated data set.

[0070] b. A device based on identification resolution technology generates corresponding identification data from the acquired information and registers a unique identification ID3 corresponding to the above-mentioned storage related information;

[0071] c. For each ID1 and ID2, only a unique ID3 is generated. Based on the monitoring of the log data of each server, the device identifies the above-mentioned new operation or task / mechanism information and condition information, and determines whether it has been stored. If it has been stored, the acquisition in a and the storage in b are not performed.

[0072] S105, forming a global data resource map M based on the first modular data resource map and the second modular map;

[0073] Specifically, the number of the first modular data resource map and the second modular map is one or more.

[0074] Forming an association relationship between at least one upstream data resource and one downstream data resource for each data table;

[0075] Based on the association relationship between the upstream data resources and downstream data resources corresponding to each data table, they are integrated and linked to finally form a global data resource map M.

[0076] The query operation obtains the call of the data source table based on different conditions and the data processing rules, the addition, deletion and modification operation obtains the data displayed on the front-end page component based on different conditions, the call of the data source table and the data processing rules, the task trigger or mechanism calls the data source table based on different conditions and the data processing rules, and generates a production data resource map of business systems, system modules and system operations or task triggers (mechanisms) and data source tables, data processing rules and other elements;

[0077] For example, the generation steps are as follows: the query operation is to obtain the data source table and data processing rules, and the results are displayed in the component. Then the query operation is associated with the data source table and the data processing rules. The display of the results on the front end is associated with the component. The data on the page is added, deleted, and modified. The processing of adding, deleting, and modifying the component display based on the rules is stored in the original data source table or the new data source table. Then the addition, deletion, and modification operations are associated with the component, that is, the data source table obtained by the query operation is associated with the data processing rules, and then associated with the processing rules for the component display data, and associated with the stored data source table or the new data source table. In this way, each step of the operation is associated with data call, data processing rules, and data storage;

[0078] Similarly, based on the task trigger or mechanism, the task trigger or mechanism, the calling data source table and the data processing rules are also associated with the data source table or the new data source table; based on the above, a global production data resource map of the business is generated, and a data flow map is generated at the same time;

[0079] Get a list of system modules, pages and related operation functions for each system;

[0080] Obtain the classification and grading level of each data table and corresponding data field (including structured and unstructured data);

[0081] Obtain the association between each operation function item and each data table and data field in the operation function list through the production data resource map, proceed to obtain all data tables and data fields corresponding to each operation function item in the operation function list, proceed to obtain the classification and grading level of all data tables and data fields corresponding to each operation function item, and take the highest classification and highest grading as the current classification and grading level of each operation function item;

[0082] Determine the authority level of each employee based on the classification and grading of operable data in each business system, and intelligently control the security of system data within the enterprise.

[0083] The data resource map generation method based on big data mining technology obtains component B1 in the front-end page A1 in the development environment, the back-end data interface or data interaction call information request C1 to which component B1 is operated, and obtains server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtains the data set, data table, data field and data processing rules corresponding to the server log data under different request conditions; associates the relationship between the data set, data table, data field and data processing rules to form a first modular data resource map; obtains the data set, data table, data field and data processing rules called by the back-end trigger task and trigger mechanism, associates the relationship between the data set, data table, data field and data processing rules to form a second modular data resource map; forms a global data resource map M based on the first modular data resource map and the second modular map, wherein the number of the first modular data resource map and the second modular map is one or more. The data resource map generation method based on big data mining technology solves the problem that a comprehensive data resource map of an enterprise cannot be formed in the prior art.

[0084] Figure 2-Figure 3 FIG. 1 is a flow chart of an embodiment of a data resource map generation system based on big data mining technology of the present invention; Figure 2-Figure 3 As shown, an embodiment of the present invention provides a data resource map generation system based on big data mining technology, comprising the following steps:

[0085] The acquisition module 10 is used to acquire the component B1 in the front-end page A1 in the development environment, the back-end data interface or the data interaction call information request C1 for the component B1 operation docking, and acquire the server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more;

[0086] Obtaining data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions;

[0087] The acquisition module 10 is also used for:

[0088] Generate a unique A for the front-end page A1, and generate a unique D for each data entry element component D1 in A;

[0089] Generate a unique B for each component B1, generate a unique C for each data interaction call information request C1, generate a unique T based on different request conditions of C, and generate a unique TT based on T.

[0090] A map generation module 20 is used to associate the relationship between the front-end page A1, the component B1, the data interaction call information request C1, the data set, the data table, the data field and the data processing rule to form a first modular data resource map;

[0091] Obtain the data set, data table, data field and data processing rules called by the back-end trigger task and trigger mechanism, associate the relationship between the front-end page A1, component B1, data interaction call information request C1, data set, data table, data field and data processing rules, and form a second modular data resource map;

[0092] A global data resource map M is formed based on the first modular data resource map and the second modular map, wherein the number of the first modular data resource map and the second modular map is one or more.

[0093] The map generation module 20 is also used for:

[0094] Obtain the front data table FT, the front data set FA, the front data field FT-FT, the front data field FA-FT, the data processing rule PR, the back data table BT, the back data field BT-FT, the back data set BA and the back data field BA-FT corresponding to TT;

[0095] Based on the data processing rule decomposition of the data processing rule PR, the front data table FT, the front data set FA, the front data field FT-FT, the front data field FA-FT are logically associated with the post data table BT, the post data field BT-FT, the post data set BA, and the post data field BA-FT to form a first modular data resource map.

[0096] When the previous data table FT and the previous data set FA do not exist, it is determined that the data comes from the data entry element combination D in A.

[0097] Storage module 30, used for associating the relationship between A, D, B, C, T, TT, and M;

[0098] Unique identification information is generated for A, D, B, C, T, TT, M and each association relationship between A, D, B, C, T, TT, M, and the identification information is stored.

[0099] Determine whether A, D, B, C, T, TT, M and the association relationship between A, D, B, C, T, TT, M have been stored. If so, do not store them.

[0100] An association module 40, used to form an association relationship between at least one upstream data resource and one downstream data resource for each data table;

[0101] Based on the association relationship between the upstream data resources and downstream data resources corresponding to each data table, they are integrated and linked to finally form a global data resource map M.

[0102] The present invention discloses a data resource map generation system based on big data mining technology. The system obtains component B1 in a front-end page A1 in a development environment, a back-end data interface or a data interaction call information request C1 to which component B1 is connected by operation through an acquisition module 10, and obtains server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtains data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; associates the relationship between the front-end page A1, component B1, data interaction call information request C1, data sets, data tables, data fields and data processing rules through a map generation module 20 to form a first modular data resource map; obtains data sets, data tables, data fields and data processing rules called by back-end trigger tasks and trigger mechanisms, and associates the relationship between the front-end page A1, component B1, data interaction call information request C1, data sets, data tables, data fields and data processing rules to form a second modular data resource map; and forms a global data resource map M based on the first modular data resource map and the second modular map, wherein the number of the first modular data resource map and the second modular map is one or more. The data resource map generation system based on big data mining technology solves the problem that the existing technology cannot form a comprehensive data resource map for the enterprise.

[0103] Figure 4 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 4 As shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503;

[0104] The processor 501 and the memory 502 communicate with each other via the bus 503;

[0105] The processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the component B1 in the front-end page A1 in the development environment, the back-end data interface or the data interaction call information request C1 to which the component B1 is connected, and obtaining the server log data based on the data interaction call information request C1, wherein the number of the components B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtaining the data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; combining the front-end page A1, the component B1, and the data interaction call information request C1, data sets, data tables, data fields and data processing rules are associated to form a first modular data resource map; the data sets, data tables, data fields and data processing rules called by the back-end trigger tasks and trigger mechanisms are obtained, and the relationships between the front-end page A1, component B1, data interaction call information request C1, data sets, data tables, data fields and data processing rules are associated to form a second modular data resource map; based on the first modular data resource map and the second modular map, a global data resource map M is formed, wherein the number of the first modular data resource map and the second modular map is one or more.

[0106] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining component B1 in a front-end page A1 in a development environment, a back-end data interface or a data interaction call information request C1 to which component B1 is connected, and obtaining server log data based on the data interaction call information request C1, wherein the number of the components B1 is one or more, and the number of the data interaction call information request C1 is one or more; obtaining data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; converting the front-end page A1, component B1, and data fields into a data processing rule; obtaining the ... obtaining the data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; obtaining the data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; obtaining the data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; obtaining the data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; obtaining the data sets, data tables, data fields and data processing rules corresponding to B1, data interaction call information request C1, data set, data table, data field and data processing rules are associated to form a first modular data resource map; the data set, data table, data field and data processing rules called by the back-end trigger task and trigger mechanism are obtained, and the relationship between the front-end page A1, component B1, data interaction call information request C1, data set, data table, data field and data processing rules is associated to form a second modular data resource map; based on the first modular data resource map and the second modular map, a global data resource map M is formed, wherein the number of the first modular data resource map and the second modular map is one or more.

[0107] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0110] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A data resource map generation method based on big data mining technology, characterized in that: The data resource map generation method based on big data mining technology specifically includes: Obtain component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and obtain server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; Obtaining data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; Associating the relationships among the front-end page A1, the component B1, the data interaction call information request C1, the data set, the data table, the data field and the data processing rules to form a first modular data resource map; Obtain the data set, data table, data field and data processing rules called by the back-end trigger task and trigger mechanism, associate the relationship between the front-end page A1, component B1, data interaction call information request C1, data set, data table, data field and data processing rules, and form a second modular data resource map; A global data resource map M is formed based on the first modular data resource map and the second modular map, wherein the number of the first modular data resource map and the second modular map is one or more.

2. The method for generating a data resource map based on big data mining technology according to claim 1 is characterized in that: The obtaining of the data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions, and the obtaining of the data sets, data tables, data fields and data processing rules called by the backend trigger tasks and trigger mechanisms, include: Generate a unique A for the front-end page A1, and generate a unique D for each data entry element component D1 in A; Generate a unique B for each component B1, generate a unique C for each data interaction call information request C1, generate a unique T based on different request conditions of C, and generate a unique TT based on T.

3. The method for generating a data resource map based on big data mining technology according to claim 1, characterized in that: The data resource map generation method based on big data mining technology also includes: Associate the relationships between A, D, B, C, T, TT, and M; Unique identification information is generated for A, D, B, C, T, TT, M and each association relationship between A, D, B, C, T, TT, M, and the identification information is stored.

4. The method for generating a data resource map based on big data mining technology according to claim 3 is characterized in that: The data resource map generation method based on big data mining technology also includes: Determine whether A, D, B, C, T, TT, M and the association relationship between A, D, B, C, T, TT, M have been stored. If so, do not store them.

5. The method for generating a data resource map based on big data mining technology according to claim 1, characterized in that: The step of associating the relationships among data sets, data tables, data fields, and data processing rules to form a first modular data resource map includes: Obtain the front data table FT, the front data set FA, the front data field FT-FT, the front data field FA-FT, the data processing rule PR, the back data table BT, the back data field BT-FT, the back data set BA and the back data field BA-FT corresponding to TT; Based on the data processing rule decomposition of the data processing rule PR, the front data table FT, the front data set FA, the front data field FT-FT, the front data field FA-FT are logically associated with the post data table BT, the post data field BT-FT, the post data set BA, and the post data field BA-FT to form a first modular data resource map.

6. The method for generating a data resource map based on big data mining technology according to claim 5 is characterized in that: The step of associating the relationships among the data sets, data tables, data fields, and data processing rules to form a first modular data resource map further includes: When the previous data table FT and the previous data set FA do not exist, it is determined that the data comes from the data entry element combination D in A.

7. The method for generating a data resource map based on big data mining technology according to claim 1, characterized in that: The data resource map generation method based on big data mining technology also includes: Forming an association relationship between at least one upstream data resource and one downstream data resource for each data table; Based on the association relationship between the upstream data resources and downstream data resources corresponding to each data table, they are integrated and linked to finally form a global data resource map M.

8. A data resource map generation system based on big data mining technology, characterized in that: include: An acquisition module is used to acquire component B1 in the front-end page A1 in the development environment, a back-end data interface or a data interaction call information request C1 for the component B1 operation docking, and acquire server log data based on the data interaction call information request C1, wherein the number of the component B1 is one or more, and the number of the data interaction call information request C1 is one or more; Obtaining data sets, data tables, data fields and data processing rules corresponding to the server log data under different request conditions; A map generation module is used to associate the relationship between the front-end page A1, the component B1, the data interaction call information request C1, the data set, the data table, the data field and the data processing rule to form a first modular data resource map; Obtain the data set, data table, data field and data processing rules called by the back-end trigger task and trigger mechanism, associate the relationship between the front-end page A1, component B1, data interaction call information request C1, data set, data table, data field and data processing rules, and form a second modular data resource map; A global data resource map M is formed based on the first modular data resource map and the second modular map, wherein the number of the first modular data resource map and the second modular map is one or more.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Full-graph data processing method and device, equipment and storage medium

    CN115455032A

  • Application computing resource distribution statistical method and device of super computer

    CN116089065A

  • Business data processing method and device, computer equipment and storage medium

    CN116737753A

  • Data aggregation and performance assessment

    WO2018002664A1