Multi-source heterogeneous data fusion method based on dynamic loading

Abstract

The invention discloses a multi-source heterogeneous data fusion method based on dynamic loading. The multi-source heterogeneous data fusion method is applied to the field of data fusion integration of different data sources and different structures. The method comprises the following steps: (1) constructing a reliable data acquisition layer comprising a main data platform, a unified data integration interface service realized based on message middleware, and an integrated log management module; (2) in a data loading and data processing stage, cleaning and integrating data by adopting dynamic fragmentation and streaming processing, and constructing a multi-level authority control function based on role and data attribute matching and a full-process data transmission and storage encryption and decryption function based on an annotation technology aiming at a data security problem; and (3) constructing a user-defined data display layer. The method mainly solves the problems that data transmission is unreliable, authority control is dispersed and granularity is inconsistent in data processing, and dynamic data cannot be displayed in a user-defined mode, and an effective method is provided for full-process data fusion of a discrete manufacturing enterprise information system.

Description

Technical Field

[0001] This invention relates to a method for fusing multi-source heterogeneous data based on dynamic loading, and in particular to a systematic data fusion method for reliable data transmission, dynamic data processing, and dynamic data customization display in an information system, which is applied to the field of multi-source heterogeneous data integration and fusion. Background Technology

[0002] In the end-to-end data management system of an information system, one of the core objectives is to achieve the integration and application of master data (such as organization, personnel, projects, contracts, etc.) and business processes (including market, project initiation, research plans, production, etc.). Traditional software solutions have four shortcomings: First, data integration often adopts a point-to-point interface approach, and its message transmission mechanism has reliability defects, making it difficult to guarantee that messages are not lost, not duplicated, and transmitted in an orderly manner, thus failing to meet the stable communication requirements between systems for end-to-end data management; second, for dynamic data loading and batch processing, traditional architectures use static sharding and fixed processing windows, which are difficult to adapt to dynamically changing business loads and are prone to forming processing bottlenecks. Meanwhile, rule-based data cleaning methods have limited ability to identify complex anomaly patterns, leading to data quality issues propagating downstream business processes. Regarding data processing security control, enterprise-wide process management involves multi-level data access permissions across departments and positions. The traditional approach of adding separate permission filtering logic to each business module carries risks of fragmented control and inconsistent granularity. Thirdly, traditional dynamic data visualization solutions achieve data display by pre-fixing data table attributes and hard-coding related programs. Given the constantly changing business needs and data elements, this approach requires frequent program modifications and cannot quickly respond to changes. To address these data integration, fusion, and application issues, a reliable and secure systematic data fusion method is needed. Summary of the Invention

[0003] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a method for fusion of multi-source heterogeneous data based on dynamic loading, which performs dynamic loading and fusion processing on the data.

[0004] A hierarchical data scheduling hub is constructed, establishing a reliable data acquisition layer, a dynamic data processing layer, and a dynamic data display layer. A unified data access control method is added during the data integration, fusion, and application process, effectively solving the problem of data integration throughout the entire process of information systems in discrete manufacturing enterprises.

[0005] The technical solution of this invention is: This invention discloses a method for fusing multi-source heterogeneous data based on dynamic loading, comprising: Build a data acquisition layer; Build a data processing layer; Build a custom data presentation layer; The data acquisition layer collects heterogeneous data from multiple sources and sends it to the data processing layer. The data processing layer loads, cleans, filters data permissions, and encrypts / decrypts multi-source heterogeneous data to obtain the data processing results, which are then sent to the data presentation layer. The data presentation layer displays the results of data processing.

[0006] Furthermore, in the above method, a data acquisition layer is constructed as follows: Establish master data standards and an interface list; Establish a master data platform; Configure the master data push interface on the master data platform according to the master data standard and interface list; Build a unified integration interface service based on message middleware, and store the formatted data pushed by the master data push interface into the database; An integrated log interceptor monitors the execution of the unified integration interface service and records whether the operations stored in the database were successful or failed. The master data application service class extracts data from the database based on the user's query parameters and returns it to the user.

[0007] Furthermore, in the above method, the unified integration interface service implemented based on message middleware includes: The producer interface is responsible for receiving data pushed by the master data push interface; defining a corresponding enumeration value for each master data; and formatting the master data and publishing it to the bound queue according to the enumeration value. Listen to the queue and retrieve the master data from the bound queue; The data processing service formats the master data obtained from the listening queue in the bound queue, generating formatted data. The data ingestion service stores formatted data into the database and verifies the data's correct entry into the database through a message verification mechanism. Once the data has been successfully ingested, it is removed from the binding queue.

[0008] Furthermore, in the above method, the data processing layer performs data loading, data cleaning, data permission filtering, and data encryption / decryption processing on multi-source heterogeneous data, specifically as follows: The collected heterogeneous data from multiple sources is loaded and processed in real time using a lightweight stream processing engine that combines a remote dictionary server (Redis) and a database, resulting in loaded data. The loaded data is cleaned by performing default value processing, outlier detection, duplicate data elimination, and standard formatting to obtain cleaned and valid data. Implement access control for the cleaned and valid data, and establish a hierarchical access control model. By using a permission hierarchy model and Mybatis interceptor technology, a mapping relationship between user identity and permission level is established to achieve data permission filtering; Annotation technology is used to achieve end-to-end data transmission, storage, and encryption / decryption.

[0009] Furthermore, in the above method, the permission hierarchy model specifically refers to: Level 1 access is global access, granted to the enterprise's decision-making level and core management personnel, who can query and access all data within the system. Level 2 access is a department-level access permission granted to department heads and management, allowing them to access the department's business-related data sets. Level 3 access is a personal-level access permission granted to ordinary business personnel, who can only access data directly related to their personal job responsibilities.

[0010] Furthermore, in the above method, a custom data presentation layer is constructed, specifically as follows: Create a dynamic configuration business class responsible for recording the configuration relationship between master data and project data, including list fields, table names, aliases, and JOIN conditions; Obtain the configuration relationship between master data and project data from the dynamically configured business class; Based on the configuration relationship between master data and project data, parse and assemble the query statement; Execute the query statement and obtain the query results; Apply access control filtering to the query results to obtain the filtered data; The filtered data will be displayed to the user.

[0011] The advantages of this invention compared to the prior art are: (1) This invention solves the problem of unreliable data transmission between heterogeneous systems. Traditional data transmission methods typically fall into two categories: first, point-to-point interface calls based on the HTTP protocol, which have high integration coupling and are prone to communication interruption due to upgrades or failures of any party; second, intermediate table integration methods that rely on scheduled tasks or polling for data exchange, which suffer from data synchronization delays and data consistency risks. This invention adopts a systematic data transmission scheme based on a master data platform, message middleware, and integrated log management. The master data platform unifies data standards to ensure data standardization and consistency; the message middleware improves reliability through message transmission mechanisms to ensure that messages are not lost, duplicated, or transmitted in an orderly manner; and a unified integrated log management module is designed and implemented to receive integrated data logs in real time, ensuring timely detection and handling of data transmission problems.

[0012] (2) This invention solves the problems of fragmented and inconsistent data access control. In manufacturing enterprises, end-to-end data management involves multiple levels of varying data access permissions. Traditional methods, which add separate permission filtering logic to each module, are prone to access control vulnerabilities and inconsistent control granularity. This invention adopts a centralized access control method to achieve unified design and implementation of data access permissions for different departments and positions.

[0013] (3) This invention solves the problem of dynamic data not being able to be customized for display. Traditional methods usually use hard-coded SQL and manual concatenation, which are not configurable, have poor scalability, and involve a lot of repetitive code; ORM frameworks (such as MyBatis) and XML mapping methods, although supporting some dynamic tags, still require developer intervention, are complex to configure, and cannot be reused. This invention provides a method for customizing dynamic data display, allowing users to adjust data display attributes as needed, without repetitive coding, and responding promptly to changes in business requirements. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the dynamic data transmission implementation in this invention. Figure 2 This is a flowchart illustrating the dynamic data processing procedure in this invention. Figure 3 This is a flowchart illustrating the implementation of dynamic data display in this invention. Figure 4 This is a flowchart of the multi-source heterogeneous data fusion method based on dynamic loading according to the present invention. Detailed Implementation

[0015] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0016] like Figure 4 As shown, this invention discloses a method for fusing multi-source heterogeneous data based on dynamic loading, comprising: Build a data acquisition layer; Build a data processing layer; Build a custom data presentation layer; The data acquisition layer collects heterogeneous data from multiple sources and sends it to the data processing layer. The data processing layer loads, cleans, filters data permissions, and encrypts / decrypts multi-source heterogeneous data to obtain the data processing results, which are then sent to the data presentation layer. The data presentation layer displays the results of data processing.

[0017] Preferably, a data acquisition layer is constructed, specifically as follows: Establish master data standards and an interface list; Establish a master data platform; Configure the master data push interface on the master data platform according to the master data standard and interface list; Build a unified integration interface service based on message middleware, and store the formatted data pushed by the master data push interface into the database; An integrated log interceptor monitors the execution of the unified integration interface service and records whether the operations stored in the database were successful or failed. The master data application service class extracts data from the database based on the user's query parameters and returns it to the user.

[0018] Preferably, the unified integration interface service implemented based on message middleware includes: The producer interface is responsible for receiving data pushed by the master data push interface; defining a corresponding enumeration value for each master data; and formatting the master data and publishing it to the bound queue according to the enumeration value. Listen to the queue and retrieve the master data from the bound queue; The data processing service formats the master data obtained from the listening queue in the bound queue, generating formatted data. The data ingestion service stores formatted data into the database and verifies the data's correct entry into the database through a message verification mechanism. Once the data has been successfully ingested, it is removed from the binding queue.

[0019] Preferably, the data processing layer performs data loading, data cleaning, data permission filtering, and data encryption / decryption processing on multi-source heterogeneous data, specifically as follows: The collected heterogeneous data from multiple sources is loaded and processed in real time using a lightweight stream processing engine that combines a remote dictionary server (Redis) and a database, resulting in loaded data. The loaded data is cleaned by performing default value processing, outlier detection, duplicate data elimination, and standard formatting to obtain cleaned and valid data. Implement access control for the cleaned and valid data, and establish a hierarchical access control model. By using a permission hierarchy model and Mybatis interceptor technology, a mapping relationship between user identity and permission level is established to achieve data permission filtering; Annotation technology is used to achieve end-to-end data transmission, storage, and encryption / decryption.

[0020] Preferably, the access control hierarchy model is as follows: Level 1 access is global access, granted to the enterprise's decision-making level and core management personnel, who can query and access all data within the system. Level 2 access is a department-level access permission granted to department heads and management, allowing them to access the department's business-related data sets. Level 3 access is a personal-level access permission granted to ordinary business personnel, who can only access data directly related to their personal job responsibilities.

[0021] Preferably, a custom data presentation layer is constructed, specifically as follows: Create a dynamic configuration business class responsible for recording the configuration relationship between master data and project data, including list fields, table names, aliases, and JOIN conditions; Obtain the configuration relationship between master data and project data from the dynamically configured business class; Based on the configuration relationship between master data and project data, parse and assemble the query statement; Execute the query statement and obtain the query results; Apply access control filtering to the query results to obtain the filtered data; The filtered data will be displayed to the user.

[0022] Example The following description, in conjunction with the accompanying drawings, provides a more detailed explanation of a multi-source heterogeneous data fusion method based on dynamic loading provided in this application. Specific implementation methods may include (e.g.) Figure 1 , Figure 2 and Figure 3 (as shown) (1) Construct a reliable multi-source heterogeneous data transmission architecture, including a master data platform, a unified integrated business service based on message middleware, and a reliable data acquisition layer including an integrated log management module. The master data platform establishes master data standards, configures and publishes master data push interfaces, and focuses on developing unified integrated services based on message middleware, covering master data message queues for personnel, organizations, projects, and contracts, so as to realize the application of master data in business modules such as market, operation, scientific research, and comprehensive, improve the quality of business data, and enhance the system's traceability and fault diagnosis capabilities through integrated log functions. The systemic solution solves the problems of inconsistent multi-source heterogeneous data standards and unreliable transmission.

[0023] (2) Data loading and processing adopt dynamic sharding technology, with sharding dynamically configured according to business scenarios; a Redis three-level caching architecture is integrated, and a timed refresh strategy is used to achieve stream processing. Data cleaning utilizes a lightweight machine learning framework to classify and identify core features and evaluate data quality. Steps such as missing value handling, deduplication, and format standardization are used to improve the efficiency and quality of data integration. A unified access control method is designed and implemented to centrally manage data access for each module within the system. Based on the mapping relationship between user roles and data attributes, it is ensured that different roles can only access data that meets their permissions. At the same time, data encryption control is added, using AES encryption and decryption, reflection mechanism, annotation recognition, and interceptor technology to realize encryption and decryption operations of data throughout the transmission and storage process.

[0024] (3) Configure dynamic tables for data association, parse and construct the relationship statements between the main table and sub-tables of the project, add table connection conditions, add permission filtering conditions, assemble complete SQL objects, execute and return results, and dynamically render the front end to realize dynamic data customization display.

[0025] Specifically, the technical solution of this application includes: (1) Construct a reliable data acquisition layer that includes a master data platform, a unified integration interface service based on message middleware, and an integrated log management module: (1.1) Establish master data standards and interface list, and configure master data push interfaces on the master data platform; (1.2) Design and develop a unified integration interface service based on message middleware: (1.2.1) Establish an interface service binding queue enumeration class, and define the interface queue name, enumeration value, routing chain and description information; (1.2.2) Implement the producer interface, which is responsible for receiving data pushed from the master data. The producer interface defines the master data access enumeration value and is also responsible for formatting the master data and then publishing it to the bound queue according to the enumeration value. (1.2.3) Listen to the queue of the bound object and obtain the main data message bound to the queue; (1.2.4) Implement the data processing service, format the data obtained from the listening queue into the corresponding entity class, and execute 1.2.5 to save the data; (1.2.5) Implement data entry service, and verify whether the data is correctly stored in the database through a message verification mechanism. If it is correct, proceed to step 1.2.6. (1.2.6) Remove messages that have been successfully added to the database; (1.3) Establish an integrated log interceptor to monitor the execution of the called interface functions and record the success or failure information of message passing; (1.4) Establish the master data application interface: (1.4.1) Establish a general master data application service class and assemble data according to a fixed format; (1.4.2) Construct a front-end API interface based on the application service class to initiate requests to obtain data. It can return filtered data based on query parameters.

[0026] (2) In the data loading and processing stage, dynamic sharding, streaming processing, and machine learning are used to clean and integrate the data. After cleaning, multi-level access control based on role and data attribute matching and annotation-based end-to-end data transmission, storage encryption and decryption are implemented. (2.1) A lightweight stream processing engine combining Redis and a database is used to load and process data in real time, constructing a three-level cache architecture consisting of a first-level Caffeine cache, a second-level Redis cache, and a DM database. The first-level cache can be configured with the area size and expiration time as needed; the second-level cache uses Redis with a hash storage mode and a passive cache cleanup mode. When the first-level cache expires, all nodes are notified to clear both the first and second-level caches. (2.2) A lightweight machine learning framework was used for master data cleaning, different master data standards were established, core features were identified, and the quality and data processing were evaluated: (2.2.1) Calculate the missing rate of key fields, distinguish the missing types, and complete or remove records with a missing rate of key fields exceeding 30%; (2.2.2) Use 3σ or box plots to detect outliers; (2.2.3) Calculate text field similarity using cosine similarity, detect duplicate values ​​using Euclidean or Manhattan distance, feed back to manual processing for integration, remove duplicates based on composite hash values ​​of feature attribute combinations, retain the latest valid records of historical version data, and eliminate duplicate data; (2.2.4) Convert dates to ISO format, standardize units, unify case in text fields and remove leading and trailing spaces; (2.2.5) Tag the cleaned data, record the processing rules, and support backtracking analysis.

[0027] (2.3) Establish a mapping relationship between user roles and data attributes to meet the business needs of users at different levels; (2.3.1) Establish a hierarchical access control model: a. Level 1 access control (global access control): granted to the enterprise's decision-making level and core management personnel, who can query and access all data in the system; b. Level 2 access control (department-level access control): granted to department heads and management, who can access the data set related to their department's business; c. Level 3 access control (personal access control): granted to ordinary business personnel, who can only access data directly related to their personal job responsibilities. (2.3.2) Establish the correspondence between user identity and permission level through the user role configuration module; (2.3.3) Using Mybatis interceptor technology, the SQL query conditions are dynamically extended, and the department and personnel field identifier sequences are injected into the permission level control logic to achieve automatic filtering of the dataset; (2.4) Design and implement full-process data transmission, storage encryption and decryption functions based on annotation technology; (2.4.1) The intercept method intercept is implemented using Java reflection mechanism, and the parameterObject data structure of the interface input parameter object is dynamically obtained from ParameterHandler through reflection; (2.4.2) Customize entity encryption and decryption annotation @SensitiveData, construct a public function AnnotationUtils.findAnnotation to determine whether the input parameter object parameterObject contains the @SensitiveData annotation. If it does, obtain all fields of the class through reflection; (2.4.3) Custom attribute encryption annotation @Encrypt, iterate through the input parameter object obtained through reflection, find the field annotated with @Encrypt, first get its original value, and then call the AES encryption utility class EncryptUtil.encrypt to encrypt the data; (2.4.4) Execute a standard database query to obtain an encrypted result set, and use the custom annotation @Decrypt to identify whether the query results require decryption; if so, call the AES decryption utility class EncryptUtil.decrypt to decrypt the data; and return the complete decrypted data result set.

[0028] (3) Construct a custom data presentation layer: (3.1) Create a dynamic configuration business class DynamicTableConfigService, which is responsible for recording the configuration relationship between master data and project data, including list fields, table name, alias, JOIN conditions, etc.; (3.2) When displaying data, obtain the relationship of the currently enabled configuration from the dynamic configuration business class; parse and assemble it into a complete SQL statement, including the SELECT field list, FROM and LEFT JOIN clauses. When assembling, the main table is directly added, and non-main tables are connected using LEFT JOIN. After parsing, it is encapsulated into a NativeSqlContent object. (3.3) When assembling SQL, add the EmpUtils.updateWhereSql method to implement permission filtering conditions such as department, user, role, etc., form SQL containing permission filtering, execute SQL to obtain data, and finally the front end completes the data display.

[0029] The solution provided in this application takes the process of integrating contracts and product master data into a project system as an example. The contracts and product master data are generated in the contract management system and the product design system, respectively. The present invention integrates and applies these two master data into the project system, including the following steps: (1) Establish contract and product master data standards, configure the master data publishing interface, implement the contract and product master data receiving service interface based on message middleware, and simultaneously implement integrated log management: (1.1) Establish contract and product master data standards, and determine that contract master data attributes include: contract number, name, category, user unit, contract amount, etc., and product master data attributes include product number, name, type, model, drawing number, material number, technical specifications, etc. The master data platform completes the collection of the two master data sets, publishes the master data push interface, the interface protocol is HTTP, and the data format is JSON; (1.2) Define contract and product master data message queues in the project system based on message middleware, implement contract and product master data receiving interfaces, and proceed to 1.3 for data processing if the data is successfully received; (1.3) Monitor the contract and product master data message queues; if there is data, proceed to step 1.4 for actual data processing. (1.4) Implement the contract and product master data processing interface, first format the message data, analyze whether the data is added or modified, execute the corresponding service interface, complete the import of contract and product master data into the database, and send a confirmation message to the message queue. Once the data is successfully imported into the database, remove it from the message queue. (1.5) Establish an integrated log interceptor to monitor the operation of the contract and product master data integration interface and record the success or failure information of data transmission; (1.6) Implement local services for contract and product master data in the project system, and return contract and product master data in the specified format; (1.7) Implement the front-end API interface for contract and product master data, and return the corresponding data when the user selects the contract or product master data.

[0030] (2) In the product data loading and processing stage, data cleaning and integration are achieved through dynamic sharding, streaming processing and machine learning. The utility class EmpUtils.updateWhere is designed to implement centralized project data access control: (2.1) During loading, dynamic sharding is performed based on characteristics such as change frequency and business importance. Combined with real-time processing by the Redis stream processing engine, a write-through plus timed refresh mechanism is adopted to ensure eventual data consistency. (2.2) The data is intelligently cleaned using a lightweight machine learning framework during the processing stage; (2.3) Missing value handling: Calculate the missing rate of key fields, and prioritize filling in those with a missing rate of more than 30% (for numerical types, use random forest regression for prediction, and for categorical types, use the mode to fill in or add an "unknown" category), and remove those that are invalid; (2.4) Outlier detection: The 3σ principle or box plot is used to identify numerical outliers, with a focus on monitoring continuous features such as product weight and size; for classification features, low-frequency labels (such as categories with less than 5 occurrences) are identified through frequency statistics. (2.5) Duplicate data elimination: Text fields are identified as duplicates using cosine similarity (threshold > 0.8), and numerical features are identified as abnormal records using Euclidean distance or Manhattan distance; deduplication is performed based on composite hash values, retaining the latest valid records from historical versions; (2.6) Format standardization: convert dates to ISO format (e.g., 2025-10-23), unify units (e.g., "KG" → "kg"), unify capitalization and remove leading and trailing spaces from text fields; (2.7) Tag the cleaned data, record the processing rules, and support version control and backtracking analysis; (2.8) The data processing access control scenario is that the leader of the Marketing Department (User A) needs to review project data (including contracts and product information), including market review and project approval operations; (2.9) Configure the role "Marketing Department Leader" with level 2 permissions (department-level access permissions) and assign user A to this role; (2.10) User A initiates a market audit query request and automatically configures the access control parameters: userParams=[MARKET_LEAD, MARKET_SUP] (market leader, market supervisor), officeParams=[CHARGE_DEPT] (competent authority); (2.11) Call the access control utility class method to dynamically generate SQL filter conditions based on the permission parameters: WHERE(MARKET_LEAD='User A' OR MARKET_SUP='User A' OR CHARGE_DEPT='Marketing Department 1'), add filter conditions, and process the result set to return only the data that meets the conditions, that is, the project data of User A as the market leader or market supervisor, and the other project data of the market supervisor as "Marketing Department 1"; verify access compliance and block data from other departments (such as data from Marketing Department 2 and R&D Department); (2.12) Use the @Encrypt annotation to mark the product's technical attribute for data encryption; (2.13) When inserting / updating data, the MyBatis interceptor captures the annotation field and triggers encryption; it calls EncryptUtil.encrypt to generate ciphertext through AES encryption; it writes the ciphertext into the database (such as storing technical indicator attributes as AES:3F2A1B...) to complete the storage process, while other attribute data are stored normally as unencrypted fields (such as product name and specifications); (2.14) User A initiates a query, and the system recognizes that he has viewing permissions; the MyBatis interceptor captures the @Decrypt annotated field and triggers decryption; it calls EncryptUtil.decrypt to decrypt and restore the plaintext, returns the complete result, displays the plaintext content of the technical indicators, and keeps the unencrypted fields as they are.

[0031] (3) Display contract and product master data information in the project system through customization: (3.1) Configure the relationship between project data and contract and product master data, including all fields of the project information table VIEW_QLC_INDEX_SCKS, the id and name fields of the product master data tables QLC_MT_PRODUCT and QLC_MT_CONTRACT_INCOME, and the relationship conditions between the two tables, etc. (3.2) When displaying data, obtain the currently enabled configuration relationships from the dynamic configuration business class; parse and assemble them into complete SQL statements, including the SELECT field list, FROM and LEFT JOIN clauses. During assembly, the main table is added directly, and non-main tables are connected using LEFT JOIN. After parsing, it is encapsulated into a NativeSqlContent object. (3.3) Execute the EmpUtils.updateWhereSql method. The login account this time is the Marketing Department Leader role. The data is automatically matched with the information of the competent authority, personnel, etc. to achieve permission filtering.

[0032] The above provides a detailed description of a multi-source heterogeneous data fusion method based on dynamic loading provided by the present invention. For those skilled in the art, there may be changes in the specific implementation process and application scope of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0033] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0034] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A method for fusing multi-source heterogeneous data based on dynamic loading, characterized in that, include: Build a data acquisition layer; Build a data processing layer; Build a custom data presentation layer; The data acquisition layer collects heterogeneous data from multiple sources and sends it to the data processing layer. The data processing layer loads, cleans, filters data permissions, and encrypts / decrypts multi-source heterogeneous data to obtain the data processing results, which are then sent to the data presentation layer. The data presentation layer displays the results of data processing.

2. The method for fusing multi-source heterogeneous data based on dynamic loading according to claim 1, characterized in that, The data acquisition layer is constructed as follows: Establish master data standards and an interface list; Establish a master data platform; Configure the master data push interface on the master data platform according to the master data standard and interface list; Build a unified integration interface service based on message middleware, and store the formatted data pushed by the master data push interface into the database; An integrated log interceptor monitors the execution of the unified integration interface service and records whether the operations stored in the database were successful or failed. The master data application service class extracts data from the database based on the user's query parameters and returns it to the user.

3. The method for fusing multi-source heterogeneous data based on dynamic loading according to claim 2, characterized in that, A unified integration interface service based on message middleware includes: The producer interface is responsible for receiving data pushed by the master data push interface; defining a corresponding enumeration value for each master data; and formatting the master data and publishing it to the bound queue according to the enumeration value. Listen to the queue and retrieve the master data from the bound queue; The data processing service formats the master data obtained from the listening queue in the bound queue, generating formatted data. The data ingestion service stores formatted data into the database and verifies the data's correct entry into the database through a message verification mechanism. Once the data has been successfully ingested, it is removed from the binding queue.

4. The method for fusing multi-source heterogeneous data based on dynamic loading according to claim 1, characterized in that, The data processing layer performs data loading, data cleaning, data permission filtering, and data encryption / decryption on multi-source heterogeneous data, specifically as follows: The collected heterogeneous data from multiple sources is loaded and processed in real time using a lightweight stream processing engine that combines a remote dictionary server (Redis) and a database, resulting in loaded data. The loaded data is cleaned by performing default value processing, outlier detection, duplicate data elimination, and standard formatting to obtain cleaned and valid data. Implement access control for the cleaned and valid data, and establish a hierarchical access control model. By using a permission hierarchy model and Mybatis interceptor technology, a mapping relationship between user identity and permission level is established to achieve data permission filtering; Annotation technology is used to achieve end-to-end data transmission, storage, and encryption / decryption.

5. The method for fusing multi-source heterogeneous data based on dynamic loading according to claim 1, characterized in that, The permission hierarchy model is specifically as follows: Level 1 access is global access, granted to the enterprise's decision-making level and core management personnel, who can query and access all data within the system. Level 2 access is a department-level access permission granted to department heads and management, allowing them to access the department's business-related data sets. Level 3 access is a personal-level access permission granted to ordinary business personnel, who can only access data directly related to their personal job responsibilities.

6. The method for multi-source heterogeneous data fusion based on dynamic loading according to claim 1, characterized in that, Build a custom data presentation layer, specifically as follows: Create a dynamic configuration business class responsible for recording the configuration relationship between master data and project data, including list fields, table names, aliases, and JOIN conditions; Obtain the configuration relationship between master data and project data from the dynamically configured business class; Based on the configuration relationship between master data and project data, parse and assemble the query statement; Execute the query statement and obtain the query results; Apply access control filtering to the query results to obtain the filtered data; The filtered data will be displayed to the user.

Images