A digital foundation system for heterogeneous data source big data in a campus

Abstract

The digital base system for the heterogeneous data source big data in the campus provided by the application comprises a base database, a front-end framework and a flow / batch integrated processing collection unit. The application utilizes the front-end framework, and realizes the processing and display of various campus data based on the dimension analysis calculation capability of the B / S architecture. The system can provide a unified collection interface in a terminal device, realize the graphical processing of the campus data, and realize the collection and exchange of the flow batch integrated big data of various structured, semi-structured and unstructured big data collection and access interconnection; the flow batch integrated big data collection and exchange can realize the wisdom campus digital base of the campus global data center, realize the data collection, cleaning, processing, gathering, calculation of various structured, unstructured and semi-structured big data in and outside the campus, realize the standardized and thematized output of the data, and simultaneously utilize the link channel between the two to ensure the stability and safety of the data flow in the collection and exchange process.

Description

Technical Field

[0001] This application relates to the field of campus data management technology, and more specifically to a digital infrastructure system for heterogeneous data sources and big data on campus. Background Technology

[0002] Existing campus data is scattered across various independent systems, requiring individual connection devices or data processing servers for integration and processing. Current microservice architectures merely divide a single application, breaking down different functionalities into a set of small services that coordinate and cooperate with each other to achieve software functionality.

[0003] The existing campus data system and microservice architecture operate independently, with loose coordination between processing units and low interaction efficiency. This results in a large amount of duplicate and redundant information in the network between units, affecting interaction efficiency and making it impossible to guarantee the real-time performance and accuracy of data processing. At the same time, it is also difficult to accurately update various relevant information in the campus internal data system in a timely manner. Summary of the Invention

[0004] This application addresses the shortcomings of existing technologies by providing a digital foundation system for heterogeneous data sources on campus. The application utilizes a B / S architecture for dimensional analysis and computation, along with a lakeware database architecture and a combined stream / batch processing and acquisition unit. This enables dynamic processing of heterogeneous campus data during the integrated stream / batch processing process and allows for flexible graphical visualization of the processing status of various types of unstructured and semi-structured data from both inside and outside the campus. The specific technical solution adopted in this application is as follows.

[0005] First, to achieve the above objectives, a digital foundation system for heterogeneous data sources in campuses is proposed, comprising: a foundation database, which includes a batch-stream integrated lake warehouse constructed from a source data layer, a standard data layer, and an application data layer. The source data layer connects to business source microservice acquisition units to collect and store dynamic business data. Each dynamic business data item points to a corresponding campus data item in the batch-stream integrated lake warehouse through a corresponding link channel. The standard data layer connects to a cleaning and acquisition unit in the batch-stream integrated lake warehouse to store the processed data. The application data layer connects to several thematic databases, each storing various campus data items collected and confirmed by the business source microservice acquisition units, as well as thematic data obtained through secondary calculation and solidification. The link channels for campus data items also authorize the collection of corresponding dynamic business data via HTTP(S) / TCP / UDP protocols, ensuring data encryption security. A front-end framework is also provided. A B / S architecture is adopted to display various campus data items. The front-end framework communicates and interacts with the theme database via HTTP and WebSocket protocols. The front-end framework also includes a dynamic preview unit, which extracts and displays the results of each campus data item after collection, cleaning, and transformation. It also retrieves corresponding dynamic business data from the theme database through a link channel, collects, processes, and displays the corresponding dynamic business data according to the display mode of the corresponding campus data item in the front-end framework. The dynamic business data is output as the corresponding display content after processing in an independent layer. The stream / batch integrated processing and collection unit adopts a microservice architecture and connects to the open interfaces of various existing system terminals on campus through containerization technology. The stream / batch integrated processing and collection unit is connected to the source data layer to obtain heterogeneous data from various existing system terminals on campus and updates the corresponding campus data items in the base database through business data flow.

[0006] Optionally, in any of the above-described digital foundation systems for heterogeneous data sources in a campus, the stream / batch integrated processing and acquisition unit is further provided with a data flow control unit. The data flow control unit is used to extract various campus data items for collection, cleaning, management, and transformation, and to retrieve the corresponding dynamic data from the link channel of the business source microservice acquisition unit, and process the corresponding dynamic data according to the hierarchical aggregation mode in the foundation database.

[0007] Optionally, the digital foundation system for heterogeneous data sources in a campus, as described above, further includes a data publishing gateway. The data publishing gateway uses a visual construction device to call an SQL database for data weaving and publishes corresponding microservices to provide thematic data as support for thematic databases.

[0008] Optionally, in the digital foundation system for heterogeneous data sources in a campus, as described above, the flow control unit is specifically constructed based on the collection process of campus data items and the canvas of corresponding independent components. It is used to set corresponding data loading, cleaning, and writing requirements for the parameters of each component in the collection process of campus data items, and to dynamically iterate and update the campus data items in real time or periodically according to the update requirements of each parameter.

[0009] Optionally, in the digital infrastructure system for heterogeneous data sources in a campus, as described above, the microservice architecture of the integrated stream / batch processing acquisition unit includes two data processing modes: 1) Stream processing mode: The integrated stream / batch processing acquisition unit collects data from each data acquisition device in real time based on the archived logs of the infrastructure database (CDC mode) and transmits the collected data to the target storage unit; the characteristic of this mode is that it can obtain changes in source data with low latency; 2) Batch processing mode: The integrated stream / batch processing acquisition collects data from each data acquisition device based on a timed scheduling mechanism, and loads data accordingly through timer triggers and writes the collected data to the target storage unit.

[0010] Optionally, in any of the above-described digital infrastructure systems for heterogeneous data sources in a campus, the business source microservice acquisition unit is used to connect to all types of existing dynamic business data within the campus, and to acquire and connect data in any of the following ways: a) Establishing a connection with the source end of the existing system within the campus through the JDBC data source connection communication protocol, and accessing dynamic data within the authorized scope; the data loading device acquires data information and writes it into the message pipeline; the data writing device consumes data from the message pipeline and writes it into the target storage unit; b) Based on message data acquisition technology, connecting to the message pipeline through the Kafka message pipeline connector, acquiring the data stored in the Topic, parsing the data in the message format through the data writing device, and writing the parsed data into the target storage unit; c) Based on HTTP(s) / TCP / UDP protocol data acquisition technology, establishing a communication connection with the terminal device through authorized IP and port according to the HTTP(s) / TCP / UDP protocol, acquiring the data collected by the terminal device, parsing the data format through the writing device, and writing the parsed content into the target storage unit.

[0011] Optionally, as described above, the digital foundation system for heterogeneous data sources in the campus adopts a more secure, stable, and controllable full encryption algorithm for the loading and storage of data in each component of the campus data item collection process, including: SM2 (256), SM4 (128), RSA (2048, 4096), and AES (128, 192, 256), and the encryption key can be centrally managed by KMS key.

[0012] Optionally, in any of the above-described digital infrastructure systems for heterogeneous data sources on campus, the public key and / or private key in the fully encrypted algorithm are data strings generated by a hash algorithm based on the feature values ​​of the system terminal or the address encoding information of the corresponding interface.

[0013] Optionally, in any of the above-described digital infrastructure systems for heterogeneous data sources in a campus, the data acquisition device includes a computing unit. This computing unit, in response to user operations based on the authorization permissions of the terminal device, performs the following steps: retrieving the terminal device's authorization authentication information and sending corresponding authorization verification information to the data source server; triggering the data source server to perform verification calculations on the authorization verification information, and after confirming that the verification calculation result meets the data source connectivity requirements, returning the verification result to the microservice unit via the HTTP protocol; triggering the microservice acquisition unit to respond to the terminal device's authorization confirmation verification result for campus data, parsing the corresponding campus data's metadata description information according to the user request parameters, and triggering the data source server to call SQL statements based on the database location information corresponding to the metadata, loading the source data according to encrypted instructions, and updating the corresponding campus data item in the infrastructure database to the dynamic data pointed to by its link channel. Beneficial effects

[0014] The digital infrastructure system for heterogeneous data sources in campuses provided in this application includes a foundation database, a front-end framework, and a stream / batch integrated processing and acquisition unit. The foundation database comprises a batch-stream integrated lakehouse built from a source data layer, a standard data layer, and an application data layer. This application utilizes the front-end framework, based on the dimensional analysis and computing capabilities of a B / S architecture, to process and display underlying data for various campus data application areas. This system provides a unified acquisition interface in terminal devices, enabling graphical processing of campus data and the collection and exchange of various structured, semi-structured, and unstructured big data in a stream-batch integrated manner. This application can realize a smart campus digital infrastructure for the entire campus data center through stream-batch integrated big data collection and exchange, enabling data collection, cleaning, processing, aggregation, and computation for various structured, unstructured, and semi-structured big data both inside and outside the campus, achieving standardized and thematic output of data, while simultaneously ensuring the stability and security of data flow during the collection and exchange process through the link channel between the two.

[0015] This application utilizes a B / S architecture to interface with various heterogeneous mainstream databases, providing integrated batch and stream big data acquisition and exchange. It enables integrated access and interconnection of big data acquisition, and connects to dedicated network platforms to handle various structured, semi-structured, and unstructured data. The authorized campus data in this application can interact with the Lakewareg database through the data source layer database. Each heterogeneous data source terminal system captures open data interfaces and uses them as corresponding microservice acquisition units, achieving integrated batch and stream data acquisition and processing based on a microservice architecture. This application also utilizes a B / S architecture to process and output various campus data. Therefore, this system can provide a unified visual presentation on terminal devices via the Web, using a front-end framework to achieve graphical processing and corresponding analysis of campus data.

[0016] Other features and advantages of this application will be set forth in the following description and in part as described herein, or may be learned by practicing this application. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the overall structure of the digital infrastructure system for heterogeneous data sources on campus provided in this application; Figure 2 This is a schematic diagram of the interactive steps of the campus big data digital infrastructure system provided in this application; Figure 3This is a schematic diagram of the campus data interaction and display method based on the B / S architecture used in this application. Implementation

[0018] To make the objectives and technical solutions of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0019] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0020] The meaning of "and / or" as used in this application includes situations where each exists alone or both exist simultaneously.

[0021] The term "connection" as used in this application can mean a direct connection between components or an indirect connection between components through other components.

[0022] Figure 1 According to this application, a digital foundation system for heterogeneous data sources and big data in a campus includes the following modules to achieve standardized and thematic output of data, supporting upper-layer application modeling and intelligent applications: The underlying database comprises a batch-stream integrated lake warehouse built from a source data layer, a standard data layer, and an application data layer. The source data layer connects to business source microservice acquisition units to collect and store dynamic business data. Each dynamic business data item points to a corresponding campus data item in the batch-stream integrated lake warehouse through a corresponding link channel. The standard data layer connects to the cleaning acquisition unit in the batch-stream integrated lake warehouse to store the processed data. The application data layer connects to several thematic databases, each storing various campus data items collected and confirmed by the business source microservice acquisition units, as well as thematic data obtained through secondary calculation and solidification. The link channels for the campus data items also authorize the collection of corresponding dynamic business data via HTTP(S) / TCP / UDP protocols, ensuring data encryption and security. The base database stores various campus data collected and confirmed by the microservice unit. Each campus data item is also matched with a data flow link channel, which points to dynamic data awaiting authorization in the data source layer database. The data source layer database, connected between the standard database and the microservice collection unit, stores the collected dynamic business data. Each dynamic data item points to the corresponding campus data item in the data lake through a link channel. The data in the data source layer is cleaned and processed by the data cleaning device and then collected to the corresponding data item in the standard database. Finally, through indicators and big data computing units, corresponding thematic application data is formed, and the various campus data categories are extracted, data reduction is performed, and data service interfaces are provided for the independent layer output and display of dynamic data. The microservice acquisition unit connects to the open interfaces of various terminal systems and links to the data source database. It is used to acquire business data streams from heterogeneous data source terminal systems and update the corresponding campus data in the database. It interfaces with distributed file systems such as HDFS, Hive, and Spark; relational database types such as GaussDB, Oracle, MySQL, PostgreSQL, SQL Server, DB2, and Sybase; offline text data sources such as CSV, EXCEL, and JSON formats; and other data sources such as Elasticsearch, Kylin, and Hadoop. It also connects to the source database to acquire business data streams from heterogeneous data source terminal systems and update the corresponding campus data in the database. The above system also includes: The base database governance unit provides graphical drag-and-drop data cleaning and transformation (ETL) operations based on the extraction, transformation, and loading processes received by the microservice collection unit, enabling functions such as data connection, data merging, data filtering, data deduplication, and data transformation. The front-end framework adopts a B / S architecture to handle data interaction and display various campus data. It communicates with the database via the WebSocket protocol. The dynamic preview unit set in the front-end framework is used to display various campus data items. It communicates with the theme database via HTTP and WebSocket protocols, and retrieves the corresponding dynamic data from the main database according to the link channel. The dynamic data is processed according to the display mode of campus data in the front-end framework, and the display content corresponding to the processed dynamic data is output as an independent layer. In specific display, the front-end framework can extract the results of each campus data item after collection, cleaning, and transformation through its dynamic preview unit for display, and retrieve the corresponding business dynamic data from the theme database through the link channel. The corresponding business dynamic data is collected, processed, and displayed according to the display mode of the corresponding campus data item in the front-end framework. The business dynamic data is output as the corresponding display content after processing as an independent layer. The integrated stream / batch processing and acquisition unit adopts a microservice architecture and uses containerization technology to connect to the open interfaces of various existing system terminals on campus. The integrated stream / batch processing and acquisition unit can be referenced from... Figure 2 The method connects to the source data layer to obtain heterogeneous data from existing system terminals on campus, and updates the corresponding campus data items in the base database through business data flow.

[0023] The aforementioned system requires data architecture isolation of the data to be managed during the processing of campus data. Therefore, this application preferably sets the base database to consist of three parts: a source layer, a standard layer, and an application layer (topic database). The source layer stores various campus data collected and confirmed by the microservice unit. Each campus data is also matched with a data flow link channel, which points to dynamic data to be authorized for collection in the data source layer database (including structured, semi-structured, and unstructured data, and from the data source location, including business database and campus IoT device gateway). The standard layer is connected to the governance acquisition unit of the base database and is used to store the data after governance. The application layer connects to various thematic databases, storing campus data collected and confirmed by the microservice collection unit, as well as thematic data solidified through secondary calculations.

[0024] Therefore, this application can extract the corresponding dynamic data from the theme database according to the link channel through the dynamic display unit designed in the front-end application, process the dynamic data required for collection and display according to the location of campus data in the data business architecture, and output the display content corresponding to the processed dynamic data as an independent layer.

[0025] In the aforementioned system, the integrated stream / batch processing acquisition unit can generally also be equipped with a data flow control unit. The data flow control unit is specifically constructed based on the acquisition process of campus data items and the canvas of the corresponding independent components. It is used to set corresponding data loading, cleaning, and writing requirements for the parameters of each component in the acquisition process of campus data items, and to extract each campus data item in real time or periodically for acquisition, cleaning, management, and transformation according to the update requirements of each parameter. It also retrieves the corresponding dynamic data from the link channel of the business source microservice acquisition unit, processes the corresponding dynamic data according to the hierarchical aggregation mode in the base database, so as to cover the corresponding campus data items and perform dynamic iterative updates. The microservice architecture of this integrated stream / batch processing acquisition unit includes two data processing modes: In stream processing mode: The stream / batch integrated processing acquisition unit collects data from each data acquisition device in real time based on the archived logs of the base database, and transmits the collected data to the target storage unit; In batch processing mode: The integrated stream / batch processing acquires data from each data acquisition device based on a timed scheduling mechanism, and loads the data accordingly through timer triggers and writes the acquired data into the target storage unit.

[0026] The digital infrastructure system may also include a data publishing gateway. This gateway, through a visual construction device, can invoke an SQL database for data weaving and publish corresponding microservices to support thematic databases. The data publishing gateway, based on a dynamically published data sharing unit, consists of an underlying service unit, a service layer gateway control, and an application layer authentication. The application layer is invoked by web terminals or campus smart terminal IoT devices after authorization. The authorization code is dynamically generated by the application unit. The calling end sends an authentication command, and only after successful authentication can it obtain data sharing services or establish a data writing pipeline.

[0027] Taking the data collection and uploading process of multiple terminal devices in the above system as an example: Scenario 1: When a network device on campus receives a device status signal, it reads the device's SNMP protocol information through a microservice unit called within the base and records the corresponding status data to form a dynamic data record. Then, it triggers a batch stream read / write device to send a request to enter the dynamic data record. The Skin terminal calls an SQL statement to enter the corresponding dynamic data into the target database based on the database location information.

[0028] Scenario 2: The campus energy gateway device receives information sent by the energy gateway device via the HTTP protocol through the HTTP interface called inside the base, forming a dynamic data record. Then, the touch writing device sends an input request for the dynamic data record; the HTTP server calls SQL statements to input the corresponding dynamic data into the target database based on the specified database location information.

[0029] Scenario 3: Campus business database data. Through the JDBC protocol, the data acquisition unit connects to the source server to obtain data within the authorized scope. Through graphical capabilities, the extraction, transformation, and loading processes are designed and defined to form a dynamic data record. Then, the batch stream read / write device is triggered to send an input request for the dynamic data record. The Skin terminal calls SQL statements to input the corresponding dynamic data into the target database based on the database location information, ensuring the stability and security of data acquisition and exchange.

[0030] In addition, the business source microservice collection unit is used to connect with all types of existing dynamic business data on campus, and can be collected and connected through any of the following methods: Based on message data acquisition technology, the message pipeline is connected through the Kafka message pipeline connector to obtain the data stored in the Topic. The data writing device parses the data in the message format and writes the parsed data into the target storage unit. Based on HTTP(s) / TCP / UDP protocol data acquisition technology, a communication connection is established with the terminal device through authorized IP and port according to the HTTP(s) / TCP / UDP protocol to obtain the data collected by the terminal device. The data format is parsed by the writing device and the parsed content is written to the target storage unit.

[0031] For data sources with authorized access, this application can specifically implement database updates through the following steps: The system retrieves the authorization and authentication information of the terminal device and sends the authorization verification information to the data source server. It then triggers the data source to perform verification calculations on the authorization verification information. After confirming that the verification calculation results meet the data source connectivity requirements, the system returns the verification result instruction to the microservice unit via the HTTP protocol. The microservice acquisition unit responds to the terminal device's authorization confirmation operation for campus data, parses the corresponding campus data metadata description information according to the request parameters, and enables the data source server to call SQL statements according to the database location information, load the source data according to the encrypted instructions, and update the corresponding campus data in the target database to the dynamic data pointed to by its link channel.

[0032] Meanwhile, this application can provide a graphical drag-and-drop data cleaning and transformation (ETL) operation, and supports common data connection, data merging, data filtering, data deduplication, data transformation and other components; this application can also pre-set common transformation rules for the data transformation component, such as removing nulls, converting uppercase and lowercase letters, retaining decimal places, truncating characters, date conversion, etc.

[0033] For highly sensitive data, this application employs a more secure, stable, and controllable full encryption strategy for data loading and storage. The supported algorithms include SM2 (256), SM4 (128), RSA (2048, 4096), and AES (128, 192, 256). KMS key centralized management is also supported for encryption keys. When performing full encryption operations, the public and / or private keys used by the aforementioned algorithms can be set to data strings generated by hash algorithms based on characteristic values ​​such as the system terminal's port number, device identification code, port address, or the address encoding information of the corresponding interface.

[0034] For microservice units with authorized confirmation permissions, they can update the database through the following steps: Retrieve the authorization code of this terminal device and send the authorization verification information to the web server via the WebSocket protocol; The authorization verification information triggers the web server to perform verification operations. The verification operation result is calculated using any one or more hash algorithms among MD5, SHA1, SHA256, SHA384, and SHA512. After confirming that the verification result corresponds to the pre-stored authorization information, the verification pass instruction is returned to the microservice unit through the WebSocket protocol. Therefore, based on the instruction-triggered microservice unit accepting the terminal device's authorization confirmation operation for campus data, sending the campus data confirmation request to the Web server via the WebSocket protocol, and triggering the Web server to parse the database location information of the corresponding campus data according to the confirmation request, so that the Web server can call SQL statements to update the corresponding campus data in the main database to the dynamic data pointed to by its connection channel according to the database location information.

[0035] In the above process, the authorization verification information is generated by the authorization code of the terminal device and the attribute code of the campus data targeted by the terminal device's confirmation operation according to a preset concatenation rule. Pre-stored authorization information can be pre-stored in the server according to the same steps as the verification operation, based on the terminal device's authorization code and the campus data's attribute code. The verification operation can preferably be set as follows: first, the authorization verification information is input into a first hash algorithm to calculate a first hash value; then, the first hash value is interleaved with the terminal device's device code or address code and input into a second hash algorithm to calculate a second hash value. The resulting verification operation can clearly restrict the terminal device's authorization permissions and verify its network status to prevent network attacks that could steal the terminal device's authorization code and cause malicious tampering of data in the database. The first and second hash algorithms selected in the above verification process can be any combination of any two of the following hash algorithms: MD5, SHA1, SHA256, SHA384, and SHA512. The input length of different hash algorithms can be adjusted to a standard length using the concatenation rules of the device code, address code, and campus data attribute code.

[0036] Building upon the aforementioned operations, this system can further leverage the Web server's calls to SQL statements to query and process authorized static data and unauthorized dynamic data in the database. Specifically, the terminal device first calls the corresponding microservice unit to establish the corresponding data processing model. This model establishment process can be achieved through a graphical drag-and-drop interface. Specifically, several calling interfaces can be set up within the service unit, and each interface is graphically displayed in the interactive interface. Then, through user drag-and-drop and connection, the data source received by each calling interface and the processing relationships between them are clearly defined, gradually realizing functions such as connecting, merging, filtering, deduplicating, and transforming various campus data. The calling interfaces can also perform data cleaning and transformation (ETL) operations through a preset data cleaning model.

[0037] Therefore, for reference Figure 3 When the service unit triggers the front-end framework to send a request for processing dynamic data to the web server via the WebSocket protocol, the web server can be triggered to parse the database location information corresponding to the type of the dynamic data based on the request. The web server then calls an SQL statement based on the database location information to obtain the corresponding dynamic data and inputs it into the data processing model. This generates corresponding HTML forms according to the user-defined interfaces and sends the HTML form feedback file corresponding to the dynamic data to the web server via the WebSocket protocol. This allows the front-end framework to display the processing results on the terminal device's display page based on the HTML form content. The front-end framework also stores the processing results in the dynamic database according to the user's definition of the processing results in the interactive interface, establishing a link channel between the dynamic database and the main database for subsequent authorization updates by the terminal device.

[0038] Therefore, this system can provide a unified, visual data acquisition and processing tool on terminal devices via the Web, allowing users to customize and select appropriate data sources for addition, deletion, and modification within authorized permissions. This application can achieve integration with various mainstream databases through a B / S architecture, and can quickly build custom data models based on secondary modeling within microservice units. While ensuring the stability of the main database content, it utilizes dynamic databases to isolate the data processing process, enabling real-time dynamic processing and display of campus data. This application can flexibly integrate with existing mainstream databases in the campus system via the WebSocket protocol through a B / S architecture, and flexibly implement functions such as ETL, connection, merging, filtering, deduplication, and transformation of various types of data through a graphical drag-and-drop interface.

[0039] The aforementioned application system and operating mechanism are specifically applied to the data platform of Hohai University. Since the existing databases within the university need to connect to 23 business systems across 13 departments, and approximately 300 data integration and synchronization tasks are required, resulting in 590 standard tables and 144 integrated data tables, the above method is used to integrate various shared data resources, providing an accurate, timely, unique, and complete data foundation platform system for data sharing in the campus big data scenario.

[0040] The above are merely embodiments of this application, and their descriptions are quite specific and detailed, but they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A digital infrastructure system for heterogeneous data sources in a campus, characterized in that, include: The underlying database comprises a batch-stream integrated lakehouse built from a source data layer, a standard data layer, and an application data layer. The source data layer connects to microservice acquisition units to collect and store dynamic business data. Each dynamic business data item points to a corresponding campus data item in the batch-stream integrated lakehouse via a corresponding link channel. The standard data layer connects to the cleaning and acquisition units in the batch-stream integrated lakehouse to store the processed data. The application data layer connects to several thematic databases, each storing various campus data items confirmed by the microservice acquisition units and thematic data obtained through secondary calculations and solidification. The link channels for campus data items also authorize the collection of corresponding dynamic business data via HTTP / TCP / UDP protocols, ensuring data encryption and security. The front-end framework adopts a B / S architecture to display various campus data items. The front-end framework communicates and interacts with the theme database via HTTP and WebSocket protocols. The front-end framework also includes a dynamic preview unit, which extracts and displays the results of each campus data item after collection, cleaning, and transformation. It also retrieves corresponding dynamic business data from the theme database through a link channel, collects, processes, and displays the corresponding dynamic business data according to the display mode of the corresponding campus data item in the front-end framework. The dynamic business data is output as the corresponding display content after processing in an independent layer. The integrated stream / batch processing and acquisition unit adopts a microservice architecture and uses containerization technology to connect to the open interfaces of various existing system terminals on campus. The integrated stream / batch processing and acquisition unit is connected to the source data layer to acquire heterogeneous data from various existing system terminals on campus and update the corresponding campus data items in the base database through business data flow. Among them, the microservice acquisition unit responds to the terminal device's authorization confirmation operation for campus data, parses the corresponding campus data metadata description information according to the request parameters, and enables the data source server to call SQL statements according to the database location information, load the source data according to the encryption instructions, and update the corresponding campus data in the target database to the dynamic data pointed to by its link channel; For microservice units with authorized confirmation permissions, the database update is implemented through the following steps: Retrieve the authorization code of this terminal device and send the authorization verification information to the web server via the WebSocket protocol; The authorization verification information triggers the web server to perform verification operations. The verification operation result is calculated using any two hash algorithms among MD5, SHA1, SHA256, SHA384, and SHA512. After confirming that the verification result corresponds to the pre-stored authorization information, the verification pass instruction is returned to the microservice unit via the WebSocket protocol. Thus, based on the instruction-triggered microservice unit accepting the terminal device's authorization confirmation operation for campus data, sending a confirmation request for campus data to the Web server via the WebSocket protocol, and triggering the Web server to parse the database location information of the corresponding campus data according to the confirmation request, so that the Web server can call SQL statements to update the corresponding campus data in the main database to the dynamic data pointed to by its connection channel according to the database location information; The pre-stored authorization information is pre-stored in the server according to the authorization code of the terminal device and the attribute code of the campus data, following the same steps as the verification operation; The verification operation is set as follows: first, the authorization verification information is input into the first hash algorithm to calculate the first hash value; then, the first hash value is interleaved with the device code or address code of the terminal device and input into the second hash algorithm to calculate the second hash value; the length of the input value of different hash algorithms is adjusted to the standard length through the splicing rules of device code, address code and campus data attribute code.

2. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 1, characterized in that, The integrated stream / batch processing and acquisition unit is also equipped with a data flow control unit. The data flow control unit is used to extract various campus data items for collection, cleaning, management, and transformation, and retrieve the corresponding dynamic data from the link channel of the microservice acquisition unit, and process the corresponding dynamic data according to the hierarchical aggregation mode in the base database.

3. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 1, characterized in that, The digital base system also includes a data publishing gateway. The data publishing gateway uses a visual construction device to call an SQL database to perform data weaving and publishes corresponding microservices to provide thematic data as support for the thematic database.

4. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 2, characterized in that, The data flow control unit is specifically constructed based on the collection process of campus data items and the canvas of corresponding independent components. It is used to set corresponding data loading, cleaning, and writing requirements for the parameters of each component in the collection process of campus data items, and to dynamically iterate and update the campus data items in real time or periodically according to the update requirements of each parameter.

5. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 1, characterized in that, The microservice architecture of the integrated stream / batch processing acquisition unit includes two data processing modes: In stream processing mode: The microservice architecture of the stream / batch integrated processing acquisition unit collects data from each data acquisition device in real time based on the archived logs of the base database, and transmits the collected data to the target storage unit; In batch processing mode: The microservice architecture of the stream / batch integrated processing acquisition unit collects data from each data acquisition device based on a timed scheduling mechanism, and loads data accordingly through timer triggers and writes the collected data to the target storage unit.

6. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 5, characterized in that, The microservice data collection unit is used to connect with all types of existing dynamic business data on campus, and can collect and connect data through any of the following methods: A connection is established with the source end of the existing system on campus through the JDBC data source connection communication protocol, and dynamic data is accessed within the authorized scope; the data loading device obtains data information and writes it into the message channel; the data writing device consumes data from the message channel and writes it into the target storage unit. Based on message data acquisition technology, the message pipeline is connected through the Kafka message pipeline connector to obtain the data stored in the Topic. The data writing device parses the data in the message format and writes the parsed data into the target storage unit. Based on HTTP / TCP / UDP protocol data acquisition technology, a communication connection is established with the terminal device through authorized IP and port according to the HTTP / TCP / UDP protocol to obtain the data collected by the terminal device. The data format is parsed by the writing device and the parsed content is written to the target storage unit.

7. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 6, characterized in that, The loading and storage of data in each component of the campus data collection process employs corresponding full encryption algorithms, including SM2, SM4, RSA, and AES.

8. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 7, characterized in that, In a fully encrypted algorithm, the public key and / or private key are data strings generated by a hash algorithm based on the characteristic values ​​of the system terminal or the address encoding information of the corresponding interface.

9. The digital infrastructure system for heterogeneous data sources in a campus as described in claim 8, characterized in that, The data acquisition device includes a computing unit, which is used to perform the following steps in response to user operations based on the authorization permissions of the terminal device: Retrieve the authorization and authentication information of the terminal device and send the corresponding authorization verification information to the data source server; The data source server is triggered to perform verification calculations on the authorization verification information. After confirming that the verification calculation result meets the data source connectivity requirements, the verification result is returned to the microservice unit via the HTTP protocol. The microservice collection unit responds to the terminal device's authorization confirmation and verification result for campus data, parses the corresponding campus data metadata description information according to the user request parameters, and triggers the data source server to call SQL statements according to the database location information corresponding to the metadata, loads the source data according to the encrypted instructions, and updates the corresponding campus data item in the base database to the dynamic data pointed to by its linked channel.

Images