Data storage method and system of transportation hub energy management and control system

Abstract

The application provides a traffic hub energy management and control system data warehousing method and system, comprising: receiving multi-source heterogeneous data streams from an energy data collection gateway and extracting key business features, constructing metadata according to the key business features, and performing multi-dimensional data labeling on each business feature; according to a pre-constructed dynamic analysis weight model, routing the metadata to a multi-level analysis queue, and dynamically adjusting the warehousing thread according to an intelligent scheduling strategy; the data stream enters a real-time warehousing channel based on the energy data point object obtained by analysis and based on real-time data judgment criteria to be synchronously written to a target database. The application can realize efficient analysis, hierarchical caching, reliable temporary storage, and intelligent scheduling and warehousing of the whole process management of real-time energy data of each subsystem.

Description

Technical Field

[0001] This application belongs to the field of intelligent transportation and energy management technology, and in particular relates to a data entry method and system for an energy management and control system for transportation hubs. Background Technology

[0002] Data from energy systems in large transportation hubs is characterized by multi-source heterogeneity, high concurrency, strong real-time requirements, and high reliability. However, existing data processing architectures have the following problems when dealing with this scenario: 1) Uneven allocation of parsing resources: High-priority data (such as fire alarms) and low-priority data (such as historical data retransmission) are queued indiscriminately in the parsing queue, resulting in delays in the response of critical business operations; 2) Disconnect between caching and database writes: The common approach is to use a "memory queue + direct database write" model. When the database experiences write bottlenecks due to maintenance, high load, or network fluctuations, the memory queue quickly overflows, resulting in data loss. 3) Lack of a reliable intermediate storage layer: For data that cannot be immediately stored after parsing, existing solutions lack a persistent, structured, and schedulable intermediate storage. Simple disk log files cannot support efficient querying and retries; 4) Simple and crude scheduling strategy: Inbound tasks usually use a fixed thread pool or simple polling, which cannot be dynamically adjusted according to the value of data business and the real-time load of the system, resulting in low resource utilization. 5) Weak anomaly handling and self-healing capabilities: Single point of failure can easily cause data processing link interruption, lack of breakpoint resume and fault self-healing mechanisms, making it difficult to guarantee data consistency.

[0003] Therefore, there is an urgent need for a data entry method and system for the energy management and control system of transportation hubs that can achieve business awareness, resource collaboration, elasticity and reliability, and intelligent scheduling. Summary of the Invention

[0004] In view of this, this application aims to propose a data entry method and system for a transportation hub energy management system to solve at least one of the above-mentioned problems.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows: Firstly, this application provides a method for data entry into a transportation hub energy management system, including: Receive multi-source heterogeneous data streams from the energy data acquisition gateway and extract key business features, construct metadata based on the key business features, and perform multi-dimensional data annotation on each business feature; Based on a pre-built dynamic parsing weight model, the metadata is routed to a multi-level parsing queue, and the ingestion thread is dynamically adjusted according to an intelligent scheduling strategy. The data stream enters the real-time storage channel based on the parsed energy data point objects and real-time data judgment criteria, so as to be synchronously written to the target database.

[0006] Secondly, based on the same inventive concept, this application also provides a data entry device for a transportation hub energy management system, comprising: The data processing module is configured to receive multi-source heterogeneous data streams from the energy data acquisition gateway and extract key business features, construct metadata based on the key business features, and perform multi-dimensional data annotation on each business feature. The intelligent scheduling module is configured to route the metadata to a multi-level parsing queue according to a pre-built dynamic parsing weight model, and dynamically adjust the inbound thread according to the intelligent scheduling strategy. The synchronous write module is configured to write data to the target database synchronously by entering the real-time storage channel based on the parsed energy data point objects and real-time data judgment criteria.

[0007] Thirdly, based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0008] Fourthly, based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in the first aspect.

[0009] Compared with existing technologies, the data entry method and system for the energy management system of transportation hubs described in this application have the following advantages: The data entry method for a transportation hub energy management system described in this application adopts a high-concurrency, multi-source heterogeneous data processing mechanism, which is particularly suitable for the entire process management of massive real-time / near real-time energy data from subsystems such as power distribution, HVAC, lighting, elevators, and photovoltaic energy storage, including efficient parsing, hierarchical caching, reliable temporary storage, and intelligent scheduling entry. Attached Figure Description

[0010] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a data entry method for a transportation hub energy management system according to an embodiment of this application. Figure 2 This is a flowchart illustrating the data stream reception and service feature annotation process described in the embodiments of this application. Figure 3 This is a flowchart illustrating the dynamic parsing weight calculation and parsing queue allocation process described in an embodiment of this application. Figure 4 This is a flowchart of the three-level historical data caching mechanism described in the embodiments of this application; Figure 5 This is a schematic diagram of the data entry device for a transportation hub energy management system as described in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device described in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0012] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0014] Please see Figure 1 As shown in the figure, this embodiment provides a method for data entry into a transportation hub energy management and control system, which specifically includes the following steps: Step S10: Receive multi-source heterogeneous data streams from the energy data acquisition gateway and extract key business features, construct metadata based on key business features, and perform multi-dimensional data annotation on each business feature.

[0015] Specifically, in this embodiment, this step is the "entry and classification link" of data processing. By receiving multi-source heterogeneous data streams and extracting key business features, it provides "data identity" for subsequent dynamic parsing and priority scheduling, thus solving the problem of key business delays caused by the "indiscriminate processing" of data in traditional solutions.

[0016] Specifically, such as Figure 2 As shown, the system first receives multi-source heterogeneous data streams from the energy data acquisition gateway. These data streams cover all subsystems within the transportation hub, including power distribution, HVAC, lighting, elevators, and photovoltaic energy storage. The receiving process employs a high-concurrency network communication framework to ensure stable reception of hundreds of thousands of data packets per second. Secondly, the received data packets undergo preliminary unpacking. This preliminary unpacking stage only extracts the "surface key information" of the data packets, without performing deep protocol parsing. For example, it extracts device addresses and function codes from Modbus message headers, topics from MQTT messages, and object identifiers from BACnet messages, avoiding reception bottlenecks caused by resource consumption from deep parsing.

[0017] Based on the surface information obtained from the initial unpacking, key business characteristics are extracted. Complete energy data metadata is constructed for each data packet, covering four dimensions: "equipment attributes," "protocol characteristics," "business priority," and "spatiotemporal tags," ensuring that data characteristics are quantifiable and schedulable. Specifically, this includes: 1) Equipment and energy attributes: Equipment type: Clearly define the functional category of the equipment to which the data belongs, such as lighting (public area LED lights, emergency lights), elevators (passenger elevators, freight elevators), air conditioning (central air conditioning unit, cabinet air conditioner), photovoltaics (roof photovoltaic panels, energy storage inverters), for subsequent data aggregation by equipment type; Energy type: Label the energy medium corresponding to the data, such as electricity (voltage, current, power), water (cold water flow rate, hot water flow rate), cold (temperature difference between chilled water supply and return water), and heat (heating water supply temperature), to support energy classification statistics; Data Categories: Differentiate between the real-time nature and purpose of data, such as real-time telemetry (current data updated every second), equipment status (air conditioner "running / stopping" status), and alarm events (fire protection power supply "normal / power failure" alarm), to provide a basis for data processing timeliness requirements.

[0018] 2) Protocol and Complexity: Protocol type: Identifies the industrial protocol used for data transmission, such as Modbus (commonly used in electricity meters and water pumps), BACnet (commonly used in HVAC), MQTT (commonly used in IoT devices), and custom protocols (such as the photovoltaic inverter protocols of some manufacturers). Estimated Parsing Complexity (Parse_Complexity): Assigns a value to the parsing difficulty based on the protocol type and data structure. For example, Modbus single register read (only one 16-bit register needs to be parsed) has a complexity of 1, Modbus multi-register read (multiple consecutive registers need to be parsed and the data concatenated) has a complexity of 2, BACnet complex objects (such as "multi-state value objects" in an air conditioning system) have a complexity of 3, MQTT nested JSON (requiring multiple layers of JSON field parsing) has a complexity of 4, and custom protocols (requiring parsing according to the manufacturer's proprietary format) have a complexity of 5. This value directly affects the subsequent allocation of parsing resources.

[0019] 3) Business Priority: Dynamic annotation based on a predefined rule base covers all core business scenarios in transportation hubs, ensuring that high-value data is processed first. For example: P0 (highest priority, emergency): such as "fire power failure alarm" and "elevator entrapment alarm". This type of data is directly related to safety and requires millisecond-level response. P1 (High Priority): Such as "Air conditioning unit shutdown status" or "Photovoltaic inverter failure", which are related to the operation of core energy-consuming equipment and require processing within seconds; P2 (Medium Priority): Such as "public area lighting power consumption data" and "toilet ventilation fan status", which are related to regular energy consumption and can be processed in minutes; P3 (low priority): such as "historical data supplement" (e.g., the previous day's lighting electricity consumption details), which has no real-time requirement and can be processed during idle periods.

[0020] 4) Spatiotemporal tags: Physical location: The specific space where the associated equipment is installed, such as "Departure Hall on the 2nd floor of Terminal A" or "Platform No. 3 of High-speed Railway Station Area B"; supports energy consumption analysis by area, which facilitates the analysis of the load rate of the equipment in the area; Energy subsystems: Energy management subsystems to which the associated equipment belongs, such as "transformer and distribution electronic system", "heating, ventilation and air conditioning subsystem", "photovoltaic and energy storage electronic system", which facilitates subsystem-level data monitoring and fault location, equipment energy consumption statistics, and supports equipment load rate statistics and energy efficiency statistics in different energy subsystems.

[0021] Spatiotemporal tags are mainly used to mark the physical location of equipment and the energy subsystem to which the equipment belongs, so as to facilitate subsequent energy consumption analysis and load rate statistics.

[0022] Step S20: Based on the pre-built dynamic parsing weight model, route the metadata to the multi-level parsing queue, and dynamically adjust the ingestion thread according to the intelligent scheduling strategy.

[0023] Specifically, in this embodiment, this step is the core of solving the problem of "uneven allocation of parsing resources". By constructing a dynamic parsing weight model, the business value, parsing difficulty, timeliness and system load of the data are combined to achieve "priority parsing of high-value data" and avoid delays in critical business due to queuing of low-priority data.

[0024] Specifically, such as Figure 3 As shown, the system obtains metadata based on business priority, parsing complexity, data freshness, and system load, and obtains dynamic parsing weight values ​​according to the adjustment coefficients assigned to each feature. The data is then routed to multi-level parsing queues by comparing the dynamic parsing weight values ​​with a set threshold range. The number of threads for data entry is dynamically adjusted according to the thread allocation ratio of each queue. Specifically, an independent resident high-priority thread pool is configured for the emergency queue, an elastic thread pool is configured for the high-priority queue, and a shared thread pool is configured for the ordinary queue.

[0025] This model ensures that energy data with high business value, low parsing overhead, and high timeliness are prioritized for processing. The formula for calculating the dynamic parsing weight value, Score, is as follows: ; In the formula, The values ​​represent service priority, such as P0 (emergency): 5 (e.g., fire power alarm), P1 (high): 3 (e.g., air conditioning unit status), P2 (medium): 2 (e.g., public area lighting power), P3 (low): 1 (e.g., historical data retransmission). Indicates the complexity coefficient of protocol parsing; This represents the data freshness (in seconds), with a value of max(0, Current_Time - Data_Generate_Time). This indicates the overall load of the current parsing module (0-100). This represents the freshness decay coefficient, which controls the effect of freshness on... The extent of the impact; , , , This indicates a configurable adjustment factor that can be dynamically adjusted according to the hub's operating hours (such as peak / off-peak flight periods) to meet [specific needs / requirements]. .

[0026] according to The data is routed to different parsing queues based on the value. The threshold is not fixed, but dynamically adapted by the system based on the real-time load. The higher the load, the higher the threshold, ensuring that only core data enters the high-priority queue and avoiding queue backlog. Emergency queue: Score ≥ 2.0, only handles P0 level data (such as alarms), configures an independent resident high-priority thread pool (the number of threads is fixed at 1 / 2 of the number of CPU cores) to ensure that such data does not need to wait and starts parsing in milliseconds; High-priority queue: 1.0≤Score<2.0, handles P1 level data (such as core device status), configured with an elastic thread pool (the number of threads ranges from 1 / 4 to 1 / 2 of the number of CPU cores), and dynamically expands according to the queue backlog length; Normal queue: Score < 1.0, processes P2 / P3 level data (such as lighting electricity consumption, historical data retransmission), configured with a shared thread pool (the number of threads is 1 / 4 of the number of CPU cores), and only occupies resources when there are no tasks in the emergency and high priority queues to avoid affecting core business.

[0027] This embodiment uses business feature annotation (priority, protocol complexity, spatiotemporal labels, etc.) and a dynamic parsing weight model to ensure that high-value data (such as fire alarms and elevator entrapment) is processed first, reducing the response latency of critical business by more than 90% and solving the response latency problem caused by uneven resource allocation in traditional solutions.

[0028] Furthermore, this embodiment establishes a three-level memory queue system: "Urgent" (P0), "High" (P1), and "Normal" (P2), routing data packets to the appropriate queue based on the score. The multi-level parsing queue is implemented based on a memory-blocking queue (such as Java's LinkedBlocking Queue), and different queues employ a "priority isolation" mechanism to ensure that tasks in high-priority queues are not preempted by low-priority tasks. 1) Emergency Queue (P0): It adopts "unbounded queue + exclusive thread pool". The queue has no capacity limit (to avoid P0 level data loss). The threads in the thread pool are "core threads" (which will not be destroyed). Even if the system load is high, it can ensure real-time parsing of alarm data. 2) High-priority queue (P1): A "bounded queue (capacity 10000) + elastic thread pool" is used. When the queue is full, low-score P1 level data is temporarily rejected (and placed in a temporary cache). The thread pool automatically expands according to the queue backlog length (e.g., backlog > 5000) and shrinks when the backlog < 1000, balancing resource consumption and processing efficiency. 3) Normal queue (P2): It adopts "bounded queue (capacity 50000) + shared thread pool". When the queue is full, the data is written to the local disk temporary file (to avoid loss). The thread pool is only started when the number of urgent and high priority queue tasks is less than 100, and uses idle resources to process regular data.

[0029] In addition, to avoid the thread pool being "idle" or "overloaded" in this embodiment, the system monitor collects the "backlog of tasks" and "average processing time" of each queue every second, and dynamically adjusts the thread allocation ratio. : ; In the formula, This indicates the number of backlogged tasks in the current queue. This indicates the average processing time of the current queue. Indicates the smoothing factor (default 0.1).

[0030] By calculating the R value (proportion) of each queue, the total thread resources are allocated according to the R value (total number of threads = number of CPU cores × 2). For example: if the total number of threads in the system is 16, the emergency queue R = 0.3 (100 queues backlog, Tavg = 5ms), the high priority queue R = 0.5 (500 queues backlog, Tavg = 10ms), and the normal queue R = 0.2 (1000 queues backlog, Tavg = 20ms), then the emergency queue will be allocated 4.8 → 5 threads, the high priority queue will be allocated 8 threads, and the normal queue will be allocated 3.2 → 3 threads. It should be noted that the allocation of each queue is subject to the following restrictions: the number of threads in the emergency queue is a minimum of 2 (to ensure that P0 level data is not interrupted), and the number of threads in the normal queue is a maximum of 1 / 3 of the total number of threads (to avoid preempting core resources).

[0031] Step S30: The data stream enters the real-time storage channel based on the parsed energy data point objects and the real-time data judgment criteria, so as to synchronously write them to the target database.

[0032] Specifically, in this embodiment, a protocol parser is used to parse the message and convert the parsing result into an energy data point object with a unified format; Once the energy data point object meets the set business priority standard and data freshness standard, the data stream enters the real-time inbound channel; wherein, the real-time inbound channel adopts a "direct memory connection + batch database write" architecture.

[0033] Furthermore, this embodiment employs a "factory pattern" design for the protocol parsing factory to achieve automatic matching of "protocol type → parser," avoiding the poor scalability issues caused by hard coding. A detailed explanation follows: When the system starts, all protocol-specific parsers (such as Modbus Parser, BACnet Parser, MQTT Parser, etc.) automatically register with the factory. The registration information includes "protocol type + supported function code / object type". For example, Modbus Parser registers "Modbus+03 function code (read holding register)" and "Modbus+04 function code (read input register)".

[0034] Once a data packet enters the parsing queue, the protocol identification module extracts its protocol type and function identifier (such as Modbus function code 03) and passes the information to the parsing factory. The factory matches a unique parser based on the "protocol type + function identifier", and the allocation process takes less than 1 millisecond. In this embodiment, each parser is designed with the syntax rules and data structures specific to the protocol to ensure parsing accuracy and efficiency, for example: Modbus Parser: Supports function codes such as 01 (read coil), 03 (read holding register), and 04 (read input register), and can automatically handle register high and low bit concatenation (e.g., 32-bit floating-point numbers require concatenation of two 16-bit registers) and data format conversion (e.g., binary to decimal, BCD to integer). BACnet Parser: Supports parsing of "simulated input objects", "simulated output objects", and "multi-state value objects" of the BACnet protocol, and can extract information such as object identifiers, attribute values ​​(such as air conditioning temperature setpoints) and timestamps; MQTT Parser: Supports multi-level parsing of JSON data, and can convert nested JSON into standardized key-value pairs based on predefined field mapping tables (such as "current" → "current", "power" → "power").

[0035] After parsing the original message, the parser converts the result into a unified format "Energy Data Point Object" to ensure that subsequent caching and database entry processes do not require handling protocol differences. The format definition is as follows: ; In the formula, This represents the device ID number; each device has only one unique code. Indicates the data type number; Represents numerical values. This indicates the time when this data point was collected; This represents the data quality code, which is assigned by the parser based on the data validity verification result: 0 (Normal): The parsed data conforms to the device's physical range (e.g., voltage 220V±10%) and is in the correct format (e.g., the value is not empty or NaN). 1 (Abnormal): Data exceeds physical range (e.g., current is -5A, which is obviously unreasonable) or format is incorrect (e.g., missing JSON fields); 2 (Missing): Some key fields were not retrieved during the parsing process (e.g., Modbus register read timed out, no return value).

[0036] After parsing, the data point object carries the original Score and business priority label, and enters the subsequent caching or database entry process.

[0037] This step supports concurrent parsing of hundreds of thousands of data packets per second through protocol-specific parsers (Modbus / BACnet / MQTT, etc.) and dynamic thread pools; real-time data direct connection channels (P0 / P1 level) ensure millisecond-level data entry, meeting the strong real-time requirements of transportation hubs.

[0038] Furthermore, the parsed data points ( An object must meet both of the following conditions to be included in the real-time data entry channel: ① Business priority ≥ P1: that is For P0 (emergency) or P1 (high) data, this type of data is associated with the operation of security or core equipment and needs to be stored in real time for display on the monitoring interface and alarm triggering. ② Data freshness ≤ 5 seconds: that is, the difference between the current system time and the data point timestamp (dataPoint.timestamp) is ≤ 5 seconds, to ensure that the data entering the database is "near real-time data" and avoid old data occupying real-time database resources.

[0039] It should be noted that there are exceptions. Real-time control commands (such as "turn on the air conditioner" or "turn off the lights") are still forced into the real-time channel even if their freshness is slightly over 5 seconds (e.g., a difference of 6 seconds due to transmission delay), to ensure that the control commands are executed quickly.

[0040] In this embodiment, the real-time inbound channel adopts an architecture of "direct memory connection + batch database writing" to avoid delays in the intermediate storage stage. The specific process is as follows: 1) Data filtering and aggregation: After receiving data point objects in real time, the data is aggregated in a 1-second window according to "device ID + data category". For example, if the current data of the same air conditioner is generated 5 times per second, it is aggregated into 1 time (taking the average value) to reduce the number of database writes. 2) Real-time database selection: Select the target database based on the data category: P0 level alarm data: written to a time series database (such as InfluxDB, Prometheus), supporting fast querying by time range, and simultaneously written to a relational database (such as MySQL) for alarm log storage; 3) P1 level real-time telemetry / equipment status data: written to the time series database for real-time monitoring curve display (such as air conditioner power change curve). 4) Real-time control commands: written to the Redis cache (for fast response to control result queries) and simultaneously written to the MySQL transaction table (to ensure that command execution records are traceable); 5) Write optimization: Use the database's native batch write API, such as MySQL's BATCH INSERT and InfluxDB's Line Protocol batch write, with a single batch size of 100-500 rows (balancing write efficiency and latency), and control the write latency to within 100 milliseconds.

[0041] For data such as "real-time control commands" (e.g., "elevator emergency stop" and "photovoltaic inverter power adjustment"), they must be immediately forwarded to the control processing thread upon being entered into the database, as follows: 1) Command validity verification: After receiving a command, the control processing thread verifies the "device online status" (e.g., whether the device is online via MQTT heartbeat) and the "command permissions" (e.g., whether the command was issued by an authorized user). If the verification fails, the thread returns "invalid command" and logs the result. 2) Command Conversion and Distribution: After verification, the standardized command is converted into a protocol format supported by the device (e.g., converting the "stop elevator" command into a Modbus write coil instruction) and distributed to the device through the data acquisition gateway; 3) Execution result feedback: Receive the execution result returned by the device (such as "elevator stopped successfully" or "execution failed"), update the value field of the data point object (such as "execution result: success"), and synchronously update the real-time library and cache to ensure that the command status is displayed in real time on the monitoring interface.

[0042] In some implementations, in response to energy data point objects failing to meet the set business priority and data freshness standards, data caching is performed through a three-level historical data caching mechanism; By integrating "data business value" and "database load status" through an intelligent scheduler, the ingestion strategy is dynamically adjusted to prioritize the ingestion of high-value cached data.

[0043] Specifically, in this embodiment, this step is the core innovation to solve the problems of "disconnect between caching and database entry" and "lack of reliable intermediate storage layer". Through a three-level cache of "memory-memory-mapped file-SQLite", reliable temporary storage of historical data (P2 / P3 level) is achieved, avoiding data loss caused by database failure, and providing a structured data source for subsequent intelligent scheduling of database entry.

[0044] like Figure 4 As shown, the three-level historical data caching mechanism used in this embodiment includes: (1) Level 1 cache (L1, memory queue); L1 is the "real-time buffer layer," used to cache parsed data within a collection cycle (default 1 minute, configurable). Its features include "high-speed read / write and periodic clearing," as detailed below: 1) Data storage structure: A hash table structure of "device type + time slice" is adopted, that is, the key is "deviceType + current minute (e.g. "air conditioner_202406150800")", and the value is a list of DataPoints of the device type in the current minute, which facilitates subsequent aggregation by device type; 2) Periodic Management: The system checks every second whether the current time has reached the end point of the period (e.g., 00 seconds per minute). Once reached, it triggers the data synchronization of "L1→L2". After synchronization is completed, the L1 queue is cleared to ensure stable memory usage (L1 memory usage in a single period is <100MB). 3) Anomaly protection: If L2 write fails (e.g., insufficient disk space), the L1 queue will not be cleared temporarily, and a retry mechanism will be started (retrying once every 5 seconds). At the same time, the data will be backed up to a temporary memory buffer (maximum capacity 500MB) to avoid data loss.

[0045] (2) Second-level cache (L2, local memory-mapped file) L2 is a "persistence transition layer" that uses memory-mapped files (.mmap) to store data synchronized from L1, balancing memory read / write speed with disk persistence features. A detailed description follows: 1) File creation rules: Files should be named using the format "Equipment Type + Period", such as "Air Conditioner_202406150800.mmap", representing air conditioning equipment data for the period ending at 8:00 AM on June 15, 2024. The maximum size of each file is set to 100MB (configurable). If the data of a certain device type exceeds 100MB in a single period, a new file will be created automatically (such as "Air Conditioner_202406150800_1.mmap"), and an "end status" flag will be added to the old file (such as writing "COMPLETE" at the end of the file). 2) Data writing optimization: By employing memory-mapped I / O technology, files are directly mapped to the process address space, and data writing does not need to go through the kernel buffer, making it 3-5 times faster than traditional file I / O. Data is stored in a "binary serialization" format (such as Protocol Buffers), which saves more than 60% of storage space compared to JSON format, and the size of a single data point after serialization is less than 100 bytes; File monitoring: A dedicated monitoring thread scans the L2 file directory every second to detect files with the "end status" flag. Once a file is found, the "L2→L3" data synchronization is immediately triggered.

[0046] (3) Level 3 cache (L3, SQLite temporary cache) L3 is a "structured temporary storage layer" that converts the binary files of L2 into SQLite database files, providing a "highly efficient and structured" data source for subsequent intelligent scheduling and storage, and solving the problem of difficulty in querying and retrying traditional disk log files.

[0047] 1) SQLite file creation: Create an SQLite file by “device type + date”, such as “air conditioner_20240615.db”, which represents the air conditioner equipment data for June 15th; Each SQLite file contains a standardized data table, with the table structure corresponding one-to-one with the fields of the data point object.

[0048] 2) Data conversion and writing: After the monitoring thread reads the L2 "end status" file, it deserializes the data using Protocol Buffers, converting the data point objects into row records of an SQLite table. This embodiment uses SQLite's batch insert API (such as executemany) to insert 1,000 records at a time, with a write speed of up to 10,000 records per second. At the same time, a transaction is started to ensure data atomicity (either all records are written or all records are rolled back).

[0049] This step ensures zero data loss by building a three-level reliable caching system. It adopts a three-level caching mechanism of memory queue (L1) → memory-mapped file (L2) → SQLite temporary storage (L3) to achieve reliable temporary storage of parsed data. Even if the database fails or the network is interrupted, the data can still be persistently stored, avoiding data loss caused by memory queue overflow, and the data reliability reaches 99.999%.

[0050] This embodiment integrates "data service value" and "database load status" through an intelligent scheduler to dynamically adjust the inbound strategy, avoid database overload, and ensure that high-value data is inbound first.

[0051] Specifically, in this embodiment, the scheduler calculates a comprehensive scheduling priority for each L3SQLite file. The order in which files are added to the database is determined, and the comprehensive scheduling priority formula is as follows: ; In the formula, Indicates the weighting coefficient. , This indicates the average business priority of the data within the file. Indicates the file creation time (in seconds). This indicates the maximum allowed storage time; files exceeding this time will have their priority increased. The data volume factor is set to 1.5 for small files (<10MB) and 1.0 for large files. This encourages small files to be processed first and avoids large files (such as 100MB) from occupying the data entry thread for a long time, thus balancing data entry efficiency.

[0052] The scheduler collects core load metrics every second through a "load proxy" deployed on the database server, as shown in Table 1. It then classifies the load levels based on the metrics and executes the corresponding scheduling strategies to ensure that the inbound tasks match the database's carrying capacity.

[0053] Table 1. Scheduling Strategies Corresponding to Load Levels low load CPU utilization ≤60%, IOPS (Input / Output Operations per Second) ≤6000, write latency ≤10ms Allow parallel insertion of all priority files into the database, set the number of insertion threads to the maximum (e.g., CPU cores × 2), and set the batch write size to 500 records per write to maximize insertion efficiency. medium load CPU utilization 60%-85%, IOPS 6000-9000, write latency 10-20ms Only the top 80% of files in P_schedule are allowed to be written to the database, the number of threads for writing is halved (e.g., the number of CPU cores × 1), and the batch write size is reduced to 300 records per write, thus reducing database pressure. High load CPU utilization > 85%, IOPS > 9000, write latency > 20ms Only P0 level data (such as alarm logs) is allowed to be stored in the database through the "reserved channel" (one dedicated thread is reserved, with a batch size of 100 records per batch). All other files are placed in the "load waiting queue," and the load is checked every 30 seconds. Scheduling is resumed only when the load drops below medium. In Table 1, CPU utilization was collected using operating system commands of the database server (such as the top command in Linux and the tasklist command in Windows). IOPS is collected through database built-in tools (such as MySQL's SHOWENGINE INNODBSTATUS) or third-party monitoring tools (such as Prometheus + NodeExporter); Write latency is calculated by the difference between "time to send write request → time to receive write completion response". Latency is recorded for each write operation and the average of the last 10 write operations is taken.

[0054] In some implementations, it also includes: The cache file replenishment mechanism triggers the replenishment of cache files into the database; the cache file replenishment mechanism prioritizes high-value replenishment data by prioritizing replenishment tasks.

[0055] Specifically, in this embodiment, this step is a supplementary processing of "missing data." For scenarios such as system startup, database recovery, and manual intervention, it triggers the re-entry of L2 / L3 cache files into the database, ensuring that all data ultimately enters the target database and avoiding long-term backlog. A detailed explanation follows: (1) Refill trigger condition The replenishment mechanism combines automatic and manual triggering to cover all scenarios that may result in data not being entered into the database.

[0056] 1) Automatic trigger: After the system restarts, it automatically scans the L2 cache directory (.mmap files) and the L3 cache directory (.db files) to check the "recovery flag" of the files—the "end status" flag of the L2 files and the processStatus field of the L3 files (if there is unprocessed data). If a file is completed (L2 has a "COMPLETE" flag, and L3 processStatus has 0 / 3) and no backup log (a separate backup log file) has been recorded, then add it to the backup task queue and trigger automatic backup.

[0057] Automatically triggered during database recovery: The database connection pool determines the database status through "heartbeat detection" (sending a connection test every 5 seconds). When the status changes from "unavailable" (connection failed, write timeout) to "available" (connection successful, write normal), a 30-second delay is set to trigger a recovery. The purpose of the 30-second delay is to wait for the database load to stabilize (to avoid a large number of patch requests immediately after recovery) and to ensure that all cached files not yet entered into the database have been scanned.

[0058] 2) Manual trigger: Operations and maintenance personnel can initiate targeted rollback through a management interface (such as a web console), supporting two triggering methods: By time period: Select the target time period (e.g., 2024-06-15, 00:00-24:00), and the system will automatically filter out the L2 / L3 files that have not been replenished within that time period; By Device ID: Enter the target device ID (e.g., A001, B002), and the system will filter out unrecovered files containing data for that device. Manually triggered recovery tasks have higher priority than automatically triggered tasks, ensuring that urgent needs are met quickly.

[0059] (2) Calculation of priority for replenishment tasks ( ) To avoid impacting normal data ingestion tasks with data replenishment tasks, and to ensure that high-value replenished data is processed first, the priority of replenishment tasks is calculated. The formula is: ; In the formula, This indicates the business priority weight, which is fixed at 0.6 to ensure that high-value data is prioritized for recovery. This represents the time decay weight, which is fixed at 0.4 to balance the need for compiling old files; This represents the average service priority of the replacement files. Its calculation method is the same as the scheduling priority (e.g., P0=5, P1=3). The higher the average value, the better. The larger; This represents the time decay factor, with a default value of 0.0001. It controls the priority decay of older files—the longer the file was created, the lower its priority. The smaller the value, the slower the decay, ensuring that old files are not ignored for a long time.

[0060] (3) Speed ​​limit recovery control To prevent a sudden surge in database load caused by data rollback tasks (especially rollback of a large number of old files), a control mechanism of "rate limiting + load linkage" is designed, as follows: The maximum recovery rate (Vmax) is a configurable parameter (default 1000 rows / second). It should be set according to the database performance (e.g., the write limit for a single table in MySQL is 2000 rows / second) to ensure that the recovery rate does not exceed the database's capacity. The current replenishment rate V is calculated in real time as follows: V = current number of replenished records / time interval (e.g., if 8000 records are replenished within 10 seconds, V = 800 records / second). When V > Vmax, rate limiting is triggered: Calculate the thread sleep time (milliseconds): ; The thread resumes replenishment after sleeping, reducing the rate to below Vmax.

[0061] During the patching process, the database load level is monitored in real time. If the load rises to "high load" (CPU>85%, IOPS>9000), the patching task is immediately paused and the task is added back to the "patch waiting queue". The database load is checked every 30 seconds. When the load drops to below "medium load" (CPU≤85%, IOPS≤9000), the patching task is resumed and processing continues from the paused position.

[0062] Each patching task execution (start, pause, completion, failure) is logged in detail, including information such as patching file ID, number of patched entries, time taken, rate, and database load. This makes it easy for operations and maintenance personnel to trace the patching status and optimize patching strategies (such as adjusting Vmax).

[0063] This step employs an automated recovery mechanism (triggered by time period / device ID) and load-linked rate limiting to reduce manual intervention; the structured SQLite temporary database supports efficient querying, improving operational efficiency by 50% and solving the problem of traditional disk logs being difficult to retrieve.

[0064] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] Based on the same inventive concept, and corresponding to any of the above embodiments, the embodiments of this application also provide a data entry device for a transportation hub energy management system.

[0066] like Figure 5 As shown, the data entry device for the transportation hub energy management system includes: The data processing module 11 is configured to receive multi-source heterogeneous data streams from the energy data acquisition gateway and extract key business features, construct metadata based on the key business features, and perform multi-dimensional data annotation on each business feature. The intelligent scheduling module 12 is configured to route metadata to a multi-level parsing queue according to a pre-built dynamic parsing weight model, and dynamically adjust the inbound thread according to the intelligent scheduling strategy. The synchronous writing module 13 is configured to write data to the target database synchronously by entering the real-time storage channel based on the parsed energy data point objects and real-time data judgment criteria.

[0067] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0068] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0069] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0070] Figure 6This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0071] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0072] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0073] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0074] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0075] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0076] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0077] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0078] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0079] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0080] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0081] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0082] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0083] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for data entry into a transportation hub energy management and control system, characterized in that, include: Receive multi-source heterogeneous data streams from the energy data acquisition gateway and extract key business features, construct metadata based on the key business features, and perform multi-dimensional data annotation on each business feature; Based on a pre-built dynamic parsing weight model, the metadata is routed to a multi-level parsing queue, and the ingestion thread is dynamically adjusted according to an intelligent scheduling strategy. The data stream enters the real-time storage channel based on the parsed energy data point objects and real-time data judgment criteria, so as to be synchronously written to the target database. The message is parsed using a protocol parser, and the parsing result is converted into an energy data point object with a unified format. Once the energy data point object meets the set business priority standard and data freshness standard, the data stream enters the real-time storage channel; In response to the energy data point object failing to meet the set business priority and data freshness standards, data caching is performed through a three-level historical data caching mechanism; wherein the three-level caching mechanism consists of a memory queue, a memory-mapped file, and an SQLite temporary storage. The intelligent scheduler integrates data service value with database load status to dynamically adjust the ingestion strategy, so that high-value cached data is prioritized for ingestion; wherein, the data service value is obtained by quantifying business priority. When the load level is low, the scheduling strategy is to allow all priority files to be written to the database in parallel, with the number of writing threads set to the maximum and the batch write size set to 500 records / time. When the load level is medium, only the top 80% of files in terms of overall scheduling priority are allowed to be written to the database, the number of writing threads is halved, and the batch write size is reduced to 300 records / time. When the load level is high, only P0 level data is allowed to be written to the database through the reserved channel, and the remaining files enter the load waiting queue. Scheduling is resumed when the load level drops below medium. The cache file replenishment is triggered by a preset cache file replenishment mechanism; wherein, the cache file replenishment mechanism prioritizes high-value replenishment data based on replenishment task priority. The formula is: ； In the formula, Indicates business priority weight; Indicates the time decay weight; This indicates the average business priority of the backup files; Indicates the current time. Indicates the file creation time. This represents the time decay factor; the longer the file was created, the more decay occurs. The smaller.

2. The method according to claim 1, characterized in that: The multi-dimensional data annotation includes equipment and energy attribute annotation, protocol feature annotation, business priority annotation, and spatiotemporal label annotation; The equipment and energy attributes include equipment type, energy type, and data category; The protocol features include the protocol type and the estimated parsing complexity; The business priority is dynamically labeled based on a predefined rule base and divided into multiple levels; The spatiotemporal tag includes physical location and energy subsystem.

3. The method according to claim 2, characterized in that: The business priority, parsing complexity, data freshness, and system load of the metadata are obtained, and a dynamic parsing weight value is obtained based on the adjustment coefficient assigned to each feature. The data is routed to a multi-level parsing queue by comparing the dynamic parsing weight value with a set threshold range; the parsing queue includes a three-level memory queue. The number of threads for data entry is dynamically adjusted based on the thread allocation ratio for each queue. Specifically, an independent, resident high-priority thread pool is configured for the emergency queue, an elastic thread pool is configured for the high-priority queue, and a shared thread pool is configured for the ordinary queue.

4. The method according to claim 1, characterized in that: The real-time data entry channel adopts a direct memory connection and database batch writing architecture to directly save the energy data point objects into memory and establish a data transmission link with the target database through the database native batch writing interface.

5. The method according to claim 1, characterized in that: The intelligent scheduler determines the order in which cached file data is entered into the database by comprehensively scheduling priorities, wherein the formula for the comprehensively scheduling priorities is expressed as: ； In the formula, Indicates the weighting coefficient. , This indicates the average business priority of the data within the file. Indicates the file creation time. Indicates the maximum allowed storage time. Indicates the data volume factor. This indicates the overall scheduling priority.

6. A data entry device for a transportation hub energy management system, characterized in that, include: The data processing module is configured to receive multi-source heterogeneous data streams from the energy data acquisition gateway and extract key business features, construct metadata based on the key business features, and perform multi-dimensional data annotation on each business feature. The intelligent scheduling module is configured to route the metadata to a multi-level parsing queue according to a pre-built dynamic parsing weight model, and dynamically adjust the inbound thread according to the intelligent scheduling strategy. The synchronous write module is configured to write data to the target database synchronously by entering the real-time storage channel based on the parsed energy data point objects and real-time data judgment criteria. The message is parsed using a protocol parser, and the parsing result is converted into an energy data point object with a unified format. Once the energy data point object meets the set business priority standard and data freshness standard, the data stream enters the real-time storage channel; In response to the energy data point object failing to meet the set business priority and data freshness standards, data caching is performed through a three-level historical data caching mechanism; wherein the three-level caching mechanism consists of a memory queue, a memory-mapped file, and an SQLite temporary storage. The intelligent scheduler integrates data service value with database load status to dynamically adjust the ingestion strategy, so that high-value cached data is prioritized for ingestion; wherein, the data service value is obtained by quantifying business priority. When the load level is low, the scheduling strategy is to allow all priority files to be written to the database in parallel, with the number of writing threads set to the maximum and the batch write size set to 500 records / time. When the load level is medium, only the top 80% of files in terms of overall scheduling priority are allowed to be written to the database, the number of writing threads is halved, and the batch write size is reduced to 300 records / time. When the load level is high, only P0 level data is allowed to be written to the database through the reserved channel, and the remaining files enter the load waiting queue. Scheduling is resumed when the load level drops below medium. The cache file replenishment is triggered by a preset cache file replenishment mechanism; wherein, the cache file replenishment mechanism prioritizes high-value replenishment data based on replenishment task priority. The formula is: ； In the formula, Indicates business priority weight; Indicates the time decay weight; This indicates the average business priority of the backup files; Indicates the current time. Indicates the file creation time. This represents the time decay factor; the longer the file was created, the more decay occurs. The smaller.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1-5.

8. A non-transitory computer-readable storage medium, characterized in that, in, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method described in any one of claims 1-5.

Images