Resource optimization method for water meter migration data

Abstract

The invention discloses a resource optimization method for water meter migration data, and belongs to the technical field of data migration and resource scheduling. According to the method, the migration request is received, the type of the water meter is identified, the water meters are divided into different batches such as intelligent batches, mechanical batches and ultrasonic batches, the different migration strategies are associated for the batches, and the computing resource units with different concurrent processing capabilities or data verification capabilities are accurately distributed for the batches according to the strategy identifiers; according to the method and the system, migration tasks are executed through the network transmission channels and the mutually independent network transmission channels, the resource utilization rate and the network throughput rate are monitored in real time, and when it is monitored that the resource utilization rate of a specific batch is continuously low, surplus resources can be dynamically allocated to other batches with high loads, so that intelligent rebalance of the resources is achieved. According to the method, through differentiated strategies and dynamic resource scheduling, the problems of non-uniform resource utilization, low efficiency and insufficient reliability in the mixed type water meter data migration process are effectively solved, and the overall migration efficiency and the system resource utilization rate are improved.

Description

Technical Field

[0001] This invention discloses a resource optimization method for water meter migration data, belonging to the field of data migration and resource scheduling technology. Background Technology

[0002] With the rapid development of smart cities and IoT technologies, the collection and management of water meter data is undergoing a profound transformation from traditional mechanical methods to intelligent and diversified approaches. In this process, a large amount of historical water meter data needs to be migrated from old storage systems to new data centers or cloud platforms for centralized analysis and efficient utilization. This migration typically involves a large number of water meters of various types, such as basic mechanical water meters, smart water meters with remote communication capabilities, and high-precision ultrasonic water meters. Their data formats, data volumes, and reliability requirements for the migration process differ significantly.

[0003] Currently, the common practice in handling such water meter data migration tasks is to treat all water meters to be migrated as a whole or to simply group them, and to adopt a relatively uniform resource allocation strategy and migration process. This approach ignores the inherent differences in migration characteristics among different types of water meter data, which can easily lead to an imbalance in resource utilization. For example, insufficient resource allocation for smart water meters with large data volumes and concurrent processing capabilities will result in low migration efficiency, while using a high-concurrency strategy for mechanical water meters with small data volumes and sequential processing may cause data conflicts or resource waste. At the same time, there is a lack of effective integrity protection mechanisms for high-precision water meter data.

[0004] Therefore, the main problem with existing technologies lies in the lack of refined resource scheduling capabilities that match the characteristics of the data during the migration process. This results in the inability to dynamically optimize computing and network resources according to the actual load, leading to low overall migration efficiency. Furthermore, it is difficult to achieve a good balance between efficiency and reliability when facing complex and mixed water meter data migration scenarios. The industry urgently needs an optimization method that can intelligently identify water meter types and dynamically allocate and adjust resources on demand to improve the overall performance and reliability of large-scale water meter data migration. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] According to a first aspect of the present invention, the present invention claims protection for a resource optimization method for water meter migration data, comprising the following steps:

[0007] S1. Receive a water meter data migration request. Based on the water meter identifier set in the water meter data migration request, query and identify the water meter type corresponding to each water meter identifier from the water meter information database. The water meter type includes at least smart water meters, mechanical water meters, and ultrasonic water meters.

[0008] S2. Based on the identified water meter type, the water meter identifier set is divided into multiple migration batches. Water meter identifiers within the same batch have the same water meter type, and each batch is associated with a migration strategy identifier.

[0009] S3. Based on the migration strategy identifier associated with each batch, allocate corresponding computing resource units and network transmission channels to the batch. The resource units allocated to the batch associated with the first migration strategy identifier have a first concurrent processing capability, and the resource units allocated to the batch associated with the second migration strategy identifier have a second concurrent processing capability. The first concurrent processing capability is higher than the second concurrent processing capability. Allocate mutually independent network transmission channels to the batches associated with different migration strategy identifiers.

[0010] S4. Based on the computing resource units and network transmission channels allocated to each batch, execute the historical data migration task corresponding to the water meter identifier within each batch;

[0011] S5. During the execution of the migration task, monitor the resource utilization rate of each computing resource unit and the data throughput rate of each network transmission channel in real time.

[0012] S6. When the resource utilization rate of the computing resource unit corresponding to a specific batch is continuously lower than the first threshold and the data throughput rate of its corresponding network transmission channel is continuously lower than the second threshold, the number of computing resource units or the bandwidth quota of the network transmission channel allocated to that batch is dynamically reduced, and the reduced resources are allocated to other batches that have not completed the migration task.

[0013] Further, in step S2, the step of dividing the migration batches according to the identified water meter type specifically includes:

[0014] S21. All water meter identifiers of the water meter type being smart water meters are assigned to the first migration batch, and the first migration strategy identifier is associated with the first migration batch.

[0015] S22. All water meter identifiers of the water meter type being mechanical water meters are assigned to the second migration batch, and the second migration strategy identifier is associated with the second migration batch.

[0016] S23. All water meter identifiers of ultrasonic water meter type are assigned to the third migration batch, and the third migration strategy identifier is associated with the third migration batch. The resource units allocated to the batch associated with the third migration strategy identifier have real-time data verification capability.

[0017] Furthermore, the allocation of computing resource units and network transmission channels corresponding to the batch associated with the first migration strategy identifier specifically involves the following resource allocation steps:

[0018] S31a. Calculate the estimated total data load of the first migration batch based on the total number of water meter identifiers in the first migration batch and the preset average data volume estimate for each smart water meter migration task.

[0019] S32a. Based on the estimated total data load and the preset baseline processing capacity unit specifications, determine the minimum initial number N of the required computing resource units, where N is an integer greater than or equal to 1.

[0020] S33a. Create a resource allocation queue and apply for N computing resource units from the idle resource pool to add to the resource allocation queue. Each computing resource unit is configured with an equal share of processor time slices and memory space.

[0021] S34a. Allocate at least two parallel network transmission channels for the first migration batch, each channel is configured with an initial bandwidth quota, and the sum of the initial bandwidth quotas of each channel does not exceed the total bandwidth limit preset for the first migration batch.

[0022] S35a. Establish a dynamic mapping relationship between the computing resource units in the resource allocation queue and the at least two network transmission channels, and make periodic adjustments based on the real-time load status of each computing resource unit so that computing resource units with low load are preferentially bound to transmission channels with high throughput to process large data migration subtasks.

[0023] S36a. The water meter identifiers in the first migration batch are evenly distributed to each computing resource unit in the resource allocation queue to form an initial migration task set. Each computing resource unit independently obtains data from its bound network transmission channel and executes the migration task of the assigned water meter identifier.

[0024] Furthermore, the allocation of computing resource units and network transmission channels corresponding to the batch associated with the second migration strategy identifier specifically involves the following resource allocation steps:

[0025] S31b. Based on the total number of water meter identifiers in the second migration batch, determine a dedicated computing resource unit and configure an independent task queue for the dedicated computing resource unit.

[0026] S32b. All water meter identifiers in the second migration batch are inserted into the independent task queue in the order of their coding to form an ordered migration task sequence.

[0027] S33b: Allocate a dedicated network transmission channel for the second migration batch and configure a fixed bandwidth quota for the dedicated network transmission channel, which is lower than the initial bandwidth quota of any parallel network transmission channel allocated for the first migration batch.

[0028] S34b. Configure a task scheduler, wherein the task scheduler controls the dedicated computing resource unit to sequentially retrieve water meter identifiers from the independent task queue and generate corresponding migration tasks.

[0029] S35b: Set a task lock for each migration task taken from the queue. The task lock is activated when the migration task starts to be executed and is released after the data corresponding to the migration task is completely written to the target storage system.

[0030] S36b. Configure a conflict detection mechanism. When the task scheduler detects that the geographical location code of the water meter identifier corresponding to the migration task to be retrieved meets the preset proximity condition with the geographical location code of the water meter identifier of the previous task being executed, the migration task to be retrieved is delayed until the previous task releases its task lock, and then the delayed migration task is reinserted at the head of the task queue.

[0031] S37b. All migration tasks share the exclusive network transmission channel, and at any given time only one migration task holding the task lock is allowed to transmit data through the exclusive network transmission channel.

[0032] Furthermore, the batch allocation corresponding to the associated third migration strategy identifier, including the computing resource unit and network transmission channel, specifically includes:

[0033] Allocate computing resource units with real-time data verification capability to the third migration batch, and allocate network transmission channels with redundant verification data packet transmission capability to the batch.

[0034] The computing resource unit with real-time data verification capability performs an instant integrity check on each data packet read before writing it to the target storage when migrating ultrasonic water meter data.

[0035] Furthermore, step S3, which involves batch allocation of network transmission channels, further includes:

[0036] S301. Based on the batch migration strategy identifier and the estimated data sensitivity level of the batch, select a target channel pool from multiple physically isolated network channel pools.

[0037] S302. In the target channel pool, based on the current available bandwidth historical records of each channel, select channels with available bandwidth volatility lower than a preset volatility threshold as candidate channels.

[0038] S303. From the candidate channels, allocate channels to the batch and set an initial bandwidth weight for each allocated channel.

[0039] Furthermore, during the execution of the migration task in step S4, a migration integrity verification step is also included:

[0040] S41. In the source storage system, generate a migration snapshot marker for each water meter identifier that successfully initiates a migration task, and record the summary characteristics of the source data at the start of the migration.

[0041] S42. After the target storage system confirms that it has received all the migration data corresponding to a certain water meter identifier, it sends a reception confirmation signal to the source storage system.

[0042] S43. After receiving the reception confirmation signal, the source storage system compares the summary characteristics of the current water meter identification data with the summary characteristics of the migration snapshot mark record.

[0043] S44. If the comparison is consistent, it is determined that the migration task of the water meter identifier is completely completed, and the corresponding migration snapshot mark is cleared.

[0044] S45. If the comparison is inconsistent, it is determined that the source data may have changed during the migration of the water meter identifier, triggering the re-migration process of the water meter identifier and adding it to the high-priority retry queue.

[0045] Furthermore, step S6 dynamically reduces the number of computing resource units or the bandwidth quota of network transmission channels allocated to this batch, and reallocates the reduced resources to other batches that have not yet completed their migration tasks. Specifically, the following dynamic resource scheduling steps are performed:

[0046] S61. Continuously collect the CPU utilization, memory utilization, I / O wait time indicators of the corresponding computing resource units for each batch, as well as the data packet transmission volume per second and latency indicators of each network transmission channel.

[0047] S62. For each batch, calculate a comprehensive resource utilization score based on its corresponding computing resource unit index and network transmission channel index.

[0048] S63. Set a resource recycling score threshold. When the comprehensive resource utilization rate score of a certain batch is lower than the resource recycling score threshold for M consecutive monitoring periods, trigger a resource reduction assessment for the batch. M is an integer greater than 1.

[0049] S64. When conducting resource reduction assessment, first assess the total amount of data and task complexity of the remaining unmigrated tasks in this batch, and predict the task completion time under reduced resources.

[0050] S65. If the predicted task completion time is still before the preset deadline for the batch migration, generate a resource reduction instruction; otherwise, suspend resource reduction.

[0051] S66. When executing the resource reduction instruction, if there are multiple computing resource units currently allocated to this batch, select the computing resource unit with the lowest current load index, remove it from the task allocation list of this batch, suspend its processing task, and mark its status as recyclable.

[0052] S67. At the same time, reduce the bandwidth quota allocated to the network transmission channel for this batch according to a preset ratio;

[0053] S68. Query all batches with incomplete migration tasks to see if there are any batches whose comprehensive resource utilization score exceeds the resource overload score threshold for N consecutive monitoring cycles, where N is an integer greater than 1.

[0054] S69. If it exists, the computing resource unit marked as recyclable in step S76 is reassigned to the overloaded batch with the highest score, and the computing resource unit is added to the task allocation list of the overloaded batch, and the task scheduler of the overloaded batch assigns it a migration task.

[0055] S70. Simultaneously, a portion of the bandwidth quota recovered from the low-utilization batch in step S67 is additionally allocated to the network transmission channel corresponding to the overloaded batch.

[0056] Furthermore, the method also includes a migration failure handling step:

[0057] When the number of consecutive failures of a migration task for a certain water meter identifier reaches the preset limit, the migration attempt for that water meter identifier will be suspended.

[0058] Record detailed error logs for the failed task, including the associated water meter type, the computing resource unit used, and the network transmission channel identifier.

[0059] Package the water meter identifier and its associated error information into an abnormal migration work order and store it in the abnormal task database;

[0060] After all other normal batch migration tasks are completed, the administrator interface retrieves the work orders from the abnormal task database to perform manual intervention or retry based on the updated rules.

[0061] Furthermore, the method is executed through a resource scheduling system, the system comprising:

[0062] The migration request receiving module is used to receive and parse water meter data migration requests;

[0063] The water meter type identification module is used to query the water meter information database based on the water meter identifier to identify the water meter type.

[0064] The batch segmentation and strategy matching module is used to segment water meters into batches based on water meter type and associate migration strategy identifiers with the batches.

[0065] The resource scheduling engine is used to allocate and manage computing resource units and network transmission channels for batches based on migration policy identifiers;

[0066] The task execution monitoring module is used to monitor the migration task execution process, resource utilization, and data throughput.

[0067] The dynamic resource allocation module is used to dynamically adjust the resource allocation between batches based on monitoring results.

[0068] This invention discloses a resource optimization method for water meter migration data, belonging to the field of data migration and resource scheduling technology. The invention receives migration requests and identifies water meter types, classifying them into different batches such as smart, mechanical, and ultrasonic meters. Differentiated migration strategies are associated with each batch. Based on the strategy identifier, computing resource units with different concurrent processing capabilities or data verification capabilities are precisely allocated to each batch, along with independent network transmission channels to execute the migration task. Resource utilization and network throughput are monitored in real time. When the resource utilization of a specific batch is consistently low, its surplus resources are dynamically allocated to other batches with higher loads, achieving intelligent resource rebalancing. Through differentiated strategies and dynamic resource scheduling, this invention effectively solves the problems of uneven resource utilization, low efficiency, and insufficient reliability during the migration of mixed-type water meter data, improving overall migration efficiency and system resource utilization. Attached Figure Description

[0069] Figure 1 A flowchart illustrating a resource optimization method for water meter migration data claimed in an embodiment of the present invention;

[0070] Figure 2 This is a second flowchart of a resource optimization method for water meter migration data claimed in an embodiment of the present invention;

[0071] Figure 3 A third flowchart of a resource optimization method for water meter migration data claimed in an embodiment of the present invention;

[0072] Figure 4 The fourth flowchart is a resource optimization method for water meter migration data claimed in an embodiment of the present invention.

[0073] Figure 5 The fifth flowchart is a resource optimization method for water meter migration data claimed in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0075] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications in the embodiments of this application, such as up, down, left, right, front, back, etc., are only used to explain the relative positional relationships and movements between components in a specific orientation as shown in the accompanying drawings. If the specific orientation changes, the directional indications will change accordingly. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0076] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0077] According to the first embodiment of the present invention, referring to Figure 1 This invention claims protection for a resource optimization method for water meter migration data, comprising the following steps:

[0078] S1. Receive water meter data migration request, and according to the water meter identifier set in the water meter data migration request, query and identify the water meter type corresponding to each water meter identifier from the water meter information database. The water meter type includes at least smart water meters, mechanical water meters and ultrasonic water meters.

[0079] S2. Based on the identified water meter type, the water meter identifier set is divided into multiple migration batches. Water meter identifiers within the same batch have the same water meter type, and each batch is associated with a migration strategy identifier.

[0080] S3. Based on the migration strategy identifier associated with each batch, allocate corresponding computing resource units and network transmission channels to the batch. The resource units allocated to the batch associated with the first migration strategy identifier have first concurrent processing capability, and the resource units allocated to the batch associated with the second migration strategy identifier have second concurrent processing capability. The first concurrent processing capability is higher than the second concurrent processing capability. Allocate mutually independent network transmission channels to the batches associated with different migration strategy identifiers.

[0081] S4. Based on the computing resource units and network transmission channels allocated to each batch, execute the historical data migration task corresponding to the water meter identifier within each batch;

[0082] S5. During the migration task, monitor the resource utilization of each computing resource unit and the data throughput of each network transmission channel in real time.

[0083] S6. When the resource utilization rate of the computing resource unit corresponding to a specific batch is continuously lower than the first threshold and the data throughput rate of its corresponding network transmission channel is continuously lower than the second threshold, the number of computing resource units or the bandwidth quota of the network transmission channel allocated to that batch is dynamically reduced, and the reduced resources are allocated to other batches that have not completed the migration task.

[0084] In this embodiment, a water meter data migration request initiated by an upstream business or management platform is received through a predefined API interface or message queue. The request is a structured data packet, and the core field contains a water meter identifier set, which lists the unique numbers, such as serial numbers or codes, of all water meters whose historical data needs to be migrated this time.

[0085] The request is parsed to extract the set of water meter identifiers; then, the water meter information database storing all static information of water meters is accessed; this database uses the water meter identifier as the primary key and records detailed information for each water meter.

[0086] Iterate through each identifier in the water meter identifier set and initiate a query to the water meter information database. The query results include a water meter type field. Based on the value of this field, the water meter identifier is classified as a smart water meter, mechanical water meter, or ultrasonic water meter. If an identifier is found to be missing from the database or the type field is abnormal, the identifier will be recorded as an unknown type and temporarily stored, triggering an exception handling process, but without interrupting the processing of other identifiers.

[0087] After identifying the type of all water meter identifiers, the original set of water meter identifiers is grouped according to the key attribute of water meter type.

[0088] The grouping principle is to group all water meter identifiers of the same type into the same subset, which constitutes a migration batch; therefore, at least three batches will be formed: smart water meter batch, mechanical water meter batch, and ultrasonic water meter batch. Each batch is assigned a unique internal batch number.

[0089] When each batch is created, it is associated with a preset migration strategy identifier based on its water meter type; for example, smart water meter batches are associated with strategy A, which is the first migration strategy identifier, and mechanical water meter batches are associated with strategy B, which is the second migration strategy identifier. This identifier determines the specific mode of subsequent resource allocation and task execution.

[0090] The resource management module is invoked to configure execution resources based on the migration strategy identifier for each batch.

[0091] For batches associated with the first migration strategy identifier, such as smart water meter batches, a special type of computing resource unit will be allocated to them. These units are designed to handle multiple migration tasks simultaneously, meaning they have high initial concurrency capabilities. This may mean that the resource unit has multiple processing threads or processes, capable of processing the reading, conversion, and sending of data from multiple water meters in parallel.

[0092] For batches associated with the second migration strategy identifier, such as mechanical water meter batches, another type of computing resource unit is allocated. This type of unit may be designed to process tasks sequentially or at a lower level of concurrency, i.e., it has a second level of concurrency processing capability, with a lower concurrency limit than the first type.

[0093] At the network level, independent network transmission channels are allocated to batches of different migration strategies. This means that the network bandwidth and connection paths allocated to strategy A batch are physically or logically separate from those allocated to strategy B batch, to prevent tasks of different strategies from competing for or interfering with each other on the network.

[0094] Once resources are allocated, start the migration task executor.

[0095] The actuator generates specific, schedulable migration task instances by batching up the list of water meter identifiers within each batch, combining the allocated computing resource units and network transmission channel information.

[0096] Each task instance is responsible for transferring, transforming, and writing all historical data corresponding to the water meter identifier, such as reading records and event logs, from source storage (e.g., an old database) to target storage (e.g., a new database or data platform). The task runs on the computing resources allocated to its batch and transmits data through a specified network channel.

[0097] The monitoring agent continues to operate during the migration task.

[0098] The monitoring agent collects operational status metrics for each computing resource unit, focusing primarily on resource utilization, which typically includes, but is not limited to, the percentage of CPU usage, memory usage, and disk I / O activity for that unit.

[0099] Meanwhile, the monitoring agent also collects performance metrics for each network transmission channel, with data throughput being the core metric—the amount of data successfully transmitted through that channel per unit of time, such as megabytes per second. Additionally, it may monitor auxiliary metrics such as network latency and packet loss rate.

[0100] This monitoring data is collected in real time and aggregated at the monitoring center for analysis.

[0101] The monitoring center's analysis engine makes judgments based on preset rules. One of the rules is: for a specific batch, if the resource utilization rate of its associated computing resource unit is lower than a certain first threshold value for a continuous period of time, such as the past 5 monitoring cycles, for example, the CPU utilization rate is continuously lower than 30%, and the data throughput rate of its associated network transmission channel is also continuously lower than another set second threshold value for the same period of time, such as the bandwidth utilization rate is continuously lower than 20%, then a dynamic resource allocation assessment is triggered.

[0102] Once triggered, a resource reduction operation is first performed on the batch. The reduction method can be: reducing the number of computing resource units allocated to the batch, for example, from 3 to 2, or reducing the bandwidth quota of its network transmission channel, for example, from 100Mbps to 50Mbps.

[0103] Next, the resources freed up by the aforementioned reduction, such as idle computing resource units and excess bandwidth, are reclaimed and immediately redistributed to other batches whose migration tasks are not yet completed and may be experiencing resource shortages. The resource allocation module determines the target and quantity of resource allocation based on the remaining task volume and current resource utilization of each incomplete batch.

[0104] Furthermore, referring to Figure 2 In step S2, the step of dividing the migration batches according to the identified water meter type specifically includes:

[0105] S21. All water meter identifiers of the water meter type being smart water meters are assigned to the first migration batch, and the first migration strategy identifier is associated with the first migration batch.

[0106] S22. All water meter identifiers of the water meter type being mechanical water meters are assigned to the second migration batch, and the second migration strategy identifier is associated with the second migration batch.

[0107] S23. All water meter identifiers of ultrasonic water meter type are assigned to the third migration batch, and the third migration strategy identifier is associated with the third migration batch. The resource units allocated to the batch associated with the third migration strategy identifier have real-time data verification capability.

[0108] In this embodiment, after type identification is completed, all water meter identifiers identified as smart water meters are filtered out and placed into a newly created batch container, which is named or marked as the first migration batch.

[0109] Simultaneously, an identifier representing high-performance concurrent migration, namely the first migration strategy identifier, is bound to this batch. This strategy identifier is stored as a metadata attribute of this batch, and all subsequent resource scheduling and behavior control for this batch will query this attribute.

[0110] All water meter identifiers identified as mechanical water meters are filtered out and placed into a newly created batch container, which is named or labeled as the second migration batch.

[0111] Simultaneously, an identifier representing a standard or sequential migration, namely the second migration strategy identifier, is bound to this batch. This strategy pre-determines a different processing method and resource requirements than those for smart water meters.

[0112] All water meter identifiers identified as ultrasonic water meters are filtered out and placed into a third newly created batch container, which is named or labeled as the third migration batch.

[0113] At the same time, an identifier representing the high-reliability verification migration, namely the third migration strategy identifier, is bound to this batch.

[0114] Based on this strategy, during subsequent resource allocation, computing resource units with real-time data verification capabilities will be specifically sought out or configured for the third batch. This means that during the migration process, in addition to performing basic data transmission, this resource unit also has built-in data verification logic, enabling it to immediately verify the integrity or accuracy of each data block before performing subsequent operations.

[0115] Furthermore, referring to Figure 3 The process of allocating computing resource units and network transmission channels corresponding to the batch associated with the first migration strategy identifier specifically involves the following resource allocation steps:

[0116] S31a. Calculate the estimated total data load of the first migration batch based on the total number of water meter identifiers in the first migration batch and the preset average data volume estimate for each smart water meter migration task.

[0117] S32a. Based on the estimated total data load and the preset baseline processing capacity unit specifications, determine the minimum initial number N of the required computing resource units, where N is an integer greater than or equal to 1.

[0118] S33a. Create a resource allocation queue and apply for N computing resource units from the idle resource pool to add to the resource allocation queue. Each computing resource unit is configured with an equal share of processor time slices and memory space.

[0119] S34a. Allocate at least two parallel network transmission channels for the first migration batch, each channel is configured with an initial bandwidth quota, and the sum of the initial bandwidth quotas of each channel does not exceed the total bandwidth limit preset for the first migration batch.

[0120] S35a. Establish a dynamic mapping relationship between the computing resource units in the resource allocation queue and the at least two network transmission channels, and make periodic adjustments based on the real-time load status of each computing resource unit so that computing resource units with low load are preferentially bound to transmission channels with high throughput to process large data migration subtasks.

[0121] S36a. The water meter identifiers in the first migration batch are evenly distributed to each computing resource unit in the resource allocation queue to form an initial migration task set. Each computing resource unit independently obtains data from its bound network transmission channel and executes the migration task of the assigned water meter identifier.

[0122] In this embodiment, the total number of water meter markers contained in the first batch of containers is first counted.

[0123] At the same time, a preset parameter is read from the configuration: the estimated average data volume for each smart water meter migration task. This estimate may be based on historical migration data analysis and represents the estimated data volume that needs to be processed, such as 50MB, when migrating all historical data from a typical smart water meter.

[0124] Multiply the total number of water meter labels by the average estimated data volume to calculate the total data load that needs to be migrated in this batch.

[0125] There is also a preset baseline processing capacity unit specification, which describes the amount of data that a standard computing resource unit can process per unit of time.

[0126] Divide the total data load calculated in the previous step by the baseline processing capacity unit specification to obtain a theoretically required number of units. Round the result up to determine the minimum initial number N of computing resource units required to meet the needs of this migration task.

[0127] Create a logical resource allocation queue for this first migration batch.

[0128] Next, a request is sent to the global idle resource pool, a module that manages all available computing resources, requesting the allocation of N computing resource units that meet the specifications. These units can be virtual machine containers, process groups on physical servers, etc.

[0129] After a successful application, these N units are recorded in the resource allocation queue for that batch. It is ensured that each unit in the queue is allocated an equal share of processing power; for example, each unit receives the same proportion of processor time and a fixed amount of memory.

[0130] In addition to computing resources, allocate network resources for this batch. From the available network resources, allocate at least two network paths that can transmit data simultaneously, i.e., parallel network transmission channels, for this batch.

[0131] Each channel is initially allocated an initial bandwidth quota, for example, 50 Mbps per channel. It is ensured that the total initial quota allocated to all channels in this batch does not exceed the pre-set total bandwidth limit for this type of high-performance migration task, for example, 150 Mbps, to prevent it from consuming excessive network resources.

[0132] Establish a management mechanism to dynamically associate each computing resource unit in the resource allocation queue with the allocated network transmission channel, forming a mapping relationship.

[0133] This mapping is not fixed. It periodically checks the real-time load of each computing resource unit, such as the number of tasks currently being processed, CPU utilization, and real-time throughput of each network channel, for example, every 30 seconds.

[0134] Based on the inspection results, the mapping relationship is dynamically adjusted. A typical adjustment strategy is to prioritize binding currently idle or low-load computing resource units with network channels that have high throughput and ample idle bandwidth. Simultaneously, when there are large-volume migration subtasks, the scheduler will prioritize allocating them to this combination of low-load computing units and high-throughput channels to maximize utilization efficiency.

[0135] Finally, task allocation is performed. The list of all water meter identifiers within the first migration batch is divided into N parts, or nearly equal parts, and each part is assigned to a computing resource unit in the resource allocation queue. This creates N initial migration task sets.

[0136] After each computing resource unit starts up, it reads the data to be migrated from the source according to the dynamic mapping relationship from the network transmission channel it is bound to, and independently executes the migration task of the set of water meter identifiers assigned to it. After processing, it also writes to the target through the channel or a designated channel.

[0137] Furthermore, the allocation of computing resource units and network transmission channels corresponding to the batch associated with the second migration strategy identifier specifically involves the following resource allocation steps:

[0138] S31b. Based on the total number of water meter identifiers in the second migration batch, determine a dedicated computing resource unit and configure an independent task queue for the dedicated computing resource unit.

[0139] S32b. All water meter identifiers in the second migration batch are inserted into the independent task queue in the order of their coding to form an ordered migration task sequence.

[0140] S33b: Allocate a dedicated network transmission channel for the second migration batch and configure a fixed bandwidth quota for the dedicated network transmission channel, which is lower than the initial bandwidth quota of any parallel network transmission channel allocated for the first migration batch.

[0141] S34b. Configure a task scheduler, wherein the task scheduler controls the dedicated computing resource unit to sequentially retrieve water meter identifiers from the independent task queue and generate corresponding migration tasks.

[0142] S35b: Set a task lock for each migration task taken from the queue. The task lock is activated when the migration task starts to be executed and is released after the data corresponding to the migration task is completely written to the target storage system.

[0143] S36b. Configure a conflict detection mechanism. When the task scheduler detects that the geographical location code of the water meter identifier corresponding to the migration task to be retrieved meets the preset proximity condition with the geographical location code of the water meter identifier of the previous task being executed, the migration task to be retrieved is delayed until the previous task releases its task lock, and then the delayed migration task is reinserted at the head of the task queue.

[0144] S37b. All migration tasks share the exclusive network transmission channel, and at any given time only one migration task holding the task lock is allowed to transmit data through the exclusive network transmission channel.

[0145] In this embodiment, a dedicated computing resource unit is designated for the second migration batch. This unit may be a single-threaded process or a service instance configured in low-concurrency mode, specifically for handling this batch of tasks.

[0146] Create a separate task queue for this dedicated computing resource unit. This is an ordered list maintained in memory or persistent storage to store descriptions of migration tasks to be processed.

[0147] All water meter identifiers in the second migration batch are sorted according to a certain order of their identifier codes, such as numerical increments or alphabetical order.

[0148] Then, following this sorted order, the water meter identification information, as task description items, is inserted one by one into the end of the independent task queue created in the previous step, thus forming an ordered migration task sequence. The task at the head of the queue is processed first.

[0149] Allocate a dedicated network transmission channel for this second migration batch, meaning that this channel will not be shared by any other migration batch until the task of this batch is completed.

[0150] For this purpose, a fixed bandwidth quota is configured for the dedicated channel. This quota is set to a value lower than the initial bandwidth quota allocated to any parallel channel for the first batch of smart water meters being migrated, reflecting the differentiation in resource allocation.

[0151] Configuring a task scheduler might involve setting up a module within a dedicated resource unit. The scheduler's responsibility is to retrieve a water meter identifier from the head of a separate task queue, strictly in sequence.

[0152] For each identifier retrieved, the scheduler generates a specific, executable migration task instance based on that identifier and submits it to the dedicated computing resource unit it manages for execution.

[0153] To ensure the atomicity of task execution and data consistency, a task lock is set for each migration task that is retrieved and begins execution.

[0154] This lock is activated or locked when the task begins reading source data, indicating that the water meter's data is being migrated.

[0155] The task lock is only released or unlocked after the task has successfully and completely written all the data to the target storage and received confirmation.

[0156] Configure a conflict detection mechanism. This mechanism will check the relationship between pending tasks and currently executing tasks in the task queue.

[0157] The specific rule is as follows: When the task scheduler is about to retrieve the next task, designated Task B, the conflict detection mechanism analyzes the water meter identifier corresponding to Task B, as well as the water meter identifier corresponding to the currently executing task, designated Task A. If it is found that the two meet certain preset proximity conditions in terms of geographical location encoding, such as belonging to the same building or the same pipeline segment, it is determined that there may be a risk of resource access conflict or data interference, such as shared physical concentrator access conflict.

[0158] To avoid potential conflicts, the scheduler will delay retrieving task B. It will wait until task A finishes execution and releases its task lock before removing task B from the queue. Furthermore, to maintain the overall order, the delayed task B will be reinserted at the front of the task queue, ensuring it is prioritized in the next scheduling round.

[0159] All migration tasks in this batch share the same dedicated network transmission channel for data transmission.

[0160] The channel access rule is that at any given time, only the migration task currently holding the task lock (i.e., the one that is currently executing) is allowed to use the channel to send or receive data. This ensures that network transmissions are also sequential, consistent with the task execution order, further avoiding potential problems caused by concurrency.

[0161] Furthermore, the batch allocation corresponding to the associated third migration strategy identifier, including the computing resource unit and network transmission channel, specifically includes:

[0162] Allocate computing resource units with real-time data verification capability to the third migration batch, and allocate network transmission channels with redundant verification data packet transmission capability to the batch.

[0163] The computing resource unit with real-time data verification capability performs an instant integrity check on each data packet read before writing it to the target storage when migrating ultrasonic water meter data.

[0164] In this embodiment, a special matching process is performed during resource allocation based on the third migration strategy identifier. This involves identifying and allocating computing resource units that are marked or possess real-time data verification capabilities for the third batch. This capability means that the unit has a built-in verification algorithm or coprocessor, enabling it to perform verification calculations while processing data.

[0165] Meanwhile, the network transmission channels allocated for the third batch also need to have the capability to transmit redundant checksum data packets. This may mean that the channel protocol supports or requires the transmission of additional checksums such as CRC or hash values, or that the channel itself has retransmission and error correction mechanisms to ensure the integrity of data during transmission.

[0166] At the task execution level, when using the aforementioned resources to migrate ultrasonic water meter data, the process stipulates that for each data packet or data block read from the source, an immediate integrity check must be performed by the computing resource unit before it is formally written to the target storage. That is, the checksum of the data packet is calculated and compared with the checksum inherent in the data packet or obtained from the source. Only if the comparison matches is the write operation performed; otherwise, a reread or error handling process is triggered. This achieves real-time verification during the data migration process.

[0167] Furthermore, referring to Figure 4 Step S3, which involves batch allocation of network transmission channels, further includes:

[0168] S301. Based on the batch migration strategy identifier and the estimated data sensitivity level of the batch, select a target channel pool from multiple physically isolated network channel pools.

[0169] S302. In the target channel pool, based on the current available bandwidth historical records of each channel, select channels with available bandwidth volatility lower than a preset volatility threshold as candidate channels.

[0170] S303. From the candidate channels, allocate channels to the batch and set an initial bandwidth weight for each allocated channel.

[0171] In this embodiment, a more refined selection logic is introduced when allocating network transmission channels. Multiple physically isolated network channel pools are maintained. These pools may correspond to different network devices, different VLANs, or different physical lines, with traffic isolated from each other.

[0172] The target pool is selected based on two factors: first, the migration strategy identifier of the batch, such as strategy A / B / C; and second, the estimated data sensitivity level assessed based on the batch water meter type and data characteristics. For example, ultrasonic water meter data may be assessed as highly sensitive.

[0173] According to the preset rules of the strategy and sensitivity combination, for example, data with a high sensitivity + high reliability strategy must be placed in a secure isolation pool, and one of the multiple channel pools is selected as the target channel pool.

[0174] Within the selected target channel pool, there may be multiple available channels. View the available bandwidth history of these channels over a past period.

[0175] Calculate the historical bandwidth fluctuation for each channel, i.e., the bandwidth volatility. Then, filter out channels whose volatility is below a certain preset volatility threshold and mark them as candidate channels. This aims to select channels with relatively stable network quality.

[0176] Finally, from the candidate channels, based on the current load balancing strategy such as round-robin or selecting the most idle channel, one or more channels are allocated to the batch currently requesting resources.

[0177] For each allocated channel, an initial bandwidth weight is assigned. This weight is a proportional value used to allocate specific bandwidth quotas among multiple channels within the same batch, or among multiple batches sharing the same channel pool resources. The weight setting may be based on batch priority or estimated data volume.

[0178] Furthermore, referring to Figure 5 During the migration task execution in step S4, a migration integrity verification step is also included:

[0179] S41. In the source storage system, generate a migration snapshot marker for each water meter identifier that successfully initiates a migration task, and record the summary characteristics of the source data at the start of the migration.

[0180] S42. After the target storage system confirms that it has received all the migration data corresponding to a certain water meter identifier, it sends a reception confirmation signal to the source storage system.

[0181] S43. After receiving the reception confirmation signal, the source storage system compares the summary characteristics of the current water meter identification data with the summary characteristics of the migration snapshot mark record.

[0182] S44. If the comparison is consistent, it is determined that the migration task of the water meter identifier is completely completed, and the corresponding migration snapshot mark is cleared.

[0183] S45. If the comparison is inconsistent, it is determined that the source data may have changed during the migration of the water meter identifier, triggering the re-migration process of the water meter identifier and adding it to the high-priority retry queue.

[0184] In this embodiment, before the migration task begins reading data from the source storage, a migration snapshot marker is created in the source storage for the data identifying the water meter. This marker does not copy all the data; instead, it records certain summary characteristics of all the data to be migrated from the water meter at the moment the migration begins, such as the total number of data records, the timestamp of the last record, and the hash value or checksum of key fields. This marker is associated with the migration task ID.

[0185] Once the migration task has completed the transfer of all data corresponding to a specific water meter identifier, and the target storage has finished receiving, writing, and persisting all data, the target storage will proactively send a reception confirmation signal to the source storage. This signal carries the water meter identifier and the migration task ID.

[0186] After receiving the confirmation signal, the source storage locates the previously created migration snapshot marker based on the water meter identifier and task ID.

[0187] Then, the source storage reads the current data of the water meter identifier again. At this time, new data may be generated after the migration starts, and the summary characteristics of its current data are calculated.

[0188] The currently calculated summary features are compared with the historical features recorded in the migration snapshot tags.

[0189] If both are completely consistent, it means that no changes have occurred to this part of the historical data at the source end during the period from the start of the migration to the target end's confirmation of receipt. Therefore, the migration task is considered to be completely completed. Subsequently, the migration snapshot marker corresponding to the water meter identifier is cleared, and resources are released.

[0190] If inconsistencies are found during the comparison, it indicates that the source data may have been modified or new historical records may have been added during the migration process. For example, new readings may have been generated while migrating old readings. Determining that the data state during this migration is inconsistent with that at the time of the snapshot does not automatically confirm success.

[0191] This triggers a re-migration process for the water meter identifier. The new migration process may require different strategies, such as migrating data up to the snapshot time or merging old and new data.

[0192] At the same time, this water meter identifier and its related information are added to a high-priority retry queue to ensure that it can be reprocessed as soon as possible, rather than waiting for regular scheduling.

[0193] Furthermore, step S6 dynamically reduces the number of computing resource units or the bandwidth quota of network transmission channels allocated to this batch, and reallocates the reduced resources to other batches that have not yet completed their migration tasks. Specifically, the following dynamic resource scheduling steps are performed:

[0194] S61. Continuously collect the CPU utilization, memory utilization, I / O wait time indicators of the corresponding computing resource units for each batch, as well as the data packet transmission volume per second and latency indicators of each network transmission channel.

[0195] S62. For each batch, calculate the comprehensive resource utilization score based on its corresponding computing resource unit index and network transmission channel index.

[0196] S63. Set a resource recycling score threshold. When the comprehensive resource utilization rate score of a certain batch is lower than the resource recycling score threshold for M consecutive monitoring periods, trigger a resource reduction assessment for the batch. M is an integer greater than 1.

[0197] S64. When conducting resource reduction assessment, first assess the total amount of data and task complexity of the remaining unmigrated tasks in this batch, and predict the task completion time under reduced resources.

[0198] S65. If the predicted task completion time is still before the preset deadline for the batch migration, generate a resource reduction instruction; otherwise, suspend resource reduction.

[0199] S66. When executing the resource reduction instruction, if there are multiple computing resource units currently allocated to this batch, select the computing resource unit with the lowest current load index, remove it from the task allocation list of this batch, suspend its processing task, and mark its status as recyclable.

[0200] S67. At the same time, reduce the bandwidth quota allocated to the network transmission channel for this batch according to a preset ratio;

[0201] S68. Query all batches with incomplete migration tasks to see if there are any batches whose comprehensive resource utilization score exceeds the resource overload score threshold for N consecutive monitoring cycles, where N is an integer greater than 1.

[0202] S69. If it exists, the computing resource unit marked as recyclable in step S76 is reassigned to the overloaded batch with the highest score, and the computing resource unit is added to the task allocation list of the overloaded batch, and the task scheduler of the overloaded batch assigns it a migration task.

[0203] S70. Simultaneously, a portion of the bandwidth quota recovered from the low-utilization batch in step S67 is additionally allocated to the network transmission channel corresponding to the overloaded batch.

[0204] In this embodiment, the monitoring continuously collects multiple performance metrics from the computing resource units associated with each batch, mainly including: CPU usage percentage, memory usage percentage, and average latency of input / output (I / O) operations.

[0205] Meanwhile, network metrics are collected from the network transmission channels associated with each batch, mainly including: the number of data packets successfully transmitted per second and the average delay time from sending to receiving data packets.

[0206] A comprehensive resource utilization scoring algorithm is designed for each batch. This algorithm takes multiple indicators collected in step S61—CPU utilization, memory utilization, I / O latency, packet transmission volume, and latency—as input, and calculates a score representing the current resource utilization stress level of that batch using a pre-defined weighted calculation formula. A lower score indicates more idle resources; a higher score indicates more strained resources.

[0207] The administrator sets a resource recycling score threshold, which is a low score value.

[0208] The monitoring and analysis engine examines the scoring history of each batch. If it finds that the overall resource utilization score of a batch is lower than the resource recovery score threshold for M consecutive monitoring periods (e.g., M=3 monitoring periods, where the value is 1 minute per period), it is determined that the resource utilization of that batch is consistently too low, and the resource reduction feasibility assessment process for that batch is formally triggered.

[0209] Once the evaluation process begins, the current status of the batch is analyzed first. The number of water meter marker migration tasks remaining in the batch is determined, and based on the type and amount of data from these incomplete tasks, the total remaining data volume and task complexity are estimated.

[0210] Then, a simulation prediction is performed: assuming a certain percentage of resources are reduced, such as reducing one computing unit and reducing some bandwidth, the approximate time required to complete all these tasks is re-predicted based on the remaining task load and the reduced processing capacity, thus obtaining the predicted task completion time.

[0211] The predicted task completion time is compared with the preset migration deadline for this batch, i.e., the latest completion deadline for the task.

[0212] If the predicted time is still earlier than the deadline, it means that even with reduced resources, the task can be completed on time, and a resource reduction instruction is generated.

[0213] If the predicted time is later than the deadline, it indicates that reducing resources may lead to timeouts and the risk is unacceptable. In this case, the resource reduction plan will be suspended and the original resource allocation will be maintained.

[0214] When executing a resource reduction instruction, computing resources are processed first. If multiple computing resource units are currently allocated in the batch, the current load metrics of these units, such as CPU utilization, are checked, and the unit with the lowest load is selected.

[0215] Removing this selected unit from the task assignment list for this batch means that no new tasks will be assigned to it subsequently. Simultaneously, pausing any remaining tasks that may be running on this unit or waiting for them to complete their current tasks, marking its status as reclaimable, and notifying the resource pool management module are also performed.

[0216] Simultaneously, network resources are processed. The bandwidth quota allocated to the network transmission channels for this batch is reduced by a preset reduction ratio, for example, 20%. The reduced quota value is then updated in the network configuration.

[0217] While reclaiming resources, we are also searching for batches that require additional resources. We are querying all batches whose migration tasks are not yet completed.

[0218] For these incomplete batches, examine their overall resource utilization score history. Look for batches whose scores exceed a higher resource overload score threshold for N consecutive monitoring periods (e.g., N=3). Such batches are considered resource-scarce and may be overloaded.

[0219] If more than one such overloaded batch is found in step S68, resource injection will be performed. The computing resource unit marked as reclaimable in step S66 will be reallocated from the resource pool to the one with the highest overall resource utilization score among these overloaded batches, i.e., the one that most urgently needs the resource.

[0220] After allocation, the computing resource unit is added to the task allocation list of the target overload batch. The target batch's own task scheduler will detect the addition of the new resource and immediately begin allocating its backlog or subsequent migration tasks to this new unit.

[0221] Simultaneously, some or all of the bandwidth quota recovered from the low-utilization batch in step S67 is reallocated. This recovered bandwidth is then additionally allocated to the network transmission channel corresponding to the overloaded batch selected in step S69, increasing its bandwidth limit to alleviate network bottlenecks.

[0222] Furthermore, the method also includes a migration failure handling step:

[0223] When the number of consecutive failures of a migration task for a certain water meter identifier reaches the preset limit, the migration attempt for that water meter identifier will be suspended.

[0224] Record detailed error logs for the failed task, including the associated water meter type, the computing resource unit used, and the network transmission channel identifier.

[0225] Package the water meter identifier and its associated error information into an abnormal migration work order and store it in the abnormal task database;

[0226] After all other normal batch migration tasks are completed, the administrator interface retrieves the work orders from the abnormal task database to perform manual intervention or retry based on the updated rules.

[0227] In this embodiment, the number of failed migration attempts for each water meter identifier is counted. When the number of consecutive failed migration attempts for a specific water meter identifier reaches a preset limit, such as 3 times, the system automatically decides to suspend any further automatic migration attempts for that water meter identifier to prevent meaningless resource consumption and error message flooding.

[0228] During the pause, the context of this failure event is logged in detail. This includes: detailed error logs and codes generated for each task failure; the water meter type associated with the water meter identifier; the number or identifier of the computing resource unit used during task execution; and the identifier of the network transmission channel used by the task. This information is packaged together.

[0229] Next, an exception migration work order is created. This work order uses the water meter identifier as the primary key and includes all error messages, failure counts, trigger times, etc., packaged in the previous step as its content. This work order is stored in a dedicated exception task database, separate from the normal task flow.

[0230] The migration continues with all other normal migration batch tasks. Once all regular, automated tasks have completed, query and operation functions are available through the administrator interface. Administrators can retrieve these work orders from the abnormal task database to view detailed error information. Based on this information, administrators can intervene manually, such as checking for source data anomalies, modifying configurations, or authorizing a new round of automated retries based on updated rules, such as changing resource paths or ignoring certain errors.

[0231] Furthermore, the method is executed through a resource scheduling system, the system comprising:

[0232] The migration request receiving module is used to receive and parse water meter data migration requests;

[0233] The water meter type identification module is used to query the water meter information database based on the water meter identifier to identify the water meter type; the batch division and strategy matching module is used to divide the water meter into batches based on the water meter type and associate migration strategy identifiers with the batches.

[0234] The resource scheduling engine is used to allocate and manage computing resource units and network transmission channels for batches based on migration policy identifiers;

[0235] The task execution monitoring module is used to monitor the migration task execution process, resource utilization, and data throughput.

[0236] The dynamic resource allocation module is used to dynamically adjust the resource allocation between batches based on monitoring results.

[0237] In this embodiment, the resource optimization method is specifically executed by a software system called a resource scheduling system. This system adopts a modular design and includes the following six core collaborative functional modules:

[0238] The migration request receiving module is responsible for interfacing with external systems. It continuously listens for preset ports or message topics, and receives migration requests upon arrival. After receiving a request, the module parses and validates the request data packet, extracting key business parameters, the most important of which is the water meter identifier set. It then converts these parameters into the system's internal standard format and passes them to downstream modules. It is also responsible for sending a receipt confirmation email.

[0239] The water meter type identification module holds the necessary configuration for connecting to the water meter database. It receives a list of water meter identifiers from the upstream module and then initiates batch or individual query requests to the database. It processes the query results, parses the water meter type field corresponding to each identifier, and standardizes the type, such as mapping it to an enumerated value. For unrecognized types or identifiers where the query fails, the module generates an exception record and decides whether to skip the query or return an error upstream.

[0240] The batch segmentation and strategy matching module includes a rule engine for batch segmentation. It receives a list of water meter identifiers with labeled types and, according to preset rules as described in claim 2, splits the list into multiple sub-lists by type. Each sub-list is encapsulated as a migration batch object. Simultaneously, it has a built-in strategy mapping table that matches and associates a corresponding migration strategy identifier, such as HIGH_CONCURRENCY or SEQUENTIAL, with each batch object based on its water meter type attribute, thus completing the strategy-based configuration of the batch.

[0241] The resource scheduling engine is the core driving module of the system. It receives batch objects that have completed policy matching. Based on the migration policy identifier associated with the batch, the engine invokes the corresponding resource allocation sub-process as described in claims 3, 4, and 5. It interacts with the underlying infrastructure management platform, such as the cloud computing platform and network control system, to perform the creation, allocation, and configuration of computing resource units (virtual machines and containers), as well as the application for network transmission channels and bandwidth configuration. It is also responsible for managing the lifecycle of these resources during task execution and responding to adjustment instructions from the dynamic resource allocation module.

[0242] The task execution monitoring module acts as the system's eyes. It includes monitoring data collectors deployed across various computing resource units and network links, or pulling data from the infrastructure platform via APIs to collect real-time metrics such as CPU, memory, I / O, network throughput, and latency. It also monitors the migration task itself, tracking its progress, success, and failure. After initial aggregation, all monitoring data is fed back in real-time to the resource scheduling engine and dynamic resource allocation module, providing data support for decision-making. It is also responsible for triggering alerts.

[0243] The dynamic resource allocation module, one of the system's core components, is responsible for optimization. It continuously receives real-time data streams from the task execution monitoring module. Internally, the module runs a resource analysis algorithm, as described in claim 8, which performs scoring and judgment logic to analyze the resource utilization and task progress of each batch in real time. When the algorithm determines that one batch of resources is idle while another batch is under resource pressure, it generates a specific resource adjustment plan on how to reclaim and allocate resources, and issues precise scheduling instructions to the resource scheduling engine. The engine then executes the actual resource addition, reduction, and rebinding operations, achieving dynamic resource rebalancing.

[0244] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0245] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0246] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A resource optimization method for water meter migration data, characterized in that, Includes the following steps: S1. Receive a water meter data migration request. Based on the water meter identifier set in the water meter data migration request, query and identify the water meter type corresponding to each water meter identifier from the water meter information database. The water meter type includes at least smart water meters, mechanical water meters, and ultrasonic water meters. S2. Based on the identified water meter type, the water meter identifier set is divided into multiple migration batches. Water meter identifiers within the same batch have the same water meter type, and each batch is associated with a migration strategy identifier. S3. Based on the migration strategy identifier associated with each batch, allocate corresponding computing resource units and network transmission channels to the batch. The resource units allocated to the batch associated with the first migration strategy identifier have a first concurrent processing capability, and the resource units allocated to the batch associated with the second migration strategy identifier have a second concurrent processing capability. The first concurrent processing capability is higher than the second concurrent processing capability. Allocate mutually independent network transmission channels to the batches associated with different migration strategy identifiers. S4. Based on the computing resource units and network transmission channels allocated to each batch, execute the historical data migration task corresponding to the water meter identifier within each batch; S5. During the execution of the migration task, monitor the resource utilization rate of each computing resource unit and the data throughput rate of each network transmission channel in real time. S6. When the resource utilization rate of the computing resource unit corresponding to a specific batch is continuously lower than the first threshold and the data throughput rate of its corresponding network transmission channel is continuously lower than the second threshold, the number of computing resource units or the bandwidth quota of the network transmission channel allocated to that batch is dynamically reduced, and the reduced resources are allocated to other batches that have not completed the migration task.

2. The method of claim 1, wherein, In step S2, the step of dividing the migration batches according to the identified water meter type specifically includes: S21. All water meter identifiers of the water meter type being smart water meters are assigned to the first migration batch, and the first migration strategy identifier is associated with the first migration batch. S22. All water meter identifiers of the water meter type being mechanical water meters are assigned to the second migration batch, and the second migration strategy identifier is associated with the second migration batch. S23. All water meter identifiers of ultrasonic water meter type are assigned to the third migration batch, and the third migration strategy identifier is associated with the third migration batch. The resource units allocated to the batch associated with the third migration strategy identifier have real-time data verification capability.

3. The method of claim 2, wherein, The process of allocating computing resource units and network transmission channels corresponding to the batch associated with the first migration strategy identifier specifically involves the following resource allocation steps: S31a. Calculate the estimated total data load of the first migration batch based on the total number of water meter identifiers in the first migration batch and the preset average data volume estimate for each smart water meter migration task. S32a. Based on the estimated total data load and the preset baseline processing capacity unit specifications, determine the minimum initial number N of the required computing resource units, where N is an integer greater than or equal to 1. S33a. Create a resource allocation queue and apply for N computing resource units from the idle resource pool to add to the resource allocation queue. Each computing resource unit is configured with an equal share of processor time slices and memory space. S34a. Allocate at least two parallel network transmission channels for the first migration batch, each channel is configured with an initial bandwidth quota, and the sum of the initial bandwidth quotas of each channel does not exceed the preset total bandwidth limit for the first migration batch. S35a. Establish a dynamic mapping relationship between the computing resource units in the resource allocation queue and the at least two network transmission channels, and make periodic adjustments based on the real-time load status of each computing resource unit so that computing resource units with low load are preferentially bound to transmission channels with high throughput to process large data migration subtasks. S36a. The water meter identifiers in the first migration batch are evenly distributed to each computing resource unit in the resource allocation queue to form an initial migration task set. Each computing resource unit independently obtains data from its bound network transmission channel and executes the migration task of the assigned water meter identifier.

4. The method of claim 2, wherein, The process of allocating computing resource units and network transmission channels corresponding to the batch associated with the second migration strategy identifier specifically involves the following resource allocation steps: S31b. Based on the total number of water meter identifiers in the second migration batch, determine a dedicated computing resource unit and configure an independent task queue for the dedicated computing resource unit. S32b. All water meter identifiers in the second migration batch are inserted into the independent task queue in the order of their coding to form an ordered migration task sequence. S33b: Allocate a dedicated network transmission channel for the second migration batch and configure a fixed bandwidth quota for the dedicated network transmission channel, which is lower than the initial bandwidth quota of any parallel network transmission channel allocated for the first migration batch. S34b. Configure a task scheduler, wherein the task scheduler controls the dedicated computing resource unit to sequentially retrieve water meter identifiers from the independent task queue and generate corresponding migration tasks. S35b: Set a task lock for each migration task taken from the queue. The task lock is activated when the migration task starts to be executed and is released after the data corresponding to the migration task is completely written to the target storage system. S36b. Configure a conflict detection mechanism. When the task scheduler detects that the geographical location code of the water meter identifier corresponding to the migration task to be retrieved meets the preset proximity condition with the geographical location code of the water meter identifier of the previous task being executed, the migration task to be retrieved is delayed until the previous task releases its task lock, and then the delayed migration task is reinserted at the head of the task queue. S37b. All migration tasks share the exclusive network transmission channel, and at any given time only one migration task holding the task lock is allowed to transmit data through the exclusive network transmission channel.

5. The method of claim 2, wherein, The computing resource units and network transmission channels corresponding to the batch allocation associated with the third migration strategy identifier specifically include: Allocate computing resource units with real-time data verification capability to the third migration batch, and allocate network transmission channels with redundant verification data packet transmission capability to the batch. The computing resource unit with real-time data verification capability performs an immediate integrity check on each data packet read before writing it to the target storage when migrating ultrasonic water meter data.

6. The method of claim 1, wherein, Step S3, which involves batch allocation of network transmission channels, further includes: S301. Based on the batch migration strategy identifier and the estimated data sensitivity level of the batch, select a target channel pool from multiple physically isolated network channel pools. S302. In the target channel pool, based on the current available bandwidth historical records of each channel, select channels with available bandwidth volatility lower than a preset volatility threshold as candidate channels. S303. From the candidate channels, allocate channels to the batch and set an initial bandwidth weight for each allocated channel.

7. The method of claim 1, wherein, During the migration task execution in step S4, a migration integrity verification step is also included: S41. In the source storage system, generate a migration snapshot marker for each water meter identifier that successfully initiates a migration task, and record the summary characteristics of the source data at the start of the migration. S42. After the target storage system confirms that it has received all the migration data corresponding to a certain water meter identifier, it sends a reception confirmation signal to the source storage system. S43. After receiving the reception confirmation signal, the source storage system compares the summary characteristics of the current water meter identification data with the summary characteristics of the migration snapshot mark record. S44. If the comparison is consistent, it is determined that the migration task of the water meter identifier is completely completed, and the corresponding migration snapshot mark is cleared. S45. If the comparison is inconsistent, it is determined that the source data may have changed during the migration of the water meter identifier, triggering the re-migration process of the water meter identifier and adding it to the high-priority retry queue.

8. The method of claim 1, wherein, Step S6 dynamically reduces the number of computing resource units or the bandwidth quota of network transmission channels allocated to this batch, and reallocates the reduced resources to other batches that have not yet completed their migration tasks. Specifically, the following dynamic resource scheduling steps are performed: S61. Continuously collect the CPU utilization, memory utilization, I / O wait time indicators of the corresponding computing resource units for each batch, as well as the data packet transmission volume per second and latency indicators of each network transmission channel. S62. For each batch, calculate a comprehensive resource utilization score based on its corresponding computing resource unit index and network transmission channel index. S63. Set a resource recycling score threshold. When the comprehensive resource utilization rate score of a certain batch is lower than the resource recycling score threshold for M consecutive monitoring periods, trigger a resource reduction assessment for the batch. M is an integer greater than 1. S64. When conducting resource reduction assessment, first assess the total amount of data and task complexity of the remaining unmigrated tasks in this batch, and predict the task completion time under reduced resources. S65. If the predicted task completion time is still before the preset deadline for the batch migration, generate a resource reduction instruction; otherwise, suspend resource reduction. S66. When executing the resource reduction instruction, if there are multiple computing resource units currently allocated to this batch, select the computing resource unit with the lowest current load index, remove it from the task allocation list of this batch, suspend its processing task, and mark its status as recyclable. S67. At the same time, reduce the bandwidth quota allocated to the network transmission channel for this batch according to a preset ratio; S68. Query all batches with incomplete migration tasks to see if there are any batches whose comprehensive resource utilization score exceeds the resource overload score threshold for N consecutive monitoring cycles, where N is an integer greater than 1. S69. If it exists, the computing resource unit marked as recyclable in step S76 is reassigned to the overloaded batch with the highest score, and the computing resource unit is added to the task allocation list of the overloaded batch, and the task scheduler of the overloaded batch assigns it a migration task. S70. Simultaneously, a portion of the bandwidth quota recovered from the low-utilization batch in step S67 is additionally allocated to the network transmission channel corresponding to the overloaded batch.

9. The method of claim 1, wherein, The method also includes migration failure handling steps: When the number of consecutive failures of a migration task for a certain water meter identifier reaches the preset limit, the migration attempt for that water meter identifier will be suspended. Record detailed error logs for the failed task, including the associated water meter type, the computing resource unit used, and the network transmission channel identifier. Package the water meter identifier and its associated error information into an abnormal migration work order and store it in the abnormal task database; After all other normal batch migration tasks are completed, the administrator interface retrieves the work orders from the abnormal task database to perform manual intervention or retry based on the updated rules.

10. The method according to any one of claims 1 to 9, characterized in that, The method is executed through a resource scheduling system, the system comprising: The migration request receiving module is used to receive and parse water meter data migration requests; The water meter type identification module is used to query the water meter information database based on the water meter identifier to identify the water meter type. The batch segmentation and strategy matching module is used to segment water meters into batches based on water meter type and associate migration strategy identifiers with the batches. The resource scheduling engine is used to allocate and manage computing resource units and network transmission channels for batches based on migration policy identifiers; The task execution monitoring module is used to monitor the migration task execution process, resource utilization, and data throughput. The dynamic resource allocation module is used to dynamically adjust the resource allocation between batches based on monitoring results.

Images