Optical power big data resource optimization method in transmission network management system

Abstract

The invention relates to the technical field of communication networks, in particular to an optical power big data resource optimization method in a transmission network management system. Comprising the following steps: dividing a storage unit into a static configuration area, a real-time stream processing area and an analysis cache area, and respectively constructing to meet different storage requirements; optical power features are extracted through edge calculation, a feature vector space is constructed, virtual task groups are divided, and a strategy is bound; creating resource allocation entries, generating a global pre-allocation table, injecting the global pre-allocation table into the analysis cache region, and marking resources capable of being preempted; the analysis engine preferentially executes a high-risk group task, dynamically monitors recycled resources and writes back a result; and a device, city and provincial three-level feedback chain is constructed, and parameter calibration and strategy optimization are realized. The method has the advantages of improving the resource utilization rate, accelerating the data processing efficiency and enhancing the dynamic adaptability of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication network, and in particular to an optical power big data resource optimization method in a transmission network management system. BACKGROUND

[0002] In the operation and maintenance management of communication transmission networks, optical power data, as a core indicator reflecting network link quality and device operating state, its collection, storage and processing capacity directly determines the accuracy and timeliness of network fault diagnosis. With the large-scale deployment of 5G networks and the evolution of all-optical network technology, optical power data presents the characteristics of "three high": first, the sampling frequency is high, in order to accurately capture instantaneous fluctuations, the optical power data sampling interval has been reduced to seconds or even milliseconds, and the amount of data generated by a single device per day can reach GB level; second, the data dimension is high, in addition to the basic power value, it also needs to associate with multiple dimensions such as optical port temperature, transmission distance and service type, resulting in a sharp increase in data structure complexity; third, the real-time requirement is high, abnormal fluctuations in optical power may indicate the risk of link interruption, and data analysis and fault warning need to be completed within milliseconds, otherwise it will cause large-scale service interruption.

[0003] The existing transmission network management system has significant bottlenecks in processing optical power big data: on the one hand, the storage and computing needs of massive optical power data have increased dramatically, and the existing storage resources lack targeted division, resulting in low data access efficiency; on the other hand, the resource allocation in the data processing process is unreasonable, and high-priority tasks may be delayed due to insufficient resource preemption, affecting the timeliness of fault diagnosis and network maintenance; in addition, traditional resource optimization methods lack dynamic adjustment mechanism, and are difficult to adapt to the dynamic fluctuations of optical power data and the differentiated needs of devices in the whole network, resulting in low resource utilization, processing delay and other problems. SUMMARY

[0004] The present application provides an optical power big data resource optimization method in a transmission network management system, aiming to solve the problems of low data access efficiency, high-priority task processing delay, lack of dynamic adjustment mechanism and low resource utilization in existing optical power big data resource optimization. The present application divides the storage resources into three areas, combines edge computing to cluster virtual task groups, globally pre-allocates resources and dynamically optimizes through a three-level feedback chain, achieving the advantages of improving resource utilization, accelerating data processing efficiency and enhancing system dynamic adaptability.

[0005] The technical solution of the present application is as follows:

[0006] The technical solution of the present application is to provide an optical power big data resource optimization method in a transmission network management system, comprising:

[0007] Divide the storage unit in the transmission network management system into a static configuration area, a real-time stream processing area and an analysis cache area;

[0008] Based on the data flow of the real-time stream processing area, the time domain mutation variable and the frequency domain noise spectrum feature of the optical power are extracted through edge computing, a feature vector space is constructed, and the device ports of the whole network are clustered into virtual task groups according to the similarity of the optical power fluctuation features in the feature vector space.

[0009] A resource allocation entry is created for each virtual task group, a global pre-allocation table is generated, the global pre-allocation table is injected into an analysis cache area, and a multiplexing resource block that can be preempted by a high-priority task is marked;

[0010] The analysis engine executes the tasks of the high fluctuation risk group in priority according to the global pre-allocation table, dynamically monitors resource occupation and recycles resources at the end of the life cycle, and writes the analysis results and the original data to the static configuration area;

[0011] A three-level feedback chain of device layer, city layer and provincial layer is constructed; the device layer reports task execution efficiency data to the city layer; the city layer calibrates the global pre-allocation table parameters and reports to the provincial layer; the provincial layer trains the global optimization model, and issues updated feature clustering rules and resource allocation strategies.

[0012] As a further optimization of the method of the application, the construction of the static configuration area includes:

[0013] Structured data is stored using a relational database, and data redundancy is achieved through RAID10 or master-slave replication;

[0014] Unstructured data is stored using a distributed file system, and data reliability is ensured through a multi-copy mechanism;

[0015] For unstructured data, device state snapshots are stored in JSON / Avro format, and a NoSQL database is used for management to support flexible data patterns and efficient queries;

[0016] The construction of the real-time stream processing area includes:

[0017] High-speed storage media is used, including DRAM, persistent memory Intel Optane DC Persistent Memory, NVMe SSD or Z-SSD;

[0018] The data structure uses a ring buffer structure or a time window structure, wherein the ring buffer structure writes data in a fixed-size memory block in a loop to reduce memory allocation overhead;

[0019] The stream processing mechanism uses a stream computing framework to realize real-time feature extraction, and introduces a back pressure mechanism to deal with burst traffic, and dynamically adjusts the data processing rate when the system detects data backlog;

[0020] The construction of the analysis cache area includes:

[0021] Intermediate results are cached in memory and stored in distributed cluster, supporting high-frequency read-write and fast eviction strategy.

[0022] The analysis result is periodically written back to the static configuration area, specifically including:

[0023] The metadata index is used to establish the mapping relationship between the analysis result and the original data, wherein the metadata structure adds an analysis result identifier to each original data record.

[0024] Batch asynchronous write-back is realized through a message queue to avoid blocking the analysis engine.

[0025] As a further optimization of the method of the application, the optical power big data resource optimization method in the transmission network management system according to claim 1, characterized in that the optical power time domain mutation variable and frequency domain noise spectrum feature extraction comprises:

[0026] The sliding window method is used for local statistical analysis of the optical power data, the mean and standard deviation of the data in the window are calculated, if the current time optical power value exceeds the range of mean ± 3 times standard deviation, it is determined as a mutation point, and the exceeding part represents the mutation strength;

[0027] The fast Fourier transform is used to convert the optical power signal to the frequency domain, and the spectral density, signal-to-noise ratio and spectral entropy are extracted as the frequency domain features;

[0028] The feature vector space is in the form of F=[f1,f2,...,f d ];

[0029] Where f i is the characteristic value of each type, including mean, standard deviation, mutation variable strength, spectral density, signal-to-noise ratio, and spectral entropy;

[0030] The feature vector is normalized:

[0031]

[0032] Where min(f) is the minimum value corresponding to the characteristic value f i , and max(f) is the maximum value corresponding to the characteristic value f i .

[0033] As a further optimization of the method of the application, the division of the virtual task group comprises:

[0034] Based on the similarity measurement method in the feature vector space, the fluctuation feature similarity between device ports is quantified, and the similarity measurement method calculation formula is:

[0035]

[0036] Where r ijis the correlation coefficient between device port i and device port j, f ik and f jk are the kth eigenvalue of device port i and device port j respectively, and are the mean of all eigenvalues of device port i and device port j respectively, d is the number of eigenvalues, r ij is in the range [-1, 1].

[0037] As a further optimization of the method of the present application, the generation of the resource allocation entry comprises:

[0038] The resource requirement vector R i =[C i ,M i ,S i ,N i ] represents the resource requirement of the ith task group, wherein: C i is the CPU core requirement; M i is the memory capacity requirement; S i is the storage bandwidth requirement; N i is the network throughput requirement;

[0039] The CPU core requirement is estimated by C , wherein D i is the data processing amount of the task group i, T i is the task execution time, and α is the task complexity coefficient; the memory capacity requirement is estimated by M i = γ·D i + β·T i , wherein γ and β are the memory consumption coefficient and the complexity weight coefficient respectively; the storage bandwidth requirement is estimated by S , wherein δ is the storage access redundancy coefficient; and the network throughput requirement is estimated by N , wherein O is the protocol overhead ratio, and is the peak flow amplification factor.

[0040] The life cycle label L i is calculated by the sliding window method to calculate the active period weight of the task group.

[0041] The reusable resource identifier U ij is based on the cosine similarity to measure the resource requirement similarity between task groups.

[0042] As a further optimization of the method of the present application, the marking of the global pre-allocation table comprises:

[0043] The structure of the global pre-allocation table is:

[0044] G = {(ID1, R1, L1, U 1j ), (ID2, R2, L2, U2j ),...,(ID n ,R n ,L n ,U nj )}

[0045] where ID i is the task group identifier, R i , L i and U ij are the resource requirement vector, the life cycle label and the reusable resource identifier respectively;

[0046] The global pre-allocation table is injected into the analysis cache area using a batch writing strategy, and is batch written according to the life cycle label grouping;

[0047] The annotation of the reusable resource block adopts a bitmap annotation method, and each resource block is allocated a bitmap B k ∈{0,1} m , where m is the total number of task groups; if the i-th bit of the bitmap is 1, it means that the resource block k can be preempted by the task group i. The annotation process is based on the reusable resource identifier U ij .

[0048] As a further optimization of the method of the application, the priority scheduling of the high fluctuation risk group and the task execution include:

[0049] The task priority evaluation adopts a weighted scoring method, and the calculation formula is where: F i is the fluctuation risk level of the task group i, and the value range is [0, 1], which is output by the anomaly detection of the feature vector space; L i is the life cycle label of the task group; R i is the resource requirement vector module length of the task group; w1, w2 and w3 are weight coefficients;

[0050] The resource preemption strategy adopts a preemptive priority queue. If the P j of a newly arrived high priority task group j is greater than the P i , and the currently running task group i allows to be preempted, the execution of i is interrupted and its state is saved, and j is executed preferentially.

[0051] The task execution queue management adopts a double queue mechanism, the main queue stores high priority task groups, and the secondary queue stores low priority task groups. The main queue adopts priority scheduling, and the secondary queue adopts round robin scheduling.

[0052] As a further optimization of the method of the application, the dynamic monitoring of resource occupation and resource recycling includes:

[0053] The resource monitoring index design covers:

[0054] Computing resource: CPU utilization, memory occupancy;

[0055] Storage resource: storage bandwidth usage, disk IOPS;

[0056] Network resource: network throughput, packet loss rate;

[0057] The resource recycling strategy distinguishes between reusable resources and exclusive resources:

[0058] The reusable resources are directly recycled to the shared resource pool;

[0059] The exclusive resources are marked as to be recycled, and are recycled after the current task group is completely stopped;

[0060] The resource recycling trigger condition is the end of the task group life cycle or the detection of resource waste.

[0061] As a further optimization of the method of the application, the construction of the three-level feedback chain includes:

[0062] The device layer collects task execution efficiency data, including response delay, resource utilization, and task failure rate, and reports it in real time to the city layer through a lightweight agent;

[0063] The city layer dynamically calibrates the global pre-allocation table parameters, including:

[0064] Optimizing resource demand weights using a linear regression model;

[0065] Adjusting the life cycle label threshold in combination with task group activity;

[0066] Calibrating the reusable resource identification threshold based on task group resource demand similarity;

[0067] The provincial layer uses a deep reinforcement learning framework to train a global optimization model, and the strategy issuance uses a hierarchical broadcasting mechanism, implements bidirectional streaming communication through the gRPC protocol, and manages strategy versions through semantic version numbers.

[0068] The technical solutions provided by the embodiments of the application have at least the following beneficial effects:

[0069] By dividing the static configuration area, the real-time stream processing area, and the analysis cache area, hierarchical storage of optical power data is achieved, improving data access efficiency and targeted use of storage resources;

[0070] Using edge computing to extract features and cluster to form virtual task groups reduces data transmission delay, and through dedicated analysis strategies, it realizes differentiated allocation of resources, improving the processing efficiency of high-priority tasks;

[0071] Based on the global pre-allocation table and the resource preemption mechanism, the resource supply of high fluctuation risk tasks is ensured, and the delay of fault diagnosis is reduced.

[0072] The three-level feedback chain realizes the dynamic iteration of the resource optimization strategy, so that the system can adapt to the fluctuation of optical power data and the differentiated needs of all network devices, and continuously improve the resource utilization rate and processing accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 Figure 1 is a schematic diagram of the overall process of the optical power big data resource optimization method in the transmission network management system.

[0074] Figure 2 Figure 2 is a flowchart of the S100 step of the optical power big data resource optimization method in the transmission network management system.

[0075] Figure 3 Figure 3 is a flowchart of the S200 step of the optical power big data resource optimization method in the transmission network management system.

[0076] Figure 4 Figure 4 is a detailed flowchart of the S300 step of the optical power big data resource optimization method in the transmission network management system.

[0077] Figure 5 Figure 5 is a flowchart of the S400 step of the optical power big data resource optimization method in the transmission network management system.

[0078] Figure 6 Figure 6 is a flowchart of the S500 step of the optical power big data resource optimization method in the transmission network management system. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0080] Please refer to Figure 1 which shows the optical power big data resource optimization method in the transmission network management system provided by an embodiment of the present application, which comprises:

[0081] S100: dividing the storage unit in the transmission network management system into a static configuration area, a real-time stream processing area and an analysis cache area.

[0082] S200: based on the data stream of the real-time stream processing area, extracting the optical power time domain mutation and frequency domain noise spectrum features through edge computing, constructing a feature vector space, and clustering the ports of all network devices into virtual task groups according to the similarity of the optical power fluctuation features in the feature vector space.

[0083] S300: Create a resource allocation entry for each virtual task group, generate a global pre-allocation table, inject the global pre-allocation table into the analysis cache area, and mark the multiplexing resource blocks that can be preempted by high-priority tasks.

[0084] S400: The analysis engine executes the tasks of the high-volatility risk group in priority according to the global pre-allocation table, dynamically monitors resource occupation and recycles resources at the end of the life cycle, and writes the analysis results and the original data back to the static configuration area.

[0085] S500: Construct a three-level feedback chain of device layer, city layer and provincial layer; the device layer reports task execution efficiency data to the city layer; the city layer calibrates the global pre-allocation table parameters and reports to the provincial layer; the provincial layer trains the global optimization model, and issues updated feature clustering rules and resource allocation strategies.

[0086] The specific scheme is as follows:

[0087] In the optical power big data resource optimization method in the transmission network management system, the division of S100 storage resources is based on different data processing needs, specifically including a static configuration area, a real-time stream processing area and an analysis cache area. The three areas bear specific functions respectively, and different technical means are adopted to realize efficient data storage and management.

[0088] Please refer to Figure 2 which shows a flowchart of an exemplary optical power big data resource optimization method S100 in a transmission network management system according to the present application, the content of which includes:

[0089] S110: Construction of the static configuration area in the storage unit.

[0090] The static configuration area is used to store the long-term stable configuration information of the transmission network management system, and the construction of the static configuration area needs to meet the needs of high reliability, low frequency access and efficient query.

[0091] In one possible implementation, the storage of the static configuration area adopts a relational database with ACID characteristics, and combines a distributed file system to provide horizontal expansion capability. The relational database can realize data redundancy through RAID10 or master-slave replication, and the distributed file system adopts a multi-copy mechanism to ensure that data can still be accessed normally in the event of hardware failure.

[0092] Based on the above implementation, for structured data, the relational database uses a table structure design for data organization. For unstructured data, the relational database uses JSON / Avro format to store device state snapshots, and combines a NoSQL database management to support flexible data patterns and efficient queries.

[0093] S120: Construction of the real-time stream processing area in the storage unit.

[0094] The real-time stream processing area is responsible for processing high-concurrency optical power data streams, meeting the needs of low-latency writing, real-time computing, and burst traffic response.

[0095] The data of the real-time stream processing area has high write frequency and short-term storage characteristics. In a possible implementation, the storage medium of the real-time stream processing area uses high-speed storage media, such as DRAM, persistent memory Intel Optane DC Persistent Memory, NVMe SSD, or ZSSD.

[0096] To improve data processing efficiency, the data structure of the real-time stream processing area uses an efficient data structure.

[0097] In a possible implementation, the data structure of the real-time stream processing area uses a ring buffer structure, which is suitable for real-time collection of optical power data. The ring buffer structure reduces memory allocation overhead by writing data in fixed-size memory blocks in a loop.

[0098] To improve data processing efficiency, in another possible implementation, the data structure of the real-time stream processing area uses a time window structure.

[0099] The stream processing mechanism of the real-time stream processing area uses a stream computing framework to implement real-time feature extraction. At the same time, to cope with burst traffic, a back pressure mechanism is introduced, that is, when the system detects data accumulation, the data processing rate is dynamically adjusted to avoid system overload.

[0100] S130: Construction of the analysis cache area in the storage unit.

[0101] The analysis cache area is used to temporarily store the intermediate results of the analysis engine, supporting high-frequency read-write and fast eviction policies.

[0102] The analysis cache area uses high-performance storage types. In a possible implementation, the storage type of the analysis cache area uses memory cache and distributed clusters, supporting efficient storage and fast eviction of intermediate results.

[0103] The computing results of the analysis cache area need to be periodically written back to the static configuration area to update the running state of the device ports and the resource allocation strategy.

[0104] In the optical power big data resource optimization method in the transport network management system, S200 performs clustering analysis based on the feature vector space to divide the network devices into virtual task groups and bind a dedicated analysis strategy to each task group. By optimizing resource allocation and computing scheduling, the efficiency of optical power big data resource optimization can be improved, ensuring that computing resources can efficiently handle different types of fluctuation patterns.

[0105] Reference is made to Figure 3 which shows a flow chart of an exemplary optical power big data resource optimization method S200 in a transport network management system, the contents of which include:

[0106] S210: Edge computing node deployment and optical power data stream acquisition.

[0107] To meet the requirements of low latency and high throughput, computing capabilities are deployed on edge computing nodes, and optical power data streams are acquired in real time. Edge computing nodes are usually deployed near data sources to reduce data transmission latency and improve response speed. In a possible implementation, edge computing nodes use embedded computing devices, intelligent gateways, or dedicated edge servers, which have sufficient computing power and storage capacity to support real-time processing and feature extraction of optical power data.

[0108] The deployment of edge computing nodes meets the requirements of network topology, device performance, and data processing at the same time. In a possible implementation, edge computing nodes use lightweight containerized deployment. For example, a Docker-based microservices architecture is used to ensure flexible scheduling and efficient use of computing resources.

[0109] In a possible implementation, the acquisition of optical power data streams uses a streaming data processing framework to support real-time data stream reception, buffering, and processing.

[0110] During data acquisition, edge computing nodes receive data streams from optical power monitoring devices through network interfaces and store them in real-time stream processing areas for subsequent feature extraction and analysis.

[0111] S220: Optical power time-domain mutation variable and frequency-domain noise spectrum feature extraction.

[0112] During real-time processing of optical power data streams, edge computing extracts time-domain mutation variables and frequency-domain noise spectrum features of optical power data to support subsequent fluctuation pattern recognition and feature vector space construction.

[0113] The time-domain mutation variable of the optical power signal reflects the drastic change of the signal in a short time, which is related to factors such as optical path failure, device aging, or external interference.

[0114] In a possible implementation, in order to effectively detect the mutation quantity, a sliding window method is used for local statistical analysis and calculation of the optical power data. Assuming that x(t) is the optical power value at a certain time, in the time window [t-Δt, t], the mean value μ and the standard deviation σ of the data in the window are calculated. If the optical power value x(t) at the current time exceeds the range of μ±kσ, usually k=3, it is determined as a mutation point, and the part exceeding μ±kσ range represents the mutation intensity.

[0115] The frequency domain characteristics of the optical power signal can reveal the periodic changes and noise components inside it.

[0116] In a possible implementation, the optical power signal is usually converted to the frequency domain by using fast Fourier transform to obtain the spectral density X(f), the signal-to-noise ratio SNR, and the spectral entropy H(f).

[0117] S230: Construct a feature vector space.

[0118] After extracting the time-domain mutation quantity and the frequency-domain noise spectrum characteristics, the time-domain mutation quantity and the frequency-domain noise spectrum characteristics are combined into a feature vector space for subsequent fluctuation pattern recognition and clustering analysis. The form of the feature vector space is represented as:

[0119] F=[f1,f2,...,f d ];

[0120] Where f i is each type of feature value, including the mean value, the standard deviation, the mutation quantity intensity, the spectral density, the signal-to-noise ratio, and the spectral entropy.

[0121] Based on the above feature vector space, in order to improve the expression ability of the feature vector, in a possible implementation, a normalization method is used to standardize the features:

[0122]

[0123] Where min(f) is the minimum value corresponding to the feature value f i , and max(f) is the maximum value corresponding to the feature value f i .

[0124] S240: Virtual task group division based on the feature vector space and exclusive analysis strategy binding.

[0125] Based on the constructed feature vector space, the device ports in the whole network are divided into a plurality of virtual task groups, and an exclusive analysis strategy is bound for each virtual task group. The division of the virtual task groups is based on the similarity analysis in the feature vector space, and the goal is to classify the device ports with similar optical power fluctuation patterns into the same group, so as to be processed by using a unified analysis strategy subsequently.

[0126] After the feature vector space is constructed, the fluctuation feature similarity between device ports is quantified based on the similarity measurement method in the feature vector space.

[0127] In a possible implementation, the similarity measurement method calculation formula is:

[0128]

[0129] wherein r ij is the correlation coefficient between device port i and device port j, f ik and f jk are the kth eigenvalue of device port i and device port j respectively, and are the mean of all features of device port i and device port j respectively, d is the number of features, and r ij is in the range of [1, 1], and the value closer to 1 indicates that the fluctuation trends of the two device ports are more consistent.

[0130] After the similarity measurement of the feature vector space is completed, the device ports in the whole network are divided into several virtual task groups based on the similarity analysis result.

[0131] In a possible implementation, the division adopts Kmeans clustering, hierarchical clustering, or density-based clustering into several virtual task groups.

[0132] In the optical power big data resource optimization method in the transmission network management system, S300 creates a resource allocation entry for each virtual task group and generates a global pre-allocation table to realize efficient scheduling and dynamic optimization of computing resources.

[0133] Please refer to Figure 4 which shows a flowchart of an exemplary optical power big data resource optimization method S300 in a transmission network management system of the present application, and the content includes:

[0134] S310: Create a resource allocation entry for each virtual task group.

[0135] In the transmission network management system, the resource allocation entry is used to accurately describe the computing resource demand, life cycle characteristics, and reusable resource identifier of the task group.

[0136] The computing resource demand is the quantified computing resource required by the virtual task group when executing the analysis task, including CPU, memory, storage bandwidth, network throughput, and other key indicators. In a possible implementation, the resource demand vector R i =[C i ,M i ,S i ,N i] represents the resource requirements of the i-th task group, where: C i CPU core count requirement; M i For memory capacity requirements; S i For storage bandwidth requirements; N i This is for network throughput requirements.

[0137] Specifically, the CPU core count requirement is achieved through... Estimate, where D i T represents the amount of data processed by task group i. i Let M be the task execution time, and α be the task complexity coefficient. Memory capacity requirements are determined by M. i =γ·D i +β·T i The estimation is performed, where γ and β are the memory consumption coefficient and complexity weighting coefficient, respectively. Storage bandwidth requirements are determined through... The estimation is performed, where δ is the storage access redundancy coefficient. Network throughput requirements are determined through... The estimate is given, where O is the protocol overhead ratio and is the peak traffic amplification factor.

[0138] Lifecycle tags are used to describe the active period of a task group and to guide the formulation of resource allocation strategies. Lifecycle tag L i The definition integrates historical task execution records and real-time resource usage. In one possible implementation, a sliding window method is used to calculate the lifecycle weight of the task group.

[0139] Reusable resource identifiers are used to describe computing resources that can be shared among task groups. Their purpose is to improve resource utilization while ensuring task independence. Reusable resource identifier U ij The mechanism is based on similarity analysis between task groups. In one possible implementation, cosine similarity is used to measure the similarity of resource requirements between task groups.

[0140] S320: Generate a global pre-allocation table.

[0141] The global pre-allocation table integrates resource allocation entries for all task groups and optimizes resource allocation strategies to maximize system throughput and resource utilization.

[0142] The structure of the global pre-allocation table G is defined as follows:

[0143] G = {ID1,R1,L1,U} 1j ),(ID2,R2,L2,U 2j ),...,(ID n ,R n ,L n U nj )};

[0144] where ID i is the task group identifier, R i , L i and U ij are the resource requirement vector, life cycle label and reusable resource identifier respectively.

[0145] S330: inject the global pre-allocation table into the analysis cache area, and mark the reusable resource blocks that can be preempted by high-priority tasks.

[0146] The analysis cache area is used to temporarily store the intermediate results of the analysis engine. The global pre-allocation table G = {(ID1, R1, L1, U 1j ), (ID2, R2, L2, U 2j ),..., (ID n , R n , L n , U nj )} is injected into the analysis cache area.

[0147] In one possible implementation, the injection of the global pre-allocation table adopts a batch writing strategy. For example, the task group entries are grouped according to the life cycle label, and the corresponding analysis cache area is written in batches.

[0148] The preemption strategy of high-priority tasks needs to balance resource utilization and task response delay. When the system detects resource contention, the reusable resources of low-priority task groups are preempted in priority. The marking of reusable resource blocks needs to clarify the preemptability and ownership of the resources.

[0149] In one possible implementation, a bitmap marking method is used to mark the reusable resource blocks.

[0150] Specifically, each resource block is allocated a bitmap B k ∈{0,1} m , where m is the total number of task groups; if the i-th bit of the bitmap is 1, it means that the resource block k can be preempted by the task group i. The marking process is based on the reusable resource identifier U ij . For example, if the reusable resource identifiers U ij of task groups i and j are greater than δ, mark B k [i] = 1 and B k [j] = 1 in the bitmap of resource block k.

[0151] In the optical power big data resource optimization method in the transmission network management system, S400 ensures the priority execution of high-volatility risk task groups through the efficient scheduling of the analysis engine, while realizing dynamic monitoring, life cycle management and result writing of resources.

[0152] Please refer to Figure 5FIG. 4 shows a flowchart of an exemplary method S400 for optimizing optical power big data resources in a transport network management system according to the present application, which includes the following steps:

[0153] S410: Priority scheduling and task execution for the high fluctuation risk group.

[0154] The priority scheduling for the high fluctuation risk group aims to ensure that the task group with abnormal fluctuation characteristics of optical power obtains priority computing resources to reduce the risk of network failure. The priority scheduling for the high fluctuation risk group includes three links: task priority evaluation, resource preemption strategy, and task execution queue management.

[0155] The task priority is evaluated in combination with the fluctuation risk level, the life cycle label, and the resource demand of the virtual task group. In one possible implementation, the weighted scoring method is used to judge the priority P i of the task group, and the calculation formula is as follows: wherein F i is the fluctuation risk level of the task group i, with a value range of [0, 1], which is output by the anomaly detection of the feature vector space; L i is the life cycle label of the task group; R i is the resource demand vector length of the task group; and w1, w2, and w3 are weight coefficients.

[0156] The resource preemption strategy is to prioritize the execution of high-priority task groups when the system resources are tight. In one possible implementation, the preemption priority queue is used to implement resource scheduling. Specifically, the queue structure of the preemption priority queue is a task queue arranged in descending order of priority, and the high-priority task group is located at the head of the queue. The preemption rule of the preemption priority queue is that if the P j of a newly arrived high-priority task group j is greater than the P i of the currently running task group i, and the task group i allows to be preempted, then the execution of i is interrupted and its state is saved, and j is executed preferentially.

[0157] The purpose of task execution queue management is to support dynamic adjustment of the task execution queue to cope with burst traffic. In one possible implementation, a double queue mechanism is used. Specifically, the main queue stores high-priority task groups, and the secondary queue stores low-priority task groups. The main queue uses priority scheduling, and the secondary queue uses round-robin scheduling, ensuring that low-priority task groups can still obtain computing opportunities when resources are idle.

[0158] S420: Dynamically monitor resource occupation and recycle resources at the end of the life cycle.

[0159] The goal of dynamic monitoring is to track resource usage in real time and adjust resource allocation strategies according to load changes to avoid system overload or resource waste.

[0160] In one possible implementation, the resource monitoring index design covers three types of core resources, namely, computing, storage, and network, and the specific indexes are as follows:

[0161] Computing resource: CPU utilization, memory occupancy.

[0162] Storage resource: storage bandwidth usage, disk IOPS;

[0163] Network resource: network throughput, packet loss rate.

[0164] The goal of resource recycling is to ensure that computing resources are recycled in time after the end of the task group life cycle, so as to improve resource utilization. Resource recycling needs to distinguish between reusable resources and exclusive resources. In one possible implementation, the following recycling strategies are adopted:

[0165] Reusable resource: recycled directly to the shared resource pool.

[0166] Exclusive resource: marked as to be recycled, and recycled after the current task group is completely stopped.

[0167] 430: Analysis result and original data correlation writing.

[0168] The analysis result correlation writing is to bind the analysis result of the task group with the original optical power data to support subsequent decision-making and optimization.

[0169] When binding the analysis result of the task group with the original optical power data, the analysis result needs to be associated with the original data. In one possible implementation, the association method adopts metadata index. Specifically, the metadata structure of the metadata index adds an analysis result identifier to each original data record. The index construction of the metadata index uses an inverted index to establish the mapping relationship between the analysis result and the original data.

[0170] After associating the analysis result with the original data, data is written back to the static configuration area. In one possible implementation, the writing back method adopts batch asynchronous writing back. Specifically, the batch asynchronous writing back steps include:

[0171] Batch writing: group the analysis result by task group, and write it to the static configuration area in batches;

[0172] Asynchronous submission: decouples the writing back operation through a message queue to avoid blocking the analysis engine.

[0173] In the optical power big data resource optimization method in the transmission network management system, S500 realizes dynamic parameter calibration and global optimization model training through a three-level feedback chain of device layer, city layer, and provincial layer, to continuously improve the adaptability of resource allocation strategies and feature clustering rules.

[0174] Please refer to Figure 6Fig. 1 shows a flowchart of an exemplary optical power big data resource optimization method S500 in a transmission network management system according to the present application, the contents of which include:

[0175] S510: Collection and reporting of device layer task execution efficiency data.

[0176] The collection and reporting of device layer task execution efficiency data is the basis of the three-level feedback chain. By collecting the running state, resource consumption and abnormal information of the task group in real time, data support is provided for parameter calibration at the city level. This step involves three core links: task execution efficiency index definition, data collection mechanism and abnormality detection.

[0177] The task execution efficiency index includes:

[0178] Response delay DLY i : the ratio of the actual execution time of the task group i to the expected execution time, reflecting the task scheduling efficiency;

[0179] Resource utilization rate RUC i : the ratio of the actual resource consumption of the task group i to the pre-allocated resource, reflecting the rationality of resource allocation;

[0180] Task failure rate TSF i : the ratio of the number of failures of the task group i to the total number of executions, reflecting the stability of the task.

[0181] In one possible implementation, the comprehensive efficiency score E i is calculated by weighting:

[0182]

[0183] where w4, w5, and w6 are weight coefficients, DLY max and RUC max are the maximum allowed delay and resource utilization rate of the system.

[0184] In one possible implementation, a lightweight agent is used to collect the index in real time, and the data is transmitted to the city level through a streaming data processing framework.

[0185] S520: Calibration and feedback of city level global pre-allocation table parameters.

[0186] Based on the task execution efficiency data reported by the device layer, the city level dynamically calibrates the key parameters of the global pre-allocation table and feeds back the calibration results to the provincial level.

[0187] In one possible implementation, the method steps for dynamically calibrating the global pre-allocation table parameters include:

[0188] The prefecture-level layer receives the raw data reported by the equipment layer, performs cleaning, normalization and feature engineering processing, and provides high-quality input for parameter calibration.

[0189] The prefecture-level layer dynamically optimizes the globally pre-allocated table parameters.

[0190] The calibration of the life cycle label threshold combines the activity changes of the task group, ensuring that resource allocation matches the actual needs of the task.

[0191] The calibration of the reusable resource identification threshold is based on the similarity of resource requirements between task groups, ensuring the rationality of the resource preemption strategy.

[0192] The prefecture-level layer feeds back the calibrated parameters to the provincial level and synchronously updates the global pre-allocation table.

[0193] S530: Training of the provincial-level global optimization model and policy issuance.

[0194] The provincial level trains the global optimization model based on the feedback data from the prefecture level and issues updated feature clustering rules and resource allocation strategies. The global optimization model needs to be trained with multi-dimensional data. In one possible implementation, a deep reinforcement learning (DRL) framework is used to optimize the resource allocation strategy.

[0195] In one possible implementation, the policy issuance adopts a hierarchical broadcasting mechanism:

[0196] Broadcasting level: provincial level → prefecture level → equipment level;

[0197] Transmission protocol: efficient remote invocation protocol based on gRPC, supporting bidirectional streaming communication;

[0198] Version control: manage policy versions through semantic version numbers, support rollback and compatibility check.

[0199] In summary: the application is aimed at the optical power big data resource optimization problem in the transmission network management system, and a complete hierarchical optimization framework is proposed. The scheme realizes through five core processes: S100 builds a storage and resource allocation architecture, divides the system into a static configuration area, a real-time stream processing area and an analysis cache area, and realizes the data full life cycle management; S200 collects optical power data based on edge computing nodes in real time, constructs a feature vector space through methods such as sliding window statistics and FFT spectrum analysis, and divides the virtual task group combined with the correlation coefficient and clustering algorithm; S300 establishes a resource demand model, a life cycle label and a reusable resource identifier, generates a global pre-allocation table and injects it into the cache area, and realizes resource preemption of high-priority tasks through bitmap annotation; S400 designs a priority scheduling mechanism, a dynamic monitoring system and a related write strategy; S500 constructs a three-level feedback chain, collects execution efficiency data at the device layer, dynamically calibrates the pre-allocation parameters at the city layer, and trains the global model at the provincial level through deep reinforcement learning and issues the update strategy. Through the technical innovations such as storage layering, feature-driven dynamic resource scheduling and closed-loop feedback optimization, the scheme realizes the coordinated improvement of resource utilization, task response speed and system stability in optical power big data processing.

[0200] The basic principles of the application are described above in conjunction with specific embodiments, but it should be noted that the advantages, advantages, effects and the like mentioned in the application are only examples and are not limiting, and these advantages, advantages, effects and the like cannot be considered as each embodiment of the application must have. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limited to the above specific details to implement the application.

[0201] The block diagrams of the devices, apparatuses, equipment, systems involved in the application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, which mean "include but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0202] It should also be noted that in the devices, equipment and methods of the application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the application.

[0203] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0204] The preferred embodiments of the application are described above in detail for the purposes of clarity and understanding. Such embodiments are provided by way of example and are not intended to limit the scope of the application, which is limited only by the scope of the appended claims.

Claims

1. A method for optimizing optical power large data resources in a transmission network management system, characterized in that, The method comprises the following steps: Divide the storage unit in the transmission network management system into a static configuration area, a real-time stream processing area, and an analysis cache area; Based on the data stream of the real-time stream processing area, extract the optical power time domain mutation variable and frequency domain noise spectrum feature through edge computing, Construct a feature vector space, and cluster the device ports in the whole network into virtual task groups according to the similarity of the optical power fluctuation features in the feature vector space; Create a resource allocation entry for each virtual task group, generate a global pre-allocation table, inject the global pre-allocation table into the analysis cache area, and mark the multiplexing resource blocks that can be preempted by high-priority tasks; The analysis engine executes the tasks of the high fluctuation risk group in priority according to the global pre-allocation table, dynamically monitors the resource occupation, and recycles the resources at the end of the life cycle, while writing the analysis results and the original data back to the static configuration area; Construct a three-level feedback chain of device layer, city layer and provincial layer; the device layer reports the task execution efficiency data to the city layer; the city layer calibrates the global pre-allocation table parameters and reports to the provincial layer; the provincial layer trains the global optimization model and issues the updated feature clustering rules and resource allocation strategies.

2. The method of claim 1, wherein the method further comprises: The construction of the static configuration area comprises: Using a relational database to store structured data, and realizing data redundancy through RAID10 or master-slave replication; Using a distributed file system to store unstructured data, and guaranteeing data reliability through a multi-copy mechanism; For unstructured data, using JSON / Avro format to store device state snapshots, and combining NoSQL database management to support flexible data mode and efficient query; The construction of the real-time stream processing area comprises: Using high-speed storage media, including DRAM, persistent memory Intel Optane DC Persistent Memory, NVMe SSD or Z-SSD; The data structure uses a ring buffer structure or a time window structure, wherein the ring buffer structure writes data through a fixed-size memory block in a loop, reducing memory allocation overhead; The stream processing mechanism realizes real-time feature extraction through a stream computing framework, and introduces a back pressure mechanism to cope with burst traffic, and dynamically adjusts the data processing rate when the system detects data backlog; The construction of the analysis cache area comprises: Using memory cache and distributed cluster storage to store intermediate results, supporting high-frequency read-write and fast elimination strategy; The analysis results are periodically written back to the static configuration area, including: Using metadata index to establish the mapping relationship between the analysis results and the original data, wherein the metadata structure adds an analysis result identifier to each original data record; Realize batch asynchronous writing through message queue to avoid blocking the analysis engine.

3. The method of claim 1, wherein the method further comprises: The optical power big data resource optimization method in the transmission network management system according to claim 1, wherein the optical power time domain mutation variable and frequency domain noise spectrum feature extraction comprises: Using sliding window method to perform local statistical analysis on optical power data, calculating the mean and standard deviation of the data in the window, if the current time optical power value exceeds the range of mean ± 3 times standard deviation, it is determined as a mutation point, and the exceeding part represents the mutation intensity; Using fast Fourier transform to convert the optical power signal to frequency domain, and extracting the frequency spectrum density, signal-to-noise ratio and spectrum entropy as the frequency domain features. The eigenvector space is in the form F = [f1, f2, ..., f d ]; where f i are various characteristic values, including mean, standard deviation, abruptness, spectral density, signal-to-noise ratio, spectral entropy; The feature vector is normalized: where min(f) is the characteristic value f i corresponding minimum value, max(f) is the characteristic value f i corresponding maximum value.

4. The method of claim 1, wherein the method further comprises: The division of the virtual task group includes: Based on the similarity measure method in the feature vector space, the fluctuation feature similarity between device ports is quantified, and the similarity measure method calculation formula is: where r ij is the correlation coefficient between device port i and device port j, f ik and f jk are the kth eigenvalue of device port i and device port j, respectively, and are the mean of all eigenvalues of device port i and device port j, respectively, d is the number of eigenvalues, and r ij is in the range [-1, 1].

5. The method of claim 1, wherein the method further comprises: The generation of the resource allocation entry includes: Resource requirement vector R i = [C i , M i , S i , N i ] represents the resource requirement of the ith task group, where: C i is the CPU core number requirement; M i is the memory capacity requirement; S i is the storage bandwidth requirement; N i is the network throughput requirement; CPU core count requirement passed Estimate, where D i T represents the amount of data processed by task group i. i Let M be the task execution time, and α be the task complexity coefficient; memory capacity requirements are specified by M. i =γ·D i +β·T i The estimation is performed, where γ and β are the memory consumption coefficient and complexity weighting coefficient, respectively; storage bandwidth requirements are determined by... The estimation is performed, where δ is the storage access redundancy coefficient; network throughput requirements are determined through... The estimate is given, where O represents the protocol overhead ratio and is the peak traffic amplification factor. Life cycle label L i The active period weight of the task group is calculated by using the sliding window method. Reusable resource identity U ij The cosine similarity is used to measure the similarity of resource requirements among task groups.

6. The method of claim 5, wherein the method further comprises: The marking of the global pre-allocation table includes: The structure of the global pre-allocation table is: G = {(ID1, R1, L1, U 1j ),(ID2, R2, L2, U 2j ),...,(ID n , R n , L n , U nj} Wherein, ID i is a task group identifier, R i , L i and U ij are a resource requirement vector, a life cycle label and a reusable resource identifier respectively; When injecting the global pre-allocation table into the analysis cache area, a batch writing strategy is adopted, and batch writing is performed according to the life cycle label group; The annotation of multiplexed resource blocks uses a bitmap annotation method, and each resource block is allocated a bitmap B k ∈ {0, 1} m where m is the total number of task groups; if the i-th bit of the bitmap is 1, it indicates that the resource block k can be preempted by the task group i, and the annotation process is based on the multiplexable resource identifier U ij .

7. The method of claim 1, wherein the method further comprises: determining a number of the optical power data resources; and determining a number of the optical power data resource groups, wherein the number of the optical power data resource groups is less than the number of the optical power data resources. The priority scheduling and task execution of the high fluctuation risk group includes: The task priority evaluation adopts a weighted scoring method, and a calculation formula is Wherein, F i is a fluctuation risk level of the task group i, and a value range [0, 1] is taken by the anomaly detection output of the feature vector space; L i is a life cycle label of the task group; R i is a resource demand vector module length of the task group; w1, w2 and w3 are weight coefficients; The resource preemption policy adopts a preemptive priority queue. If a newly arrived high priority task group j has a P j > P i higher than the currently running task group i, and the task group i allows to be preempted, the execution of i is interrupted and its state is saved, and j is executed in priority. The task execution queue management adopts a double queue mechanism, the main queue stores the high priority task group, and the secondary queue stores the low priority task group, the main queue adopts priority scheduling, and the secondary queue adopts polling scheduling.

8. The method of claim 1, wherein the method further comprises: determining a number of the optical power data resources; and determining a number of the optical power data resource groups. The dynamic monitoring of resource occupation and resource recycling includes: The resource monitoring index design covers: Computing resources: CPU utilization, memory occupancy; Storage resources: storage bandwidth usage, disk IOPS; Network resources: network throughput, packet loss rate; The resource recycling strategy distinguishes between reusable resources and exclusive resources: Reusable resources are directly recycled to the shared resource pool; Exclusive resources are marked as to-be-recycled state, and are recycled after the current task group is completely stopped; The resource recycling trigger condition is the end of the task group life cycle or the detection of resource waste.

9. The method of claim 1, wherein the method further comprises: determining a number of the optical power data resources; and determining a number of the optical power data resource groups. The construction of the three-level feedback chain includes: The device layer collects task execution efficiency data, including response delay, resource utilization, and task failure rate, and reports it to the city layer in real time through a lightweight agent; The city layer dynamically calibrates the global pre-allocation table parameters, including: Optimizing resource demand weights using linear regression models; Adjusting life cycle label thresholds in combination with task group activity levels; Calibrating reusable resource identification thresholds based on task group resource demand similarity; The provincial level uses a deep reinforcement learning framework to train a global optimization model, and the strategy issuance adopts a hierarchical broadcast mechanism, implements bidirectional streaming communication through the gRPC protocol, and manages strategy versions through semantic version numbers.

Images