Dynamic scheduling method for distributed resources in real-time processing at large-scale sensor center
By adopting dynamic resource scheduling methods in the sensor center-level distributed processing system, real-time monitoring and allocation of resources, the problems of low sensor resource utilization and low system energy efficiency are solved, efficient resource utilization and system simplification are achieved, and the Cost and complexity.
Patent Information
- Application Number
- CN202111182938.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-11
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-10-11
AI Technical Summary
The existing sensor center-level distributed processing resource scheduling methods have problems such as low resource utilization and energy efficiency ratio, high system complexity and total cost. Especially in large-area monitoring, the pre-allocation of sensor resources leads to resource waste and debris. , the system maintenance is complex and the energy efficiency is not high.
Using dynamic resource scheduling method, the server is divided into work and dormant states by managing the server cluster in real time, and the processing resources are dynamically allocated according to actual needs. After the sensor data processing is completed, resources are recycled to achieve efficient resource utilization and system simplification. .
Improve resource utilization, reduce system complexity and cost, improve system energy efficiency ratio, and support online updates and addition of new sensor processors.
Smart Images

Figure CN113918332B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of multi-sensor aerospace measurement and control, and relates to a large-scale sensor center-level real-time distributed processing resource dynamic scheduling method. Background Art
[0002] When it is necessary to monitor targets in a large area of airspace (this area is called a surveillance area), dozens or even hundreds of heterogeneous multi-sensors (such as radars, optical measurement systems, etc.) distributed in the area are needed to jointly complete the tracking and detection of targets in the area. Each sensor is responsible for a part of the detection area according to its detection capability. The detection data of all sensors are transmitted to the data center in real time for centralized processing. This processing method is also called center-level processing. Since it is difficult to obtain information such as the entry time, number of flights, and flight trajectory of the target in advance, in order to ensure the timeliness and effectiveness of target detection, all sensors need to be in working state all the time to achieve full coverage of the entire detection area. In this process, data is generated only when the sensor tracks and detects the target, and it is sent to the data center in real time for processing. The detection data of each sensor needs to be specially processed, mainly including data message decoding and data processing. Due to the large scale of sensors, a single server cannot complete the real-time processing of all sensor data, so a distributed system is used to solve the resource usage problem of large-scale sensors. In terms of resource management, the current practice is mostly to use the method of pre-allocating resources, that is, to allocate sufficient processing resources to each sensor in advance and keep the processing software in working state. The advantage of this approach is that when the system receives detection data from any sensor at any time, it can complete the processing in a timely manner. However, this approach also has obvious disadvantages, mainly including the following three aspects:
[0003] (1) Low resource utilization
[0004] The resources here mainly include processing resources, storage resources and network resources. Storage resources include memory resources and disk resources. Since disk resources are relatively abundant and the sensor processing program has a low demand for them, this patent focuses on the utilization of memory resources. Since the sensor processing system does not belong to a high-throughput system, the use of network resources does not need to be considered. Therefore, the resource utilization described in this article is mainly for the utilization of processing resources and memory resources. The existing resource allocation method will cause the utilization of these two resources to be low at the same time. The reasons are mainly from three aspects:
[0005] First, in general, the number of flying targets in the monitoring area per unit time will not reach saturation, and its flight path will not pass through the detection area of all sensors, so the ratio of the number of sensors involved in tracking it to the total number of sensors is low. In addition, the detection area of each sensor is limited. When the target flies out of its detection area, the sensor can no longer continue to track the target and generate corresponding data. Therefore, during the flight of the target, only a few sensors track it. Therefore, this way of pre-allocating resources will lead to low resource utilization of distributed systems.
[0006] Second, in order to ensure sufficient processing resources, the amount of resources allocated to each sensor is greater than the amount of resources used in its general processing process. This is because it is difficult to accurately assess the amount of resources required by a sensor processing program. Therefore, in order to avoid insufficient resources during its operation, it has to allocate more resources than the actual application needs, resulting in a waste of resources. This situation is called over-allocation of resources.
[0007] Third, when the remaining resources of a server are insufficient to accommodate any sensor processing program, the processing program can only be deployed on a new server, and these remaining resources will further reduce resource utilization, which is called resource fragmentation. As long as either the processing resources or the memory resources are insufficient to accommodate any sensor processing program, resource fragmentation will occur.
[0008] (2) High system complexity
[0009] The high complexity of the system mainly comes from two aspects: scale and maintenance. In terms of scale, due to the low resource utilization, deploying all sensor processing programs on the system requires more hardware devices, resulting in an increase in the distributed scale. In terms of maintenance, when the sensor processing program is updated, the resources it occupies need to be re-evaluated. If the required amount of resources increases compared to before the update, the amount of resources on the server where it is located needs to be re-evaluated. If the server resources are not enough to accommodate the existing sensor processing programs deployed on it, these programs need to be redeployed.
[0010] (3) Low system energy efficiency ratio
[0011] The system energy efficiency ratio refers to the energy consumption of the system when completing the same amount of tasks. Generally speaking, the higher the energy efficiency ratio, the less electricity is required to complete the same task. The system energy efficiency ratio is positively correlated with the resource utilization rate of the system, that is, the higher the resource utilization rate, the higher the energy efficiency ratio, and vice versa. Therefore, the current resource allocation method makes the energy efficiency of the entire system relatively low.
[0012] (4) High operating costs
[0013] The high system cost mainly comes from two aspects: initial investment cost and later operation and maintenance cost. In terms of initial investment cost, due to the large scale of the system, more hardware purchase costs are required. In terms of later maintenance, on the one hand, due to the high complexity of the system, more operation and maintenance costs are required; on the other hand, the lower energy efficiency ratio will also result in higher energy consumption costs. Summary of the invention
[0014] The purpose of the present invention is to provide a large-scale sensor real-time distributed processing dynamic resource scheduling method, which solves the problems of low resource utilization and energy efficiency ratio, high system complexity and total cost in the existing sensor center-level distributed processing resource scheduling method in the process of monitoring large-area airspace.
[0015] To achieve the above object, the technical solution of the present invention is as follows:
[0016] A large-scale sensor center-level real-time processing distributed resource scheduling method, the method comprising the following steps:
[0017] Step 1: Establish a management server cluster to manage the resources of all servers. The server cluster consists of a main server and several backup servers.
[0018] Step 2: According to the current working status of each server, all servers are divided into working servers and dormant servers;
[0019] Step 3: Count the processing resource and memory resource utilization of each working server once a second, denoted as SPR t and SMR t , the maximum values of processing resource and memory resource utilization within the threshold time are counted every certain threshold time, and are recorded as SPR t-max and SMR t-max , the current moment max(SPR t-max , SMR t-max ) as the current workload rate WLR of the server t , according to the value, the working server is dynamically divided into an idle server, a fully loaded server or a half-loaded server, and its corresponding working status is idle, fully loaded or half-loaded;
[0020] Step 4: Create a globally unique ID number for each sensor, and then establish a mapping relationship between the sensor ID number and the sensor data message identifier, the sensor data message decoding module and the sensor data processing module;
[0021] Step 5: Obtain the processing resources and memory resources required for each sensor data processing, denoted as SenP id and SenM id , where id is the ID number of the sensor, a sensor resource requirement table is established in the management server cluster, and the SenP id and SenM id Record in this table;
[0022] Step 6: Establish a data receiving server cluster and a data processing server cluster, where the data processing server cluster is a dynamic server cluster; establish a data receiving server cluster table and a data processing server cluster table in the management server, respectively, and record the servers belonging to the two clusters, and the management server cluster maintains these two tables; the data receiving server table cluster mainly records the IP addresses of the servers, and the data processing server cluster table mainly records the IP addresses of the servers and the corresponding sensor ID numbers on each server;
[0023] Step 7: Loading a data receiving module, a data acquisition module, a data storage module, a data checking module and a data distribution module into the data receiving server cluster, which are used for real-time reception, acquisition, storage, checking and distribution of multi-sensor data messages respectively;
[0024] Step 8: The data receiving module receives the data messages sent by the multiple sensors in real time and stores them in the dynamic data buffer area of the data receiving server. This step continues until the data receiving activity stops;
[0025] Step 9: The data acquisition module asynchronously acquires the data message from the dynamic data buffer area, copies it and sends it to the data storage module and the data inspection module respectively;
[0026] Step 10: The data storage module asynchronously stores the data message into a disk file;
[0027] Step 11: The data check module checks the integrity of the data message and discards the problematic data message; obtains the sensor ID number by using the mapping relationship table and the sensor identifier in the data message;
[0028] Step 12: According to the sensor ID number corresponding to each sensor message data, the data distribution module searches the data processing server cluster table for the processing server corresponding to the sensor. If the search is successful, the data message and the sensor ID number are sent to the corresponding processing server, and then the process goes to step 15;
[0029] Otherwise, select a server from the data processing server cluster as the processing server for the sensor data, add the server to the data processing server cluster table, and proceed to step 13;
[0030] Step 13: Load the corresponding data message decoding module and data processing module on the designated sensor data processing server;
[0031] Step 14: Complete the decoding of the specified sensor data message on the server, and the real-time processing of the decoded data;
[0032] Step 15: Each server in the data processing server cluster periodically checks the working status of each sensor data message decoding module and data processing module inside it. If no new data enters the module within the specified threshold time, the server automatically releases the processing resources related to it;
[0033] Step 16: The management server cluster periodically checks the workload rate of each server in the data processing server cluster through the work status table and the data processing server cluster table. If the workload on a server is cleared, it will be set as an idle server and included in the idle server set. If the number of servers in the idle server set exceeds the threshold, the server will be directly put into sleep and included in the sleep server set.
[0034] Furthermore, in step 3, the method for calculating the current workload rate of the server is as follows:
[0035] Step 1: SPR t-max and SMR t-max Initialized to 0, as shown in formula (7);
[0036]
[0037] Step 2: Calculate SPR every second t and SMR t The value of and update SPR according to (8) t-max and SMR t-max The value of
[0038]
[0039] Among them, SP0 and SM0 are the total processing resources and total memory resources of the server respectively. and The server is at t i The amount of processing and memory resources used at any given moment;
[0040] Step 3: Set the statistical time length threshold as T s , taking t0 as the starting time of this round of statistics, once t i -t0≥T s , then calculate the current WLR according to formula (9): t The value of
[0041] WLR t =max(SPR t-max , SMR t-max ), (t i -t0≥T s ) (9)
[0042] Step 4: Update the value of t0 to t i value, and returns to the first step for the next round of WLR t Statistical calculations.
[0043] Furthermore, in step 3, the method for dynamically classifying the working servers into idle servers, fully loaded servers or half loaded servers according to the numerical value of the current workload rate of the servers is as follows:
[0044] At any time, if the workload rate of a server satisfies formula (10), it is an idle server;
[0045] WLR t =max(SPR max , SMR max )≈0 (10);
[0046] At any time, if the server workload rate meets formula (12), then it is a fully loaded server;
[0047] WLR t =max(SPR max , SMR max )≥0.80 (12);
[0048] At any time, if the workload rate of the server meets formula (14), then it is a half-loaded server;
[0049] WLR t =max(SPR max , SMR max )<0.80 (14).
[0050] Furthermore, in step 3, each working server performs s The workload rate WLR is transmitted to the management server cluster once every threshold time. t And the corresponding SP max and SM maxThe management server updates these three values to the server working status table established in the management server cluster, and updates the current working status of each server according to the server working status conversion relationship in 3.2; the server working status table contains four sub-tables, namely, a dormant server set table, an idle server set table, a half-load server set table and a full-load server set table, which mainly record the IP address of each server including the dormant server, the current workload rate value, the corresponding working status, the total amount of processing resources, the total amount of memory resources, the current processing resource usage and the current memory resource usage.
[0051] Furthermore, in step 4, the method for establishing the globally unique ID number of the sensor is as follows:
[0052] The ID number is expressed as a non-negative integer, and the numbering principles are as follows:
[0053] First, starting from 0, use 32-bit integers for storage in the server;
[0054] Second, if a new sensor is added later, the ID number only needs to be increased by 1 to the original maximum ID number;
[0055] Third, if a sensor leaves, its number will no longer be occupied by other sensors to maintain the consistency of subsequent methods.
[0056] Furthermore, the step eleven further comprises:
[0057] Step 1: Decode the data message header to obtain description information and verification information;
[0058] Step 2: Use the verification information to verify the integrity of the data message. If there is a problem, the message data is discarded, otherwise proceed to the next step;
[0059] Step 3: Obtain sensor identification information and use the above mapping table to obtain its corresponding sensor ID number.
[0060] Furthermore, in step 12, the specific steps of selecting a server from the data processing server cluster as the sensor processing server are as follows:
[0061] Step 1: Get the sensor's data processing resource requirement SenP from the sensor resource requirement table using the sensor ID number id and SenM id ;
[0062] Step 2: Obtain all half-loaded servers in the processing server cluster from the data processing server cluster table and the server work status table, as well as the total processing resources, total memory resources, current processing resource usage, and current memory resource usage of each half-loaded server;
[0063] Step 3: Query the server work status table and data processing server cluster table to obtain all half-load work server information and form a temporary list of half-load work servers, which includes the IP address of each half-load server, the total amount of server processing and memory resources, the currently used processing and memory resources, and the current workload rate;
[0064] Step 4: for each half-loaded server, pre-estimate the change in workload rate after the data processing workload of the sensor;
[0065] The total amount of processing resources and memory resources of the server are recorded as SP0 and SM0 respectively, and the current amount of processing resources and memory resources used are recorded as SP t and SM t , by querying the sensor resource requirement table, we can obtain the processing resources and memory resources required by the sensor, which are recorded as SP s and SM s , the current server workload rate is recorded as WLR t After the server increases the data processing workload of the sensor, the calculation formula for the change in workload rate is shown in formula (16), where SPR t-max and SMR t-max They are the estimated results of server processing resource utilization and memory resource utilization, WLR t is the estimated result of server workload rate, WLR t is the change in workload rate, which is a non-negative number;
[0066]
[0067] Step 5: Remove all WLR's from the temporary list of half-loaded working servers t >0.80 servers;
[0068] Step 6: If the temporary list of half-loaded servers is not empty, select the minimum workload rate change WLR′ t The server is used as the sensor data processing server. If the temporary list of half-loaded working servers is empty, go to step 7;
[0069] Step 7: Query the idle server sub-table in the server working status table, obtain any idle server from it and use it as the data processing working server of the sensor.
[0070] Furthermore, the step thirteen further comprises:
[0071] Step 1: Using the sensor ID, obtain the disk file URL address of the corresponding data message decoding module and data processing module from the mapping table of the management server cluster;
[0072] Step 2: Query the module registry in the server. This table is used to record all sensor data message decoding modules and data processing modules loaded in the server. If the module required for the corresponding sensor processing is not in the server, use the module URL address to copy the module from the management server cluster to the server's disk, and then load it into the memory to start running;
[0073] Step 3: Write the newly loaded module information into the module registry within the server.
[0074] Furthermore, the step fourteen further comprises:
[0075] Step 1: The internal data receiving module of the server receives the sensor data message and sensor ID number sent by the data receiving server cluster and stores it in the dynamic buffer area;
[0076] Step 2: The internal data distribution module of the server uses the mapping relationship between the sensor ID number and the module in the mapping relationship table to take out the data message from the dynamic buffer area and send it to the corresponding data message decoding module for data message decoding;
[0077] Step 3: The decoding result of the data message is then sent by the internal data distribution module to the corresponding data processing module for data processing.
[0078] Furthermore, in step 15, the working status checking method of the sensor data message decoding module and the data processing module is as follows:
[0079] Step 1: Every time data enters the sensor data message decoding module and the data processing module, the module records the current time, which is called the latest data time of the corresponding module, recorded as DT r , this value is maintained by each corresponding module;
[0080] Step 2: Create a module work status table on each server in the data processing server cluster to record the latest data time of each module;
[0081] Step 3: At a fixed time interval, the server collects the DT of the internal module r Value, and update the corresponding data in the table. While updating the data, calculate each module DT r Value and current time T n Time distance T d , the calculation formula is shown in formula (17);
[0082] T d =T n -DT r (17)
[0083] Step 4: If T d If it is greater than the threshold time Tt, the processing and memory resources of the module are released;
[0084] The steps for releasing the processing resources of the sensor data message decoding module and the data processing module are as follows:
[0085] Step 1: Get all sensor ID numbers corresponding to the module from the mapping table;
[0086] Step 2: Delete the data processing correspondence between the server and the sensor ID number from the data processing server cluster;
[0087] Step 3: Delete the registration information of the module in the module registry on the processor;
[0088] Step 4: Uninstall the module from the server, freeing up the processing and memory resources it occupies.
[0089] The advantages of the present invention are: it can dynamically process the sensors actually involved in the target detection process, and dynamically allocate the resources required in the processing process according to the actual situation. In this process, there is no need to allocate servers and related processing and storage resources to any sensor processing program in advance. When a sensor data is processed, the system recycles the resources it occupies. If all resources on a server are recycled, the system recycles the server. This real-time distributed dynamic resource scheduling method can not only avoid the occurrence of excessive resource allocation, but also further reduce resource fragmentation, thereby effectively improving the utilization rate of the entire system. On this basis, the system complexity is further reduced, the initial investment and later maintenance costs of the system are reduced, and the energy efficiency ratio of the system is greatly improved. In addition, since the processing module required for sensor data processing is dynamically called during the data processing process, the new processing module can be uploaded to the server, and the corresponding old processing module can be replaced before the next dynamic call. This dynamic resource scheduling method can also realize the ability to update the sensor processing program online and add new sensor processing programs.
[0090] The present invention only describes the inventive method and concept in detail, and does not describe related engineering technologies related to distributed systems, such as disaster recovery and load balancing. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] Figure 1 Schematic diagram of the flow of a method for dynamic scheduling of distributed resources for real-time processing at the center level of large-scale sensors. DETAILED DESCRIPTION
[0092] See also Figure 1 The present invention provides a distributed resource scheduling method for real-time processing of large-scale sensor center level, which manages resources of a cluster consisting of a certain number of servers connected to each other through a network. The premise is that the resources of all servers are sufficient to process the data of all sensors in real time, and each server can process data sent by at least one sensor. The specific steps of the method are as follows:
[0093] Step 1: Establish a management server cluster to manage the resources of all servers. The server cluster consists of a main server and several backup servers.
[0094] The management server cluster is a server sub-cluster, which is used to store the management information of all servers and the decoding and processing software module disk files of all sensors. Among them, the management information related to the present invention will be explained in the subsequent steps. The sensor decoding software module disk file is an executable file of the software that performs real-time decoding of the data message sent by the specified sensor; the sensor processing software module disk file is an executable file of the software that performs real-time processing of the data message results of the specified sensor; other servers can automatically copy these software modules from the management server to the local as needed, and then load and run them.
[0095] The management server needs to have high reliability. At the same time, in order to take into account the efficiency of server management, the management server cluster adopts a dual-machine duplex mode, that is, any server can be used as a master server, and the data on each server is synchronized in real time. Since there are many methods for establishing such server clusters, the present invention will not explain the details of their establishment.
[0096] Step 2: According to the current working status of each server, all servers are divided into working servers and dormant servers.
[0097] The division rules of these two types of servers are as follows:
[0098] 2.1 Hibernation Server
[0099] This type of server does not load any software related to sensor data processing and is only used as a backup server resource. In order to save energy, it is put into a dormant state and awakened when needed.
[0100] 2.1 Work Server
[0101] All servers that are not in sleep mode are considered active servers.
[0102] Step 3: Count the processing resource and memory resource utilization of each working server once a second, denoted as SPR t and SMR t At a certain threshold time, the maximum values of the processing resource and memory resource utilization within the threshold time are counted and recorded as SPR respectively. t-max and SMR t-max In the present invention, the threshold time is 5 minutes, and the current time max(SPR t-max , SMR t-max ) as the current workload rate WLR of the server t According to the value, the working servers are dynamically divided into idle servers, fully loaded servers or half loaded servers, and their corresponding working states are idle, fully loaded or half loaded.
[0103] The workload here refers to all applications running on the server, including decoding and processing programs related to sensor data. SPR and SMR are the ratio of processing resources and memory resources used by the actual workload to the total processing and memory resources of the server, and are independent of the processing performance and memory size of the server. According to the current workload rate of the server, the division rules, conversion relationship, workload rate calculation method and workload rate storage method of the above three sets are explained as follows:
[0104] 3.1 Calculate the current workload rate of the server
[0105] Deploy a resource utilization calculation program on each server and calculate the current load rate WLR of the server according to the following method t :
[0106] Step 1: SPR t-max and SMR t-max Initialized to 0, as shown in formula (7);
[0107]
[0108] Step 2: Calculate SPR every second t and SMR t The value of and update SPR according to (8) t-max and SMRt-max Among them, SP0 and SM0 are the total processing resources and total memory resources of the server respectively. and The server is at t i The amount of processing and memory resources used at any given moment.
[0109]
[0110] Step 3: Set the statistical time length threshold as T s , taking t0 as the starting time of this round of statistics, once t i -t0≥T s , then calculate WLR according to formula (9) t The value of .
[0111] WLR t =max(SPR t-max , SMR t-max ), (t i -t0≥T s ) (9)
[0112] Step 4: Update the value of t0 to t i value, and returns to the first step for the next round of WLR t Statistical calculations.
[0113] 3.2 Work Server Classification Rules
[0114] Idle server: There is no sensor data processing program loaded on this server, only the operating system and basic management program. Although these programs will also occupy a certain amount of server processing and memory resources, they are relatively small. The current workload rate can be regarded as close to 0, that is, at any time, the workload rate of the server conforms to formula (10).
[0115] WLR t =max(SPR max , SMR max )≈0 (10)
[0116] All idle servers form the idle server set, which can be regarded as the resource pool of the entire distributed system. The mathematical expression of the idle server set is shown in formula (11), where S is the set of all servers, NS is the idle server set, and server is the server.
[0117] NS={server|WLR t ≈0,server∈S} (11)
[0118] Since it takes a certain amount of time for a server to go from sleep to wake up, when existing server resources are insufficient, the workload can be quickly allocated to idle servers. Therefore, the idle server set must maintain a certain number of servers. When the number of idle servers decreases, it is necessary to wake up the dormant servers in time to supplement the idle server set.
[0119] Fully loaded servers: The workload on these servers has occupied more than 80% of its computing or memory resources. At any time, the workload rate of the server meets the formula (12). The purpose of reserving 20% of resources is to prevent the application on the server from slowing down, freezing, or crashing due to insufficient available resources if the workload fluctuates. At this point, the server can no longer add any workload.
[0120] WLR t =max(SPR max , SMR max )≥0.80 (12)
[0121] The mathematical expression of the fully loaded server set is shown in formula (13), where FS is the fully loaded server set.
[0122] FS={server|WLR t ≥0.80, server∈S} (13)
[0123] Half-loaded server: The workload on this type of server does not exceed 80% of the total computing or memory resource utilization of the server. At any time, the workload rate of the server meets the formula (14). The server can continue to increase its workload.
[0124] WLR t =max(SPR max , SMR max )<0.80 (14)
[0125] The mathematical expression of the half-load server set is shown in formula (15), where HS is the half-load server set.
[0126] HS={server|WLR t <0.80,server∈S} (15)
[0127] 3.3 Server working state conversion relationship
[0128] Including the server's sleep state, there are 4 server working states, namely full load, half load, idle and sleep. According to the working server classification rules in 3.1, when the workload on a server changes, its working state will also change accordingly according to the change of its current workload rate. There are 8 conversion relationships:
[0129] (1) Switching from an idle server to a half-loaded server
[0130] Loading workload onto an idle server, if it satisfies equation (14), converts the server into a half-loaded server.
[0131] (2) Converting from an idle server to a fully loaded server
[0132] Load a workload onto an idle server. If it satisfies equation (12), the server is converted into a fully loaded server.
[0133] (3) Convert idle servers to dormant servers
[0134] Putting an idle server into a dormant state converts the server into a dormant server.
[0135] (4) Convert dormant servers to idle servers
[0136] Waking up a dormant server converts the server into an idle server.
[0137] (5) Switching from a half-loaded server to a fully-loaded server
[0138] Continue to load workload on a half-loaded server. If it satisfies equation (12), the server is converted into a fully-loaded server.
[0139] (6) Switching from a half-loaded server to an idle server
[0140] If all workloads on a half-loaded server are completed and all occupied resources are released so that it satisfies formula (1), it will be converted into an idle server.
[0141] (7) Switching from a fully loaded server to a half loaded server
[0142] If some workloads on a fully loaded server set finish their work and release resources so that it satisfies equation (14), it will be converted into a half-loaded server.
[0143] (8) Converting from a fully loaded server to an idle server
[0144] If all workloads on a fully loaded server finish their work at the same time and release all the resources they occupy so that it satisfies equation (10), it will be directly converted into an idle server.
[0145] 3.4 Workload Rate Storage Method
[0146] A server working status table is established in the management server cluster, which mainly records the IP address, current workload rate value, corresponding working status, total processing resources, total memory resources, current processing resource usage and current memory resource usage of each server including dormant servers.
[0147] In order to improve the efficiency of searching for servers, the server working status table contains four sub-tables, namely, the dormant server set table, the idle server set table, the half-load server set table and the full-load server set table. According to the above-mentioned server working status conversion relationship, the server is updated to different sub-tables in time.
[0148] Each working server is every T s The workload rate WLR is transmitted to the management server cluster once every threshold time. t And the corresponding SP max and SM max The management server updates the three values to the above table and updates the current working status of each server according to the server working status conversion relationship in 3.2.
[0149] Step 4: Establish a globally unique ID number for each sensor, and on this basis establish a mapping relationship between the sensor ID number and the sensor data message identifier, the sensor data message decoding module and the sensor data processing module.
[0150] The following describes the establishment and implementation methods of each mapping relationship:
[0151] 4.1 How to establish a globally unique sensor ID:
[0152] The ID number is expressed as a non-negative integer, and the numbering principles are as follows:
[0153] First, starting from 0, use 32-bit integers for storage in the server;
[0154] Second, if a new sensor is added later, the ID number only needs to be increased by 1 to the original maximum ID number;
[0155] Third, if a sensor leaves, its number will no longer be occupied by other sensors to maintain the consistency of subsequent methods.
[0156] 4.2 Method for establishing the mapping relationship between sensor data identification and sensor ID number:
[0157] The sensor identifier is a binary data in the sensor data message header, which is used to identify the sensor that generates the data. Different identifiers are mapped one-to-one with the sensor ID number, and each data message sensor identifier uniquely determines a sensor ID number.
[0158] 4.3 Method for establishing mapping relationship between sensor data message decoding module and sensor ID number
[0159] The sensor data message decoding module is mainly used to decode the binary sensor data message into a decimal measurement result. This is a one-to-many mapping relationship. Sensors with the same data message format use the same decoding module. Therefore, the same decoding module can correspond to multiple sensor ID numbers. Before the sensor data decoding module is loaded into the server for operation, it is stored in the disk of the management server cluster in the form of an executable program file. Each file has a unique URL address, which can be obtained by the server. Therefore, the mapping relationship is implemented by establishing a corresponding relationship between the sensor ID number and the corresponding sensor data message decoding module URL address.
[0160] 4.4 Method for establishing mapping relationship between sensor data processing module and sensor ID number
[0161] The sensor data processing module is mainly used to process the decoded sensor data message. This is also a one-to-many mapping relationship. The processing process is the same for sensors of the same type. Before being loaded into the server for operation, the sensor data processing module is stored in the disk of the management server cluster in the form of an executable program file. Each file has a unique URL address. The mapping relationship is also implemented by establishing a corresponding relationship between the sensor ID number and the corresponding sensor data processing module URL address.
[0162] 4.5 Implementation method of mapping relationship
[0163] A mapping table is established in the management server, which records the sensor ID number, sensor data identifier, sensor data message decoding module URL address, and sensor data processing module URL address. The server copies the table to the local as needed to improve query efficiency, and automatically updates it regularly to maintain consistency with the mapping table in the management server.
[0164] Step 5: Obtain the processing resources and memory resources required for each sensor data processing through actual operation statistics or simulation operation tests, which are recorded as SenP id and SenM id, where id is the ID number of the sensor, and the number is an approximate number. A sensor resource requirement table is established in the management server cluster, and the SenP id and SenM id Record in this table.
[0165] The processing resources and memory resource usage will fluctuate during the statistical process. The maximum processing resources and memory resource usage during the statistical time is taken as the actual resource demand of the sensor. The sensor resource demand table is mainly used to store the ID number, SenP id and SenM id The statistical result provides a basis for dynamic resource scheduling.
[0166] Step 6: Establish a data receiving server cluster and a data processing server cluster, which are used for sensor data reception and sensor data processing respectively. The data processing server cluster is a dynamic server cluster. The servers belonging to each cluster are recorded in the data receiving server cluster table and the data processing server cluster table respectively, and the management server cluster maintains these two tables.
[0167] The data management server cluster is mainly used to manage server resources, that is, to manage the use of servers. Specifically, all servers are divided into three clusters, namely, the data management server cluster, the data receiving server cluster and the data processing server cluster. The division of each server is determined by the data management server cluster. The present invention dynamically adjusts the number of working servers in the data processing server cluster according to the current task volume, adjusts the servers without workload to the idle server set and chooses whether to sleep according to the situation, thereby forming dynamic scheduling of resources. These are all completed by the management server cluster.
[0168] The functions and establishment methods of the two dynamic server clusters, the data receiving server cluster and the data processing server cluster, are described as follows:
[0169] 6.1 Functions and roles
[0170] The data receiving server cluster consists of a small number of servers and is mainly used to receive and store binary data packets sent by all sensors, analyze the validity of the data, and send the valid data packets to the specified server in the data processing server cluster.
[0171] When the amount of data transmitted by the sensor network is not large, only one server is needed to complete the reception of data packets; when the sensor transmits a large amount of data in real time, in order to ensure the effectiveness of data reception and prevent frame loss in data reception or storage due to excessive data flow, multiple servers are needed to receive the sensor data in network packets, and each server receives the data stream of the corresponding network packet.
[0172] Since the server resources required for the sensor data message receiving process are relatively small, the number of data receiving server clusters is relatively fixed, and the modules loaded on each server do not need to be changed.
[0173] The data processing server cluster is mainly used to complete the decoding and processing of sensor data messages. Since the sensor data processing process occupies relatively more server resources, the required amount of resources is determined by the number of sensors involved in the actual measurement process. Therefore, this part adopts a dynamic resource scheduling method, which can dynamically increase or decrease the number of servers in the cluster according to the actual use of resources. The specific steps of this method are described in detail later.
[0174] 6.2 Establishment Method
[0175] These two clusters are established based on a logical relationship. A data receiving server cluster table and a data processing server cluster table are respectively established in the management server to record the server information to which they belong. The data receiving server cluster table mainly records the IP address of the server to which it belongs. In addition to recording the IP address of the server to which it belongs, the data processing server cluster table also needs to record the corresponding sensor ID number on each server. Its function is to establish a corresponding relationship between the server and the sensor data processing, that is, all data sent by these sensors are processed in real time by this server. One premise of the present invention is that a server can process data sent by one or more sensors at the same time, and the specific number of sensors corresponding to the server is related to its performance.
[0176] Combined with the server working status table in the management server cluster, the workload status of each server can be quickly found. In the initial state of the two clusters, there is only one working server in each server cluster.
[0177] Step 7: Load the data receiving module, data acquisition module, data storage module, data checking module and data distribution module into the data receiving server cluster, which are used for real-time reception, acquisition, storage, checking and distribution of multi-sensor data messages respectively.
[0178] The data receiving server cluster is mainly responsible for the real-time reception, storage, inspection and distribution of multi-sensor data messages. The above functions are mainly completed by the data receiving module, data acquisition module, data storage module, data inspection module and data distribution module loaded by the server. These modules can be deployed on the same server, or the data storage module can be deployed separately on a server dedicated to data storage.
[0179] Step 8: The data receiving module receives the data messages sent by the multiple sensors in real time and stores them in the dynamic data buffer area of the data receiving server. This step continues until the data receiving activity stops.
[0180] The data receiving module is mainly used to receive data messages sent by multiple sensors in real time. In order to effectively improve the efficiency of data reception, the dynamic data buffer is used to decouple data reception from other activities to achieve asynchronous data reception. The dynamic data buffer is implemented using a bidirectional linked list. The real-time received data is directly added to the head of the bidirectional linked list. The size of the data cache is related to the memory size of the server.
[0181] If the amount of real-time data packets sent by multiple sensors is within the network receiving capacity of one server, only one server is needed to receive the data packets in real time; otherwise, multiple servers are needed to receive the data packets in network packets.
[0182] Step 9: The data acquisition module asynchronously acquires the data message from the dynamic data buffer area, copies it and sends it to the data storage module and the data checking module respectively.
[0183] The dynamic data buffer is constructed by a two-way linked list. The data acquisition module acquires data from the tail of the two-way linked list and releases the memory resources occupied by the data in the buffer after acquisition. The advantage of this method is that data synchronization is only required when there is only one data left in the two-way linked list, which has good asynchronicity.
[0184] Step 10: The data storage module asynchronously stores the data message in a disk file.
[0185] The data storage module stores the received real-time message data to disk files for later analysis and data reproduction. The data acquisition module sends the data to the data storage module, which also sets a buffer area like the data receiving module to solve the speed difference between memory and disk.
[0186] Step 11: The data checking module checks the integrity of the data message and discards the problematic data message; uses the mapping relationship table and the sensor identifier in the data message to obtain the sensor ID number.
[0187] The data check module mainly decodes the information in the data message header. The data message consists of a data message header and data content. The data content is the sensor measurement data result. The data message header is used to provide description information and verification information of the data content. The description information includes sensor identification, source, destination and other information. The verification information includes data message length, check code and other information. The data message header format under the same transmission protocol is the same. The processing steps of the data check module are as follows:
[0188] Step 1: Decode the data message header to obtain description information and verification information;
[0189] Step 2: Use the verification information to verify the integrity of the data message. If there is a problem, the message data is discarded, otherwise proceed to the next step;
[0190] Step 3: Obtain sensor identification information and use the above mapping table to obtain its corresponding sensor ID number.
[0191] Step 12: According to the sensor ID number corresponding to each sensor message data, the data distribution module searches the data processing server cluster table for the processing server corresponding to the sensor. If the search is successful, the data message and the sensor ID number are sent to the corresponding processing server, and then the process goes to step 15;
[0192] Otherwise, a server is selected from the data processing server cluster as the processing server for the sensor data, the server is added to the data processing server cluster table, and the process goes to step thirteen.
[0193] This step is the resource expansion process of the data processing server cluster.
[0194] The specific steps for selecting a server from the data processing server cluster as the sensor processing server are as follows:
[0195] Step 1: Get the sensor's data processing resource requirement SenP from the sensor resource requirement table using the sensor ID number id and SenM id ;
[0196] Step 2: Obtain all half-loaded servers in the processing server cluster from the data processing server cluster table and the server work status table, as well as the total processing resources, total memory resources, current processing resource usage, and current memory resource usage of each half-loaded server;
[0197] Step 3: Query the server work status table and data processing server cluster table to obtain all half-load work server information and form a temporary list of half-load work servers, which includes the IP address of each half-load server, the total amount of server processing and memory resources, the currently used processing and memory resources, and the current workload rate;
[0198] Step 4: for each half-loaded server, estimate the change in workload rate after the data processing workload of the sensor in advance;
[0199] The calculation method of workload rate change is as follows:
[0200] The total amount of processing resources and memory resources of the server are recorded as SP0 and SM0 respectively, and the current amount of processing resources and memory resources used are recorded as SP t and SM t , by querying the sensor resource requirement table, we can obtain the processing resources and memory resources required by the sensor, which are recorded as SP s and SM s , the current server workload rate is recorded as WLR t After the server increases the data processing workload of the sensor, the calculation formula for the change in workload rate is shown in formula (16). t-max and SMR′ t-max They are the estimation results of server processing resource utilization and memory resource utilization, WLR′ t is the estimated result of server workload ratio, WLR″ t is the change in workload rate, which is a non-negative number.
[0201]
[0202] Step 5: Remove all WLR's from the temporary list of half-loaded working servers t >0.80 servers;
[0203] Step 6: If the temporary list of half-loaded servers is not empty, select the minimum workload rate change WLR′ t The server is used as the sensor data processing server. If the temporary list of half-loaded working servers is empty, go to step 7;
[0204] Step 7: Query the idle server sub-table in the server working status table, obtain any idle server from it and use it as the data processing working server of the sensor.
[0205] Step 13: On the designated sensor data processing work server, load the corresponding data message decoding module and data processing module.
[0206] The specific loading method is as follows:
[0207] Step 1: Using the sensor ID, obtain the disk file URL address of the corresponding data message decoding module and data processing module from the mapping table of the management server cluster;
[0208] Step 2: Query the module registry in the server. This table is used to record all sensor data message decoding modules and data processing modules loaded in the server. If the module required for the corresponding sensor processing is not in the server, use the module URL address to copy the module from the management server cluster to the server's disk, and then load it into the memory to start running;
[0209] Step 3: Write the newly loaded module information into the module registry within the server.
[0210] Step 14: Complete the decoding of the specified sensor data message on the server, and the real-time processing of the decoded data.
[0211] The steps for real-time sensor data processing in the server are as follows:
[0212] Step 1: The internal data receiving module of the server receives the sensor data message and sensor ID number sent by the data receiving server cluster, and stores them in the dynamic buffer area. The dynamic buffer area is constructed in the same way as the dynamic buffer area described above.
[0213] Step 2: The internal data distribution module of the server uses the mapping relationship between the sensor ID number and the module in the mapping relationship table to take out the data message from the dynamic buffer area and send it to the corresponding data message decoding module for data message decoding;
[0214] Step 3: The decoding result of the data message is then sent by the internal data distribution module to the corresponding data processing module for data processing.
[0215] Step 15: Each server in the data processing server cluster periodically checks the working status of each sensor data message decoding module and data processing module within it. If no new data enters the module within the specified threshold time, the server automatically releases the processing resources related to it. In the present invention, the period is set to 5 minutes.
[0216] The working status checking method of the sensor data message decoding module and the data processing module is:
[0217] Step 1: Every time data enters the above two modules, the module to which it belongs records the current time, which is called the latest data time of the corresponding module, recorded as DT r , this value is maintained by each corresponding module;
[0218] Step 2: For easy reference, a module work status table is established on each server in the data processing server cluster to record the latest data time of each module;
[0219] Step 3: At a fixed time interval, the server collects the DT of the internal module r The value is calculated and the corresponding data in the table is updated. In the present invention, the fixed time interval is 5 minutes. While updating the data, each module DT is calculated. r Value and current time T n Time distance T d , the calculation formula is shown in formula (17).
[0220] T d =T n -DT r (17)
[0221] Step 4: If T d If it is greater than the threshold time Tt, the processing and memory resources of the module are released. In the present invention, Tt is 5 minutes.
[0222] 15.2 The steps for releasing the processing resources of the sensor data message decoding module and the data processing module are as follows:
[0223] Step 1: Get all sensor ID numbers corresponding to the module from the mapping table;
[0224] Step 2: Delete the data processing correspondence between the server and the sensor ID number from the data processing server cluster;
[0225] Step 3: Delete the registration information of the module in the module registry on the processor;
[0226] Step 4: Uninstall the module from the server, freeing up the processing and memory resources it occupies.
[0227] Step 16: The management server cluster periodically checks the workload rate of each server in the data processing server cluster through the work status table and the data processing server cluster table. If the workload on a server is cleared, it is set as an idle server and included in the idle server set. If the number of servers in the idle server set exceeds the threshold, the server is directly put into sleep mode and included in the sleep server set. In the present invention, the check period is set to 5 minutes.
[0228] If the server is included in the dormant server set, it needs to be deleted from the data processing server cluster table. This step is the resource shrinking process of the data processing server cluster.
[0229] Steps 8 to 16 are kept in a continuous loop to ensure the timely capture, tracking and processing of incoming targets.
[0230] The large-scale sensor real-time distributed processing dynamic resource scheduling method proposed in the present invention can dynamically process the sensors actually involved in the target detection process, and dynamically allocate the resources required in the processing process according to the actual situation. In this process, there is no need to allocate servers and related processing and storage resources to any sensor processing program in advance. When the data of a sensor is processed, the system recycles the resources it occupies. If all resources on a server are recycled, the system recycles the server. This real-time distributed dynamic resource scheduling method can not only avoid the occurrence of excessive resource allocation, but also further reduce resource fragmentation, thereby effectively improving the utilization rate of the entire system. On this basis, the system complexity is further reduced, the initial investment and later maintenance costs of the system are reduced, and the energy efficiency ratio of the system is greatly improved. In addition, this dynamic resource scheduling method can also realize the ability to update sensor processing programs online and add new sensor processing programs.
Claims
1. A large-scale sensor center-level real-time processing distributed resource scheduling method, characterized in that: The method comprises the following steps: Step 1: Establish a management server cluster to manage the resources of all servers. The server cluster consists of a main server and several backup servers. Step 2: According to the current working status of each server, all servers are divided into working servers and dormant servers; Step 3: Count the processing resource and memory resource utilization of each working server once a second, denoted as SPR t and SMR t , the maximum values of processing resource and memory resource utilization within the threshold time are counted every certain threshold time, and are recorded as SPR t-max and SMR t-max , the current moment max(SPR t-max , SMR t-max ) as the current workload rate WLR of the server t , according to the current workload rate WLR t The size of the work server is dynamically divided into idle servers, fully loaded servers or half loaded servers, and the corresponding working status is idle, fully loaded or half loaded; Step 4: Create a globally unique ID number for each sensor, and then establish a mapping relationship between the sensor ID number and the sensor data message identifier, the sensor data message decoding module and the sensor data processing module; Step 5: Obtain the processing resources and memory resources required for each sensor data processing, denoted as SenP id and SenM id , where id is the ID number of the sensor, a sensor resource requirement table is established in the management server cluster, and the SenP id and SenM id Record in this table; Step 6: Establish a data receiving server cluster and a data processing server cluster, where the data processing server cluster is a dynamic server cluster; establish a data receiving server cluster table and a data processing server cluster table in the management server, respectively, record the servers belonging to the two clusters, and the management server cluster maintains the two tables; The data receiving server cluster table records the IP address of the server to which it belongs, and the data processing server cluster table records the IP address of the server to which it belongs and the corresponding sensor ID number on each server; Step 7: Loading a data receiving module, a data acquisition module, a data storage module, a data checking module and a data distribution module into the data receiving server cluster, which are used for real-time reception, acquisition, storage, checking and distribution of multi-sensor data messages respectively; Step 8: The data receiving module receives the data messages sent by the multiple sensors in real time and stores them in the dynamic data buffer area of the data receiving server. This step continues until the data receiving activity stops; Step 9: The data acquisition module asynchronously acquires the data message from the dynamic data buffer area, copies it and sends it to the data storage module and the data inspection module respectively; Step 10: The data storage module asynchronously stores the data message into a disk file; Step 11: The data check module checks the integrity of the data message and discards the problematic data message; obtains the sensor ID number by using the mapping relationship table and the sensor identifier in the data message; Step 12: According to the sensor ID number corresponding to each sensor message data, the data distribution module searches the data processing server cluster table for the processing server corresponding to the sensor. If the search is successful, the data message and the sensor ID number are sent to the corresponding processing server, and then the process goes to step 15; Otherwise, select a server from the data processing server cluster as the processing server for the sensor data, add the server to the data processing server cluster table, and proceed to step 13; Step 13: Load the corresponding data message decoding module and data processing module on the designated sensor data processing server; Step 14: Complete the decoding of the specified sensor data message on the server, and the real-time processing of the decoded data; Step 15: Each server in the data processing server cluster periodically checks the working status of each sensor data message decoding module and data processing module inside it. If no new data enters the module within the specified threshold time, the server automatically releases the processing resources related to it; Step 16: The management server cluster periodically checks the workload rate of each server in the data processing server cluster through the work status table and the data processing server cluster table. If the workload on a server is cleared, it will be set as an idle server and included in the idle server set. If the number of servers in the idle server set exceeds the threshold, the server will be directly put into sleep and included in the sleep server set.
2. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 1, characterized in that: In step 3, the method for calculating the current workload rate of the server is as follows: Step 1: SPR t-max and SMR t-max Initialized to 0, as shown in formula (7); Step 2: Calculate SPR every second t and SMR t The value of and update SPR according to formula (8) t-max and SMR t-max The value of Among them, SP0 and SM0 are the total processing resources and total memory resources of the server respectively. and The server is at t i The amount of processing and memory resources used at any given moment; Step 3: Set the statistical time length threshold as T s , taking t0 as the starting time of this round of statistics, once t i -t0≥T s , then calculate the current WLR according to formula (9): t The value of WLR t =max(SPR t-max ,SMR t-max ),(t i -t0≥T s ) (9) Step 4: Update the value of t0 to t i value, and returns to the first step for the next round of WLR t Statistical calculations.
3. The large-scale sensor center-level real-time processing distributed resource scheduling method as claimed in claim 2, characterized in that: In step 3, the method for dynamically classifying the working servers into idle servers, fully loaded servers or half loaded servers according to the numerical value of the current workload rate of the servers is as follows: At any time, if the workload rate of a server satisfies formula (10), it is an idle server; WLR t =max(SPR max ,SMR max )≈0 (10); At any time, if the server workload rate meets formula (12), then it is a fully loaded server; WLR t =max(SPR max ,SMR max )≥0.80 (12); At any time, if the workload rate of the server meets formula (14), then it is a half-loaded server; WLR t =max(SPR max ,SMR max )<0.80 (14)。 4. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 3, characterized in that: In step 3, each working server performs s The workload rate WLR is transmitted to the management server cluster once every threshold time. t And the corresponding SP max and SM max The management server updates these three values to the server working status table established in the management server cluster, and updates the current working status of each server according to the server working status conversion relationship in 3.2; the server working status table contains four sub-tables, namely, a dormant server set table, an idle server set table, a half-load server set table and a full-load server set table, which record the IP address of each server including the dormant server, the current workload rate value, the corresponding working status, the total amount of processing resources, the total amount of memory resources, the current processing resource usage and the current memory resource usage.
5. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 4, characterized in that: In step 4, the method for establishing the globally unique ID number of the sensor is as follows: The ID number is expressed as a non-negative integer, and the numbering principles are as follows: First, starting from 0, use 32-bit integers for storage in the server; Second, if a new sensor is added later, the ID number only needs to be increased by 1 to the original maximum ID number; Third, if a sensor leaves, its number will no longer be occupied by other sensors to maintain the consistency of subsequent methods.
6. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 5, characterized in that: The step eleven further comprises: Step 1: Decode the data message header to obtain description information and verification information; Step 2: Use the verification information to verify the integrity of the data message. If there is a problem, the message data is discarded, otherwise proceed to the next step; Step 3: Obtain sensor identification information and use the above mapping table to obtain its corresponding sensor ID number.
7. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 6, characterized in that: In step 12, the specific steps of selecting a server from the data processing server cluster as the sensor processing server are as follows: Step 1: Get the sensor's data processing resource requirement SenP from the sensor resource requirement table using the sensor ID number id and SenM id ; Step 2: Obtain all half-loaded servers in the processing server cluster from the data processing server cluster table and the server work status table, as well as the total processing resources, total memory resources, current processing resource usage, and current memory resource usage of each half-loaded server; Step 3: Query the server work status table and data processing server cluster table to obtain all half-load work server information and form a temporary list of half-load work servers, which includes the IP address of each half-load server, the total amount of server processing and memory resources, the currently used processing and memory resources, and the current workload rate; Step 4: for each half-loaded server, pre-estimate the change in workload rate after the data processing workload of the sensor; The total amount of processing resources and memory resources of the server are recorded as SP0 and SM0 respectively, and the current amount of processing resources and memory resources used are recorded as SP t and SM t , by querying the sensor resource requirement table, we can obtain the processing resources and memory resources required by the sensor, which are recorded as SP s and SM s , the current server workload rate is recorded as WLR t ; After the server increases the data processing workload of the sensor, the calculation formula for the change in workload rate is shown in formula (16), where SPR′ t-max and SMR′ t-max They are the estimation results of server processing resource utilization and memory resource utilization, WLR′ t is the estimated result of server workload ratio, WLR″ t is the change in workload rate, the change in workload rate WLR″ t is a non-negative number; Step 5: Remove all WLR's from the temporary list of half-loaded working servers t >0.80 servers; Step 6: If the temporary list of half-loaded servers is not empty, select the minimum workload rate change WLR′ t The server is used as the data processing server of the sensor. If the temporary list of half-load working servers is empty, the seventh step is entered; Step 7: Query the idle server sub-table in the server working status table, obtain any idle server from it and use it as the data processing working server of the sensor.
8. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 7, characterized in that: The step thirteen further comprises: Step 1: Using the sensor ID, obtain the disk file URL address of the corresponding data message decoding module and data processing module from the mapping table of the management server cluster; Step 2: Query the module registry in the server. The module registry is used to record all sensor data message decoding modules and data processing modules loaded in the server. If the module required for the corresponding sensor processing is not in the server, use the module URL address to copy the module from the management server cluster to the server's disk, and then load it into the memory to start running; Step 3: Write the newly loaded module information into the module registry within the server.
9. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 8, characterized in that: The step fourteen further comprises: Step 1: The internal data receiving module of the server receives the sensor data message and sensor ID number sent by the data receiving server cluster and stores it in the dynamic buffer area; Step 2: The internal data distribution module of the server uses the mapping relationship between the sensor ID number and the module in the mapping relationship table to take out the data message from the dynamic buffer area and send it to the corresponding data message decoding module for data message decoding; Step 3: The decoding result of the data message is then sent by the internal data distribution module to the corresponding data processing module for data processing.
10. The method for real-time distributed resource scheduling for large-scale sensor center level processing as claimed in claim 9, characterized in that: In step 15, the working status checking method of the sensor data message decoding module and the data processing module is as follows: Step 1: Every time data enters the sensor data message decoding module and the data processing module, the module records the current time, which is called the latest data time of the corresponding module, recorded as DT r , DT r Maintained by each corresponding module; Step 2: Create a module work status table on each server in the data processing server cluster to record the latest data time of each module; Step 3: At a fixed time interval, the server collects the DT of the internal module r Value, and update the corresponding data in the table. While updating the data, calculate each module DT r Value and current time T n Time distance T d , the calculation formula is shown in formula (17); T d =T n -DT r (17) Step 4: If T d If it is greater than the threshold time Tt, the processing and memory resources of the module are released; The steps for releasing the processing resources of the sensor data message decoding module and the data processing module are as follows: Step 1: Get all sensor ID numbers corresponding to the module from the mapping table; Step 2: Delete the data processing correspondence between the server and the sensor ID number from the data processing server cluster; Step 3: Delete the registration information of the module in the module registry on the processor; Step 4: Uninstall the module from the server, freeing up the processing and memory resources it occupies.
