A flink-based message scheduling optimization method and system

Abstract

The application relates to a Flink-based message scheduling optimization method and system, and belongs to the technical field of mobile communication. The method comprises the following steps: a first device sends a RESET message to a second device, wherein the RESET message comprises identification information of the first device and a system resource identifier, the system resource identifier is associated with a terminal device; the second device sends a request instruction to a Flink engine, wherein the request instruction comprises the RESET message and IP information; the Flink engine maps all processes into a task manager TaskManager, the TaskManager comprises a plurality of process slots TaskSlot; the Flink engine maps the RESET message into a subtask SubTask, and calls an intelligent distribution model to realize uniform distribution of the subtask SubTask on the TaskManager. The method can reduce the task overhead and save system resources.

Description

Technical Field

[0001] This invention belongs to the field of mobile communication technology, and in particular relates to a message scheduling optimization method and system based on Flink. Background Technology

[0002] The 3GPP mobile communication standard defines a RESET procedure between the base station and the core network. The purpose of this procedure is to notify the base station to initialize the system resources occupied by the UE and release the network resources associated with the UE when a system resource failure in the core network affects the UE. Conversely, when a system resource failure in the base station affects the UE, the RESET procedure is used to notify the core network to initialize the system resources occupied by the UE and release the network resources associated with the UE.

[0003] like Figure 1 In a 5G network, the RESET process initiated by the core network is as follows: When the system resources of the core network device AMF fail (such as process or board reset), resulting in the loss of information of some or all UEs, the core network sends a RESET message to the base station, carrying the identifier of the affected UE. After receiving the RESET message, the base station releases the system and network resources occupied by the affected UEs. After the resources are released, the base station replies with a RESET ACKNOWLEDGE message to the core network.

[0004] In existing technologies, the upper limit for Reset message transmission is generally 1000. If this number is exceeded, the message will wait in the cache. Too many concurrent messages cause network latency and excessive resource overhead. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a message scheduling optimization method and system based on Flink, which enables Reset messages (SubTasks) to share the TaskManager process, TCP connections and heartbeat messages, and some datasets and data structures, thereby reducing task overhead and saving system resources.

[0006] According to one aspect of the present invention, a message scheduling optimization method based on Flink is provided, the method comprising the following steps:

[0007] S1: The first device sends a RESET message to the second device. The RESET message includes the identification information of the first device and a system resource identifier. The system resource identifier is associated with the terminal device.

[0008] S2: The second device sends a request instruction to the Flink engine, the request instruction including the RESET message and IP information;

[0009] S3: The Flink engine maps all processes to a TaskManager, which includes multiple process slots (TaskSlot);

[0010] The Flink engine maps the RESET message to a SubTask and calls the intelligent allocation model to achieve a uniform distribution of the SubTask on the TaskManager.

[0011] Preferably, the first device is a 5G access network device and the second device is a 5G core network device; or, the first device is a 5G core network device and the second device is a 5G access network device.

[0012] Preferably, the Flink engine's invocation of the intelligent allocation model to achieve the uniform distribution of the subtasks on the TaskManager includes:

[0013] Access information about all subtasks mapped to RESET messages deployed on TaskManager;

[0014] The difference formula is used to perform interpolation on different SubTasks with uneven file sizes, resulting in processed SubTask data; the difference formula is:

[0015]

[0016] Where x represents time, y represents the current value, and i represents the sequence number;

[0017] The processed SubTask data is then placed into different TaskSlots in sequence to achieve a uniform distribution of SubTasks on the TaskManager.

[0018] Preferably, the method further includes:

[0019] The Flink engine maps and schedules the RESET messages based on the thread pool's occupancy.

[0020] Preferably, the Flink engine's mapping and scheduling of the RESET message based on the thread pool's occupancy includes:

[0021] Add the TaskManager to the process pool;

[0022] Each Reset message is mapped to an operator, and operators of multiple Reset messages of the same type form an operation chain, which is associated with the TaskManager.

[0023] Calculate the thread pool load metric, allocate threads based on the thread pool load metric, and map and schedule the computation chains with the allocated threads. Computation chains with higher weights prioritize data transfer-level operations from threads with lower thread occupancy rates. The thread pool load metric is:

[0024]

[0025] Where N is the number of worker threads in the thread pool at runtime, N max It is the maximum number of threads set, T cur T is the number of tasks in the current data collection time window. pre Q is the number of tasks in the previous data collection window, Q is the size of the task buffer queue, and ξ1, ξ2, and ξ3 are weighting coefficients.

[0026] According to another aspect of the present invention, the present invention also provides a message scheduling optimization system based on Flink, the system comprising a first device, a second device, and a Flink engine;

[0027] The first device sends a RESET message to the second device. The RESET message includes the identification information of the first device and a system resource identifier, and the system resource identifier is associated with the terminal device.

[0028] The second device sends a request instruction to the Flink engine, the request instruction including the RESET message and IP information;

[0029] The Flink engine maps all processes to a TaskManager, which includes multiple process slots (TaskSlot).

[0030] The Flink engine maps the RESET message to a SubTask and calls the intelligent allocation model to achieve a uniform distribution of the SubTask on the TaskManager.

[0031] Preferably, the first device is a 5G access network device and the second device is a 5G core network device; or, the first device is a 5G core network device and the second device is a 5G access network device.

[0032] Preferably, the Flink engine's invocation of the intelligent allocation model to achieve the uniform distribution of the subtasks on the TaskManager includes:

[0033] Access information about all subtasks mapped to RESET messages deployed on TaskManager;

[0034] The difference formula is used to perform interpolation on different SubTasks with uneven file sizes, resulting in processed SubTask data; the difference formula is:

[0035]

[0036] Where x represents time, y represents the current value, and i represents the sequence number;

[0037] The processed SubTask data is then placed into different TaskSlots in sequence to achieve a uniform distribution of SubTasks on the TaskManager.

[0038] Preferably, the Flink engine maps and schedules the RESET message based on the thread pool's occupancy.

[0039] Preferably, the Flink engine's mapping and scheduling of the RESET message based on the thread pool's occupancy includes:

[0040] Add the TaskManager to the process pool;

[0041] Each Reset message is mapped to an operator, and operators of multiple Reset messages of the same type form an operation chain, which is associated with the TaskManager.

[0042] Calculate the thread pool load metric, allocate threads based on the thread pool load metric, and map and schedule the computation chains with the allocated threads. Computation chains with higher weights prioritize data transfer-level operations from threads with lower thread occupancy rates. The thread pool load metric is:

[0043]

[0044] Where N is the number of worker threads in the thread pool at runtime, N max It is the maximum number of threads set, T cur T is the number of tasks in the current data collection time window. pre Q is the number of tasks in the previous data collection window, Q is the size of the task buffer queue, and ξ1, ξ2, and ξ3 are weighting coefficients.

[0045] Beneficial Effects: This invention maps all processes, including those within the cache, to a single TaskManager. Each TaskManager can be divided into multiple process slots (TaskSlots). By dynamically adjusting the number of TaskSlots, users can define which Reset messages (SubTasks) can be isolated from each other. Simultaneously, by using shared slots, the resources of the slots are fully utilized, allowing more costly SubTasks to be evenly distributed across TaskManagers. This enables Reset messages (SubTasks) to share the same TaskManager process, TCP connections, heartbeat messages, and even some datasets and data structures, thereby reducing task overhead and saving system resources.

[0046] The features and advantages of the present invention will become clear from the following accompanying drawings and a detailed description of specific embodiments thereof. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the RESET process initiated by 5G network core network equipment in existing technology;

[0048] Figure 2 This is a schematic diagram of the message scheduling optimization method based on Flink of the present invention;

[0049] Figures 3(a) and 3(b) are schematic diagrams of the RESET message format after the introduction of Resource ID. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1

[0052] Figure 2 This is a schematic diagram of the message scheduling optimization method based on Flink according to the present invention. Figure 2 As shown, this invention provides a message scheduling optimization method based on Flink, the method comprising the following steps:

[0053] S1: The first device sends a RESET message to the second device. The RESET message includes the identification information of the first device and a system resource identifier, and the system resource identifier is associated with the terminal device.

[0054] Preferably, the first device is a 5G access network device and the second device is a 5G core network device; or, the first device is a 5G core network device and the second device is a 5G access network device.

[0055] Specifically, in a 5G network, when a network failure occurs between the core network and the base station, the faulty party will proactively initiate a RESET message carrying its own identifier to notify the other party. The RESET message includes a System Resource Identifier (Resource ID), which is associated with the terminal device. The RESET message format after introducing the Resource ID is shown in Figures 3(a) and 3(b).

[0056] S2: The second device sends a request instruction to the Flink engine, the request instruction including the RESET message and IP information.

[0057] Specifically, after receiving the RESET message, the non-faulty party sends a data request instruction containing the RESET message and IP information to the Flink engine through the program.

[0058] S3: The Flink engine maps all processes to a TaskManager, which includes multiple process slots (TaskSlot); the Flink engine maps the RESET message to a SubTask and calls the intelligent allocation model to achieve a uniform distribution of the SubTask on the TaskManager.

[0059] Specifically, the Flink engine maps all processes, including those in the cache, to a TaskManager. Each TaskManager can be divided into multiple Task Slots (process slots), with the default number based on the number of CPU cores.

[0060] After receiving a RESET message containing a (Resource ID) request, Flink uses the TaskManager function to map the RESET message containing the (Resource ID) to a subtask by combining the connection relationship between TaskSlots (process slots) and resources. Then, using the intelligent allocator in conjunction with Flink's TaskSlots (process slots), the subtask is evenly placed into the shared slots, thereby making full use of the resources of TaskSlots (process slots) and ensuring that larger subtasks are evenly distributed across the TaskManager.

[0061] Preferably, the Flink engine's invocation of the intelligent allocation model to achieve the uniform distribution of the subtasks on the TaskManager includes:

[0062] Access information about all subtasks mapped to RESET messages deployed on TaskManager;

[0063] The difference formula is used to perform interpolation on different SubTasks with uneven file sizes, resulting in processed SubTask data; the difference formula is:

[0064]

[0065] Where x represents time, y represents the current value, and i represents the sequence number;

[0066] The processed SubTask data is then placed into different TaskSlots in sequence to achieve a uniform distribution of SubTasks on the TaskManager.

[0067] Specifically, since the Flink SQL engine uses code generation technology (CodeGen) to generate the Transformer algorithm and compile the code into an executable JobGraph, this embodiment uses Transformer to continuously analyze data and resource usage data over a period of time. Based on the data sequence, it discovers potential characteristics before memory capacity anomalies, making memory capacity warnings more accurate.

[0068] First, access information such as all subtasks mapped to RESET messages deployed on TaskManager.

[0069] Secondly, to adapt to model processing, interpolation is performed on files from different SubTasks with uneven file sizes. A quadratic interpolation method is used, interpolating between adjacent points based on the number of CPU cores, resulting in a quadratic interpolation value. This is the SubTask data optimized by the artificial intelligence algorithm. The advantages of this method include:

[0070] 1. The intervals are uniform, which is more compatible with the Flink SQL engine's Transformer time-series processing principle.

[0071] 2. Provides a more realistic representation of missing data in the TaskSlot scenario.

[0072] Finally, the SubTasks, optimized by the artificial intelligence algorithm, are placed into different Task Slots in sequence, thereby achieving a uniform distribution of SubTasks on the TaskManager.

[0073] Preferably, the method further includes:

[0074] The Flink engine maps and schedules the RESET messages based on the thread pool's occupancy.

[0075] Preferably, the Flink engine's mapping and scheduling of the RESET message based on the thread pool's occupancy includes:

[0076] Add the TaskManager to the process pool;

[0077] Each Reset message is mapped to an operator, and operators of multiple Reset messages of the same type form an operation chain, which is associated with the TaskManager.

[0078] Calculate the thread pool load metric, allocate threads based on the thread pool load metric, and map and schedule the computation chains with the allocated threads. Computation chains with higher weights prioritize data transfer-level operations from threads with lower thread occupancy rates. The thread pool load metric is:

[0079]

[0080] Where N is the number of worker threads in the thread pool at runtime, N max It is the maximum number of threads set, T cur T is the number of tasks in the current data collection time window. pre Q is the number of tasks in the previous data collection window, Q is the size of the task buffer queue, and ξ1, ξ2, and ξ3 are weighting coefficients.

[0081] Specifically, a thread pool utilization model is constructed, and the computation chain and threads are scheduled to complete the large-scale Reset message optimization process for base stations and core networks.

[0082] First, add the TaskManager from step two to the process pool.

[0083] Secondly, each TaskManager stored in the process pool is mapped to an operator, and multiple Reset messages of the same type form an operation chain. The operation chain is associated with each TaskManager in step two (process slot).

[0084] Then, the operation chain and the threads allocated by the thread pool utilization model are mapped and scheduled. The operation chain with high weight will prioritize data transfer level operation from the high-quality thread with low thread utilization.

[0085] The thread pool load metric ω is calculated. The load is derived from data such as the number of worker threads, the maximum number of threads, and the size of the task buffer queue during thread pool operation. A percentage value is obtained by calculating different weighting ratios.

[0086] The formula is as follows:

[0087]

[0088] Where N is the number of worker threads in the thread pool at runtime. max This is the maximum number of threads set. Describes the saturation of worker threads; T cur T is the number of tasks in the current data collection time window. pre Q is the number of tasks in the previous data collection window, and Q is the size of the task buffer queue. Describe the current task saturation. Describes the growth rate of the task buffer queue; ξ is a weighting coefficient. Compare it to the preset thread pool load ω'; if it is greater than ω', trigger adaptive parameter adjustment calculation; otherwise, skip the current collection time window.

[0089] This embodiment maps all processes, including those within the cache, to a single TaskManager. Each TaskManager can be divided into multiple process slots (TaskSlots). By dynamically adjusting the number of TaskSlots, users can define which Reset messages (SubTasks) can be isolated from each other. Simultaneously, by using shared slots, the resources of the slots are fully utilized, allowing more costly SubTasks to be evenly distributed across TaskManagers. This enables Reset messages (SubTasks) to share the same TaskManager process, TCP connections, heartbeat messages, and some datasets and data structures, thereby reducing task overhead and saving system resources.

[0090] Example 2

[0091] According to another aspect of the present invention, the present invention also provides a message scheduling optimization system based on Flink, the system comprising a first device, a second device, and a Flink engine;

[0092] The first device sends a RESET message to the second device. The RESET message includes the identification information of the first device and a system resource identifier, and the system resource identifier is associated with the terminal device.

[0093] The second device sends a request instruction to the Flink engine, the request instruction including the RESET message and IP information;

[0094] The Flink engine maps all processes to a TaskManager, which includes multiple process slots (TaskSlot).

[0095] The Flink engine maps the RESET message to a SubTask and calls the intelligent allocation model to achieve a uniform distribution of the SubTask on the TaskManager.

[0096] Preferably, the first device is a 5G access network device and the second device is a 5G core network device; or, the first device is a 5G core network device and the second device is a 5G access network device.

[0097] Preferably, the Flink engine's invocation of the intelligent allocation model to achieve the uniform distribution of the subtasks on the TaskManager includes:

[0098] Access information about all subtasks mapped to RESET messages deployed on TaskManager;

[0099] The difference formula is used to perform interpolation on different SubTasks with uneven file sizes, resulting in processed SubTask data; the difference formula is:

[0100]

[0101] Where x represents time, y represents the current value, and i represents the sequence number;

[0102] The processed SubTask data is then placed into different TaskSlots in sequence to achieve a uniform distribution of SubTasks on the TaskManager.

[0103] Preferably, the Flink engine maps and schedules the RESET message based on the thread pool's occupancy.

[0104] Preferably, the Flink engine's mapping and scheduling of the RESET message based on the thread pool's occupancy includes:

[0105] Add the TaskManager to the process pool;

[0106] Each Reset message is mapped to an operator, and operators of multiple Reset messages of the same type form an operation chain, which is associated with the TaskManager.

[0107] Calculate the thread pool load metric, allocate threads based on the thread pool load metric, and map and schedule the computation chains with the allocated threads. Computation chains with higher weights prioritize data transfer-level operations from threads with lower thread occupancy rates. The thread pool load metric is:

[0108]

[0109] Where N is the number of worker threads in the thread pool at runtime, N max It is the maximum number of threads set, T cur T is the number of tasks in the current data collection time window. pre Q is the number of tasks in the previous data collection window, Q is the size of the task buffer queue, and ξ1, ξ2, and ξ3 are weighting coefficients.

[0110] The specific implementation process of the functions implemented by the first device, the second device, and the Flink engine in this embodiment 2 is the same as that in embodiment 1, and will not be repeated here.

[0111] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A message scheduling optimization method based on Flink, characterized in that, The method includes the following steps: S1: The first device sends a RESET message to the second device. The RESET message includes the identification information of the first device and a system resource identifier. The system resource identifier is associated with the terminal device. S2: The second device sends a request instruction to the Flink engine, the request instruction including the RESET message and IP information; S3: The Flink engine maps all processes to a TaskManager, which includes multiple process slots (TaskSlot); The Flink engine maps the RESET message to a SubTask and calls the intelligent allocation model to achieve a uniform distribution of the SubTask on the TaskManager. The Flink engine invokes the intelligent allocation model to achieve a uniform distribution of the SubTasks on the TaskManager, including: Access information about all subtasks mapped to RESET messages deployed on TaskManager; The difference formula is used to perform interpolation on different SubTasks with uneven file sizes, resulting in processed SubTask data; the difference formula is: Where x represents time, y represents the current value, and i represents the sequence number; The processed SubTask data are sequentially placed into different TaskSlots to achieve a uniform distribution of SubTasks on the TaskManager. The method further includes: the Flink engine mapping and scheduling the RESET message according to the thread pool occupancy, specifically including: Add the TaskManager to the process pool; Each Reset message is mapped to an operator, and operators of multiple Reset messages of the same type form an operation chain, which is associated with the TaskManager. Calculate the thread pool load metric, allocate threads based on the thread pool load metric, and map and schedule the computation chains with the allocated threads. Computation chains with higher weights prioritize data transfer-level operations from threads with lower thread occupancy rates. The thread pool load metric is: in, It is the number of worker threads in the thread pool at runtime. This is the maximum number of threads set. This is the number of tasks in the current data collection time window. This is the number of tasks in the previous data collection window. It is the size of the task buffer queue. , and It is the weighting coefficient.

2. The method according to claim 1, characterized in that, The first device is a 5G access network device, and the second device is a 5G core network device; or, the first device is a 5G core network device, and the second device is a 5G access network device.

3. A message scheduling optimization system based on Flink, characterized in that, The system includes a first device, a second device, and a Flink engine; The first device sends a RESET message to the second device. The RESET message includes the identification information of the first device and a system resource identifier, and the system resource identifier is associated with the terminal device. The second device sends a request instruction to the Flink engine, the request instruction including the RESET message and IP information; The Flink engine maps all processes to a TaskManager, which includes multiple process slots (TaskSlot). The Flink engine maps the RESET message to a SubTask and calls the intelligent allocation model to achieve a uniform distribution of the SubTask on the TaskManager. The Flink engine invokes the intelligent allocation model to achieve a uniform distribution of the SubTasks on the TaskManager, including: Access information about all subtasks mapped to RESET messages deployed on TaskManager; The difference formula is used to perform interpolation on different SubTasks with uneven file sizes, resulting in processed SubTask data; the difference formula is: Where x represents time, y represents the current value, and i represents the sequence number; The processed SubTask data are sequentially placed into different TaskSlots to achieve a uniform distribution of SubTasks on the TaskManager. The Flink engine maps and schedules the RESET messages based on the thread pool's occupancy, specifically including: Add the TaskManager to the process pool; Each Reset message is mapped to an operator, and operators of multiple Reset messages of the same type form an operation chain, which is associated with the TaskManager. Calculate the thread pool load metric, allocate threads based on the thread pool load metric, and map and schedule the computation chains with the allocated threads. Computation chains with higher weights prioritize data transfer-level operations from threads with lower thread occupancy rates. The thread pool load metric is: in, It is the number of worker threads in the thread pool at runtime. This is the maximum number of threads set. This is the number of tasks in the current data collection time window. This is the number of tasks in the previous data collection window. It is the size of the task buffer queue. , and It is the weighting coefficient.

4. The system according to claim 3, characterized in that, The first device is a 5G access network device, and the second device is a 5G core network device; or, the first device is a 5G core network device, and the second device is a 5G access network device.

Images