Tool-assisted large language model instruction fine tuning method oriented to evolutionary resource management

Abstract

The invention discloses an evolutionary resource management-oriented tool-assisted large language model instruction fine tuning method. The method comprises the following steps of: training a pre-training language model based on statistical parameter state data of federated learning; configuring a virtual environment, and interacting with the virtual environment through the large language model; comprising the steps that a large language model generates an equipment selection action based on language representation of a current observation state, and delivers the equipment selection action to a virtual environment for state transition; updating the language representation of the current observation state to obtain the language representation of the observation state at the next moment, and generating a reward at the same time; taking the language representation of the current observation state, the equipment selection action, the reward and the language representation of the observation state at the next moment as a record; according to all the recorded data, a group relative strategy is adopted to optimize and maximize the objective function, and therefore fine tuning of the large language model is achieved; reasoning is carried out through the fine-tuned large language model, an equipment selection decision action is output, and a resource allocation action is executed.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and network communication, and in particular relates to a tool-assisted method for fine-tuning large language model instructions for evolutionary resource management. Background Technology

[0002] Next-generation wireless communication systems are expected to support a wide range of intelligent applications and services, benefiting from the exponential growth of wireless edge devices such as smartphones and sensors. Federated learning (FL) has emerged as an important paradigm by facilitating collaborative model training among distributed devices while maintaining data locality in a privacy-friendly manner. However, deploying FL in wireless networks faces significant challenges stemming from the underlying heterogeneous computing and communication capabilities of the devices. Accordingly, in dynamic wireless environments, appropriate resource management strategies need to be developed to select participating devices and calibrate the communication and computing resources used.

[0003] Federated learning is implemented on wireless networks, where edge devices upload their locally trained updates to a centralized server via wireless links in each training round in exchange for the aggregated model. Through multiple training rounds, the performance of the final global model is primarily affected by resources and data heterogeneity. Differences in computing and communication capabilities between devices lead to imbalances in time and energy consumption, while variations in dataset size and non-independent and identically distributed (non-IID) data distributions result in biases in model updates. Furthermore, the deployment of federated learning is constrained by energy and time budgets. Therefore, a wireless federated learning resource management method is urgently needed to overcome these problems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method, device, and medium for wireless federated learning resource management based on a large language model.

[0005] According to a first aspect of the present invention, the present invention provides a tool-assisted method for fine-tuning large language model instructions for evolutionary resource management, the method comprising: The pre-trained language model is trained using statistical parameter state data based on federated learning. Configure a virtual environment and interact with it through a large language model; this includes: the large language model generating a device selection action based on the language representation of the current observation state and delivering it to the virtual environment for state transition; updating the language representation of the current observation state to obtain the language representation of the next observation state, and generating a reward; recording the language representation of the current observation state, the device selection action, the reward, and the language representation of the next observation state as a single record; and using a group-relative strategy to optimize and maximize the objective function based on all recorded data, thereby achieving fine-tuning of the large language model. Obtain the language representation of the current observation state in the federated learning environment, input the language representation of the current observation state into the fine-tuned large language model, the large language model outputs the device selection decision action, and executes the resource allocation action.

[0006] Furthermore, the process of training a pre-trained language model based on statistical parameter state data from federated learning includes: Linear layers are added at the beginning and end of the pre-trained language model as neural networks f(⋅) to be trained; Obtain the statistical parameter state data for each round of federated learning, and preprocess the state data into a serialized sample vector, denoted as a sample vector. ,in, This represents the statistical parameter state data during the t-th round of federated learning training. Indicates precision, This indicates the selected device serial number; multiple consecutive sample vectors constitute a complete trajectory, and multiple trajectories constitute the training dataset; Based on the training dataset, an autoregressive approach is used to train a pre-trained language model by minimizing the mean squared error loss function.

[0007] Furthermore, the pre-trained language model is the GPT-1 model.

[0008] Furthermore, the statistical parameter state data for each round of federated learning includes the local dataset size, local loss value, inner product of local and global model parameters, and percentage of sign similarity between local and global model parameters.

[0009] Furthermore, the process of interacting with the virtual environment through a large language model includes: Initialize the large language model and load its parameters as the initial strategy. Initialize the old strategy ; Configure a virtual environment, which includes device status. Statistical parameters and state data for each round of federated learning ; Language representation of the current observation state obtained from the virtual environment The large language model generates device selection actions based on the language representation of the current observation state. And deliver it to the virtual environment for state transfer; Update the linguistic representation of the current observation state to obtain the linguistic representation of the observation state at the next time step. At the same time, generate rewards The verbal representation of the current observation state, the device selection action, the reward, and the verbal representation of the next observation state are recorded as a single record. ; Based on all recorded data Standardize all rewards and calculate action advantage values; Based on all recorded data and action advantage values, an objective function is constructed. A group-relative strategy is used to optimize and maximize the objective function, thereby achieving fine-tuning of the large language model.

[0010] Furthermore, the device status This includes resource tools recording and simulating the communication status, computing status, and time and energy consumption of the device; wherein, the communication status includes bandwidth and transmission power; and the computing status includes CPU computing frequency.

[0011] Furthermore, the expression for the objective function is as follows: In the formula, Representation strategy, Indicates the old strategy, This indicates a device selection action. The language representation of the current observation state, Indicates the action advantage value. This represents hyperparameters.

[0012] Furthermore, the process of obtaining a linguistic representation of the current observation state in the federated learning environment, inputting this linguistic representation into a fine-tuned large language model, and the large language model outputting a device selection decision action and executing a resource allocation action includes: In a single communication round, obtain the linguistic representation of the current observation state in the federated learning environment. ; The language representation of the current observation state The input is fed into the fine-tuned large language model, which then uses the language representation based on the current observation state. and strategy Generate text reply The text reply That is, the equipment selection decision action. ; The large language model executes resource allocation actions for the selected device. ; Joint decision-making actions It operates in wireless federated learning environments.

[0013] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described tool-assisted large language model instruction fine-tuning method for evolutionary resource management.

[0014] According to a third aspect of the present invention, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described tool-assisted large language model instruction fine-tuning method for evolutionary resource management.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention provides a tool-assisted method for fine-tuning large language model instructions for evolutionary resource management, enabling efficient device selection and resource allocation in wireless federated learning. This method fine-tunes a large language model (LLM) to generate appropriate device selection outputs based on the language description of the wireless federated learning (FL) scenario. Simultaneously, it allocates resources with the assistance of resource tools, establishing a virtual wireless federated learning environment that allows the LLM to interact offline, solving the cost problem of online interaction. This method is particularly suitable for changing wireless environments.

[0016] 2. This invention designs a generative strategy based on human language for wireless federated learning. The high-dimensional joint decision-making problem is decomposed into joint device selection and resource management performed independently. Based on this, the state of wireless federated learning is described using human language as input to a large language model (LLM). The LLM can effectively adapt to state descriptions under various environments and tasks. The LLM performs generative selection actions and utilizes convex optimization resource management tools for the corresponding task for resource management. This invention ensures that independently generated actions maintain excellent performance by invoking and adapting to tools for joint optimization.

[0017] 3. This invention trains an environment model (i.e., the pre-trained language model GPT-1 in this example) using autoregressive learning and a model-based approach. The virtual learning environment of this environment model is integrated into a wireless federated learning (FL) scenario, combining a large language model (LLM) with the wireless federated learning scenario. Combined with resource management tools, this model-based approach can efficiently collect trajectories, enabling the LLM to evolve and be trained in a simulated environment using Group Relative Policy Optimization (GRPO). Continuous learning in a high-fidelity simulated environment improves the LLM's decision-making ability while significantly reducing the communication overhead required for real-world interactions. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating a tool-assisted large language model instruction fine-tuning method for evolutionary resource management provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating a tool-assisted large language model instruction fine-tuning method for evolutionary resource management provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0022] Resource management for wireless power lines (FLs) is typically a multidimensional optimization problem that can be partially solved using heuristic solutions or reinforcement learning (RL)-based methods. However, heuristic solutions often require domain-specific expertise and extensive tweaking to adapt to unknown scenarios, while RL-based methods rely heavily on interaction with the environment and become sluggish in large-scale system dynamics and changing conditions. Therefore, finding alternative solutions is crucial.

[0023] While LLMs exhibit strong linguistic capabilities, their fine-tuning performance remains limited by the scope of the training datasets used. Therefore, due to the lack of available expert data on decision-making in wireless FL (Flexible Interference Model) processes, fine-tuned LLMs cannot reliably interpret wireless FL data and generate effective decisions. On the other hand, training an LLM to map observations to actions requires large amounts of real-world datasets, the collection of which is time-consuming and costly, hindering scalability. Furthermore, although augmented reasoning LLMs use long token sequences to improve computational efficiency, this approach is not feasible in wireless FL systems due to time constraints. Therefore, additional effort is still required to enable LLMs to perform decision-making in wireless FLs due to domain-specific expertise and computationally intensive optimization operations.

[0024] In this example, we consider using natural language-based scene cues to enhance generalization capabilities through the flexibility of language. Building upon this, a mathematically constructed problem decomposition enables the LLM to derive device selection and resource management results by answering scene-specific questions and invoking convex optimization tools. Furthermore, to better address potential environmental changes, in addition to available convex optimization tools, we construct a sample-efficient model-based virtual learning environment. This environment can characterize the relationship between learning performance and device selection, and facilitate the evolution of the internal LLM with low communication costs.

[0025] Based on the above motivations, this invention designs a decision framework T-ELLM (Tool-Aided Evolutionary LLM) that integrates LLM, tool-aided, and model evolution-based approaches. It aims to generate device selection and resource management strategies suitable for wireless FL environments, significantly improve wireless FL efficiency, and provide an efficient solution for wireless FL operation in heterogeneous and dynamic environments.

[0026] like Figure 1 As shown, this invention provides a tool-assisted method for fine-tuning large language model instructions for evolutionary resource management, the method comprising: Step S1: Train the pre-trained language model based on the statistical parameter state data of federated learning.

[0027] Specifically, in this example, the pre-trained language model uses the GPT-1 model. The following uses the GPT-1 model as an example to illustrate the sub-steps of step S1: Step S101: Add linear layers at the beginning and end of the GPT-1 model as training neural networks f(⋅), and randomly initialize the GPT-1 model.

[0028] Step S102: Obtain the statistical parameter state data for each round of federated learning, and preprocess the state data into a serialized sample vector, denoted as a sample vector. ,in, This represents the statistical parameter state data during the t-th round of federated learning training. Indicates precision, This indicates the selected device number; multiple consecutive sample vectors constitute a complete trajectory, and multiple trajectories constitute the training dataset.

[0029] Furthermore, in this example, the statistical parameter state data includes the local dataset size, local loss value, inner product of local and global model parameters, and percentage of sign similarity between local and global model parameters.

[0030] Step S103: Based on the training dataset, train the GPT-1 model by minimizing the mean squared error loss function using an autoregressive approach.

[0031] The large language model is trained using an autoregressive approach, where each subsequent sample serves as the label for the preceding sample. The mean squared error loss function is expressed as follows: Loss= In the formula, This represents the statistical parameter state data output by the GPT-1 model during the (t+1)th round of federated learning training. This represents the accuracy of the output of the GPT-1 model during the (t+1)th round of federated learning training.

[0032] Step S2 involves configuring the virtual environment and interacting with it through a large language model. This includes: the large language model generating a device selection action based on the language representation of the current observation state and delivering it to the virtual environment for state transition; updating the language representation of the current observation state to obtain the language representation of the next observation state, and generating a reward; recording the language representation of the current observation state, the device selection action, the reward, and the language representation of the next observation state as a single record; and optimizing the objective function using a group-relative strategy based on all recorded data to achieve fine-tuning of the large language model.

[0033] Specifically, step S2 includes the following sub-steps: Step S201: Initialize the large language model and load its parameters as the initial strategy. Initialize the old strategy .

[0034] Step S202: Configure the virtual environment, which includes device status. Statistical parameters and state data for each round of federated learning .

[0035] Furthermore, the device status This includes resource tools recording and simulating the communication status, computing status, and time and energy consumption of the device; wherein, the communication status includes bandwidth and transmission power; and the computing status includes CPU computing frequency.

[0036] Step S203: The large language model interacts offline with the virtual environment; including: Language representation of the current observation state obtained from the virtual environment The large language model generates device selection actions based on the language representation of the current observation state. Record the probability of the current action. And deliver it to the virtual environment for state transfer; in, This represents the j-th token output by the large language model in the t-th round of federated learning. This represents the 0th to j-1th tokens of the large language model in the t-th round of federated learning.

[0037] Update the linguistic representation of the current observation state to obtain the linguistic representation of the observation state at the next time step. At the same time, generate rewards Thus, the verbal representation of the current observation state, the device selection action, the reward, and the verbal representation of the next observation state are recorded as a single record. ; Step S204: The large language model interacts offline with the virtual environment multiple times to generate several records to form a replay buffer dataset; all records are then acquired and denoted as... ; Based on the mean and standard deviation of rewards across all recorded data, all rewards are standardized using the following expression: In the formula, This represents the standardized reward. Indicates a reward. This represents the average reward. The standard deviation of the reward.

[0038] Based on the standardized reward, the action advantage value is calculated as follows: In the formula, This indicates the action advantage value.

[0039] Step S205: Construct an objective function based on all recorded data and action advantage values, and use Group Relative Policy Optimization (GRPO) to optimize and maximize the objective function, thereby achieving fine-tuning of the large language model.

[0040] The objective function is expressed as follows: in, Representation strategy, Indicates the old strategy, This indicates a device selection action. The language representation of the current observation state, Indicates the action advantage value. These are hyperparameters related to pruning and used for stable training.

[0041] It's important to note that LLM avoids the significant communication overhead associated with online interaction in wireless federated learning environments. The virtual environment is divided into two parts: a system environment and a statistical environment. The system environment focuses on the communication and computational status of devices based on given resources, with resource tools recording and simulating observations and time / energy consumption. The statistical environment focuses on the convergence characteristics and performance of the local and global models in federated learning, which is fitted by a separate neural network (i.e., the GPT-1 model trained in step S1). LLM undergoes fine-tuning and updates with the help of model evolution and tool assistance to complete the evolution.

[0042] Step S3: Obtain the language representation of the current observation state in the federated learning environment, input the language representation of the current observation state into the fine-tuned large language model, the large language model outputs the device selection decision action, and executes the resource allocation action.

[0043] Specifically, step S3 includes the following sub-steps: Step S301: In one communication round, obtain the language representation of the current observation state in the federated learning environment. ; Step S302: Represent the current observation state in language. The input is fed into the fine-tuned large language model, which then uses the language representation based on the current observation state. and strategy Generate text reply ,Right now The text reply That is, the equipment selection decision action. .

[0044] Step S303: The large language model calls an external resource tool to perform resource allocation actions for the selected device. .

[0045] Step S304, Joint Decision-Making Action It operates in wireless federated learning environments.

[0046] In summary, this invention provides a tool-assisted method for fine-tuning large language model instructions for evolutionary resource management. By fine-tuning a large language model (LLM), it enables the model to generate appropriate device selection outputs based on the language description of the wireless FL scenario, thereby achieving efficient device selection and resource allocation in wireless federated learning.

[0047] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the tool-assisted large language model instruction fine-tuning method for evolutionary resource management as described above. Figure 3 The diagram shown illustrates a hardware structure of any device with data processing capabilities, used in the tool-assisted large language model instruction fine-tuning method for evolutionary resource management provided in this embodiment of the invention. (Except for...) Figure 3 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0048] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the tool-assisted large language model instruction fine-tuning method for evolutionary resource management as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0049] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0050] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A tool-assisted method for fine-tuning large language model instructions for evolutionary resource management, characterized in that, The method includes: The pre-trained language model is trained using statistical parameter state data based on federated learning. Configure a virtual environment and interact with it through a large language model; this includes: the large language model generating a device selection action based on the language representation of the current observation state and delivering it to the virtual environment for state transition; updating the language representation of the current observation state to obtain the language representation of the next observation state, and generating a reward; recording the language representation of the current observation state, the device selection action, the reward, and the language representation of the next observation state as a single record; and using a group-relative strategy to optimize and maximize the objective function based on all recorded data, thereby achieving fine-tuning of the large language model. Obtain the language representation of the current observation state in the federated learning environment, input the language representation of the current observation state into the fine-tuned large language model, the large language model outputs the device selection decision action, and executes the resource allocation action.

2. The tool-assisted large language model instruction fine-tuning method for evolutionary resource management according to claim 1, characterized in that, The process of training a pre-trained language model using statistical parameter state data based on federated learning includes: Linear layers are added at the beginning and end of the pre-trained language model as neural networks f(⋅) to be trained; Obtain the statistical parameter state data for each round of federated learning, and preprocess the state data into a serialized sample vector, denoted as a sample vector. ,in, This represents the statistical parameter state data during the t-th round of federated learning training. Indicates precision, This indicates the selected device serial number; multiple consecutive sample vectors constitute a complete trajectory, and multiple trajectories constitute the training dataset; Based on the training dataset, an autoregressive approach is used to train a pre-trained language model by minimizing the mean squared error loss function.

3. A tool-assisted large language model instruction fine-tuning method for evolutionary resource management according to claim 1 or 2, characterized in that, The pre-trained language model used is the GPT-1 model.

4. The tool-assisted large language model instruction fine-tuning method for evolutionary resource management according to claim 2, characterized in that, The statistical parameter state data for each round of federated learning includes the local dataset size, local loss value, inner product of local and global model parameters, and percentage of sign similarity between local and global model parameters.

5. The tool-assisted large language model instruction fine-tuning method for evolutionary resource management according to claim 1, characterized in that, The process of interacting with a virtual environment through a large language model includes: Initialize the large language model and load its parameters as the initial strategy. Initialize the old strategy ; Configure a virtual environment, which includes device status. Statistical parameters and state data for each round of federated learning ; Language representation of the current observation state obtained from the virtual environment The large language model generates device selection actions based on the language representation of the current observation state. And deliver it to the virtual environment for state transfer; Update the linguistic representation of the current observation state to obtain the linguistic representation of the observation state at the next time step. At the same time, a reward is generated. The verbal representation of the current observation state, the device selection action, the reward, and the verbal representation of the next observation state are recorded as a single record. ; Based on all recorded data All rewards are standardized and action advantage values ​​are calculated. Based on all recorded data and action advantage values, an objective function is constructed. A group-relative strategy is used to optimize and maximize the objective function, thereby achieving fine-tuning of the large language model.

6. The tool-assisted large language model instruction fine-tuning method for evolutionary resource management according to claim 5, characterized in that, The device status This includes resource tools recording and simulating the communication status, computing status, and time and energy consumption of the device; wherein, the communication status includes bandwidth and transmission power; and the computing status includes CPU computing frequency.

7. The tool-assisted large language model instruction fine-tuning method for evolutionary resource management according to claim 5, characterized in that, The expression for the objective function is as follows: In the formula, Representation strategy, Indicates the old strategy, This indicates a device selection action. The language representation of the current observation state, Indicates the action advantage value. This represents hyperparameters.

8. The tool-assisted large language model instruction fine-tuning method for evolutionary resource management according to claim 1, characterized in that, The process of obtaining a linguistic representation of the current observation state in the federated learning environment, inputting this linguistic representation into a fine-tuned large language model, and then having the large language model output a device selection decision action and execute a resource allocation action includes: In a single communication round, obtain the linguistic representation of the current observation state in the federated learning environment. ; The language representation of the current observation state The input is fed into the fine-tuned large language model, which then uses the language representation based on the current observation state. and strategy Generate text reply The text reply That is, the equipment selection decision action. ; The large language model executes resource allocation actions for the selected device. ; Joint decision-making actions It operates in wireless federated learning environments.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the tool-assisted large language model instruction fine-tuning method for evolutionary resource management as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the tool-assisted large language model instruction fine-tuning method for evolutionary resource management as described in any one of claims 1-8.

Images