Sample weight adjustment method and system based on reinforcement learning and storage medium
Reinforcement learning is used to optimize sample weights during deep learning training, addressing inefficiencies in existing methods by learning optimal strategies without manual experimentation, especially for unbalanced datasets.
Patent Information
- Application Number
- CN202411250000.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-12-06
AI Technical Summary
Existing sample weight adjustment methods require numerous experiments and verifications to avoid noise or bias, leading to high costs and manpower, and are inefficient in handling unbalanced data sets.
A method and system utilizing reinforcement learning to optimize sample weights during deep learning training, where an agent interacts with a loss function environment to adjust weights based on feedback, learning the optimal strategy without manual experimentation.
Reduces the need for extensive manual experiments, efficiently obtaining sample weights that fit the model and data size, particularly for unbalanced datasets like special vehicle classification and medical lesion maps, thereby reducing costs and labor.
Smart Images

Figure CN119089321A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a method, system and storage medium for adjusting sample weights based on reinforcement learning. Background Art
[0002] Sample weight adjustment is a method of adjusting sample weights directly during model training. By assigning different weights to samples of different categories, the model pays more attention to minority category samples, thereby balancing the influence of different categories. Common sample weight adjustment methods include weighting based on category frequency, weighting based on category importance, etc.
[0003] When using sample weight adjustment, the weight setting needs to be handled with caution to avoid introducing too much noise or bias into the model. Therefore, in order to provide the model with sample weights that best fit the model and sample size, a large number of experiments and verifications are often required to ensure that the effect of weight adjustment meets expectations and does not cause overfitting or underfitting of the model. However, a large number of experiments and verifications require high costs and manpower. Summary of the invention
[0004] Therefore, in order to solve the above-mentioned shortcomings, the present invention provides a method, system and storage medium for sample weight adjustment based on reinforcement learning. The method for sample weight adjustment adds the part of reinforcement learning to select and optimize the weights, so that sample weights that meet the model and sample size can still be obtained without conducting a large number of manual experiments.
[0005] In a first aspect, the present invention provides a method for adjusting sample weights based on reinforcement learning, comprising: Define classification models for reinforcement learning agents and deep learning networks, The classification model is instantiated and passed into the loss function environment that interacts with reinforcement learning as an object for use in creating instance objects; The instantiated environment is passed to the agent as an object, which serves as the interactive environment between the agent and the deep learning network loss. After the agent performs random actions in the interactive environment, it obtains the loss function change through the loss function environment feedback, and updates and records it. The agent adjusts the weight of each sample by selecting an action based on the recorded value, and learns the best strategy by continuously updating this value, which is the loss value generated by the correct action feedback through the loss function.
[0006] The method also includes training the deep learning network. In each round of training, the deep learning network is used as an interactive environment for reinforcement learning. Through the learning of the intelligent agent, the sample weight that produces the maximum cumulative reward and the lowest loss in the current deep learning round are obtained, and then the sample weight of the maximum cumulative reward is assigned to the loss function.
[0007] Optionally, the reinforcement learning agent is a Q learning algorithm.
[0008] Optionally, the updating process in the updating and recording is as follows: , in: Q(s,a) is in state s Next action a of Q value, α is the learning rate, indicating the update Q The weight of the value, r is in state s Execute an action a After the reward, γ is a discount factor that measures the importance of future rewards, s’ Is to perform an action a The next state to enter is a’ is in the next state s’ Among all possible actions, choose the one that produces the greatest Q Value action.
[0009] Optionally, the random action is a random addition or subtraction of sample weights.
[0010] Optionally, the loss function change includes sample weights and losses.
[0011] In a second aspect, the present invention provides a system for sample weight adjustment based on reinforcement learning, comprising: The first module is used to define the classification model of reinforcement learning agents and deep learning networks; The second module is used to instantiate the classification model and pass it as an object into the loss function environment that interacts with reinforcement learning, which is used when creating instance objects; The third module is used to pass the instantiated environment as an object to the intelligent agent as an interactive environment for the intelligent agent and the deep learning network loss. After the intelligent agent performs random actions in the interactive environment, it obtains the loss function change through the loss function environment feedback, and updates and records it; and The fourth module is used for the agent to adjust the weight of each sample by selecting an action through the recorded value, and learn the best strategy by continuously updating the value, which is the loss value generated by the correct action feedback through the loss function.
[0012] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for sample weight adjustment based on reinforcement learning.
[0013] The present invention has the following advantages: The method for adjusting sample weights based on reinforcement learning described in the present invention adds a reinforcement learning part to select and optimize weights, so that sample weights that meet the model and sample size can still be obtained without conducting a large number of manual experiments.
[0014] The present invention can learn data that is difficult to obtain, limited in number, and has unbalanced samples, such as classification and recognition of special vehicles, classification maps of medical lesions, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a flow chart of the method for adjusting sample weights based on reinforcement learning according to the present invention; Figure 2 is a system block diagram of the sample weight adjustment system based on reinforcement learning used in the present invention; In the figure: 100, first module; 200, second module; 300, third module; 400, fourth module; 500, fifth module. DETAILED DESCRIPTION
[0016] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0017] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus.
[0018] As described in the background technology, when using sample weight adjustment, it is necessary to carefully handle the weight setting to avoid introducing too much noise or bias into the model. Therefore, in order to provide the model with sample weights that best fit the model and sample size, a large number of experiments and verifications are often required to ensure that the effect of weight adjustment meets expectations and does not cause overfitting or underfitting of the model. However, a large number of experiments and verifications require high costs and manpower.
[0019] Based on the above reasons, this embodiment provides a method for adjusting sample weights based on reinforcement learning. This method is to adjust the weight of each sample in the loss function through reinforcement learning during the deep learning training process to achieve the purpose of optimizing the training effect. This process can be regarded as a joint optimization. In each round of deep learning training, reinforcement learning is performed while the network weight remains unchanged. Through the interaction between the agent and the environment, the agent is rewarded when the environmental loss decreases, and the category weight is changed, so that the loss function can be assigned an optimal sample weight in each round of training, and the agent is continuously used for optimization during the deep learning network training process.
[0020] Exemplarily, a method for adjusting sample weights based on reinforcement learning includes: Step S100: define a classification model of a reinforcement learning agent and a deep learning network; wherein the reinforcement learning agent may be a Q learning algorithm, etc. Step S200, instantiate the classification model and pass it as an object into the loss function environment that interacts with reinforcement learning for use in creating instance objects (i.e., pass it as a parameter into the __init__ construction method of the environment class); Step S300: The instantiated environment is passed into the agent as an object, which serves as an interactive environment for the agent and the deep learning network loss. After the agent performs random actions (random addition and subtraction of sample weights) in the interactive environment, the loss function change (including sample weights and losses) is obtained through feedback from the loss function environment, and is updated and recorded. Step S400, the agent selects an action through the recorded value to adjust the weight of each sample, and learns the best strategy by continuously updating the value, which is the loss value generated by the correct action feedback through the loss function.
[0021] Step S500: train the deep learning network. In each round of training (i.e., when the network weight remains unchanged), the deep learning network is used as an interactive environment for reinforcement learning. Through the learning of the agent, the sample weight that produces the maximum cumulative reward and the lowest loss in the current deep learning round are obtained. With each round of deep learning training, the loss function can be assigned an optimal sample weight with the help of reinforcement learning, thereby finally obtaining a deep learning network model that adapts to uneven categories.
[0022] By adding the above technical features to the reinforcement learning part to select and optimize the weights, it is possible to obtain sample weights that meet the model and sample size without a large number of manual experiments, reducing the cost and labor of experiments and verification. This method can learn data that is difficult to obtain, limited in number, and has unbalanced samples, such as the classification and recognition of special vehicles, and medical lesion classification maps.
[0023] In one embodiment, the updating process in the updating and recording is as follows: , in: Q(s,a) is in state s Next action a of Q value, α is the learning rate, indicating the update Q The weight of the value, r is in state s Execute an action a After the reward, γ is a discount factor that measures the importance of future rewards, s’ Is to perform an action a The next state to enter is a’ is in the next state s’ Among all possible actions, choose the one that produces the greatest Q Value action.
[0024] Optionally, the random action is a random addition or subtraction of sample weights.
[0025] Optionally, the loss function change includes sample weights and losses.
[0026] In another embodiment, a system for sample weight adjustment based on reinforcement learning is also provided, including: The first module is used to define the classification model of reinforcement learning agents and deep learning networks; The second module is used to instantiate the classification model and pass it as an object into the loss function environment that interacts with reinforcement learning, which is used when creating instance objects; The third module is used to pass the instantiated environment as an object to the intelligent agent as an interactive environment for the intelligent agent and the deep learning network loss. After the intelligent agent performs random actions in the interactive environment, it obtains the loss function change through the loss function environment feedback, and updates and records it; and The fourth module is used for the agent to adjust the weight of each sample by selecting an action through the recorded value, and learn the best strategy by continuously updating the value, which is the loss value generated by the correct action feedback through the loss function.
[0027] In one embodiment, the system for sample weight adjustment based on reinforcement learning also includes a fifth module for training the deep learning network. In each round of training, the deep learning network is used as an interactive environment for reinforcement learning. Through the learning of the intelligent agent, the sample weight that produces the maximum cumulative reward and the lowest loss in the current deep learning round are obtained, and then the sample weight of the maximum cumulative reward is allocated to the loss function.
[0028] The system is used to implement the method of sample weight adjustment based on reinforcement learning, and can still obtain sample weights that meet the model and sample size without conducting a large number of manual experiments.
[0029] It should be noted that, in the above embodiments, the description of each embodiment has its own focus, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0030] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-readable program codes.
[0031] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0032] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0033] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0034] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the above-mentioned method for adjusting sample weights based on reinforcement learning.
[0035] If the module / unit integrated in the multi-task model training device / terminal device of the newly added task is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.
[0036] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.
[0037] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0038] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0039] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0040] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The above embodiments are only used to illustrate the technical solution of the present application, not to limit it; although the present application is described in detail with reference to the above embodiments, ordinary technicians in this field should understand that: it is still possible to modify the technical solutions recorded in the above embodiments, or to replace some of the technical features therein by equivalent; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of each embodiment of the present application.
[0041] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for adjusting sample weights based on reinforcement learning, characterized in that: include Define classification models for reinforcement learning agents and deep learning networks, The classification model is instantiated and passed into the loss function environment that interacts with reinforcement learning as an object for use in creating instance objects; The instantiated environment is passed to the agent as an object, which serves as the interactive environment between the agent and the deep learning network loss. After the agent performs random actions in the interactive environment, it obtains the loss function change through the loss function environment feedback, and updates and records it. The agent adjusts the weight of each sample by selecting an action based on the recorded value, and learns the best strategy by continuously updating this value, which is the loss value generated by the correct action feedback through the loss function.
2. The method for adjusting sample weights based on reinforcement learning according to claim 1, characterized in that: It also includes training the deep learning network. In each round of training, the deep learning network is used as an interactive environment for reinforcement learning. Through the learning of the intelligent agent, the sample weight that produces the maximum cumulative reward and the lowest loss in the current deep learning round are obtained, and then the sample weight of the maximum cumulative reward is assigned to the loss function.
3. The method for adjusting sample weights based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning agent is a Q learning algorithm.
4. The method for adjusting sample weights based on reinforcement learning according to claim 3, characterized in that: The update process in the update and record is as follows: , in: Q(s,a) is in state s Next action a of Q value, α is the learning rate, indicating the update Q The weight of the value, r is in state s Execute an action a After the reward, γ is a discount factor that measures the importance of future rewards, s’ Is to perform an action a The next state to enter is a’ is in the next state s’ Among all possible actions, choose the one that produces the greatest Q Value action.
5. The method for adjusting sample weights based on reinforcement learning according to claim 1, characterized in that: The random action is the random addition and subtraction of sample weights.
6. The method for adjusting sample weights based on reinforcement learning according to claim 1, characterized in that: The loss function change includes sample weights and losses.
7. A system for sample weight adjustment based on reinforcement learning, characterized in that: include The first module is used to define the classification model of reinforcement learning agents and deep learning networks; The second module is used to instantiate the classification model and pass it as an object into the loss function environment that interacts with reinforcement learning, which is used when creating instance objects; The third module is used to pass the instantiated environment as an object to the intelligent agent as an interactive environment for the intelligent agent and the deep learning network loss. After the intelligent agent performs random actions in the interactive environment, it obtains the loss function change through the loss function environment feedback, and updates and records it; as well as The fourth module is used for the agent to adjust the weight of each sample by selecting an action through the recorded value, and learn the best strategy by continuously updating the value, which is the loss value generated by the correct action feedback through the loss function.
8. A storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for adjusting sample weights based on reinforcement learning as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Method for training intelligent agent with dynamic award example samples based on reinforcement learning
CN111582311A
Reinforcement learning method and computer readable storage medium
CN111950703A
Average SAC deep reinforcement learning method and system based on Q learning
CN114881228A
