Machine learning method and device, equipment and storage medium
By collecting expert demonstration data and using alternating control and discriminator of forward and backward strategies to generate reward signals, combined with the Actor-Critic framework of reinforcement learning and Q-value network integration mechanism, the problems of high data dependence and poor environmental adaptability in robot learning strategies are solved, and more efficient autonomous learning and strategy optimization are achieved.
Patent Information
- Application Number
- CN202511063479.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-07
AI Technical Summary
Existing robot learning strategies suffer from high data dependence, strong human intervention, and poor environmental adaptability, making them particularly difficult to adapt to complex scenarios and demanding tasks in the medical and financial fields.
The system collects expert demonstration data, acquires interactive data through alternating control of forward and backward policies, and generates reward signals using forward and backward discriminators. It then performs joint optimization by combining the Actor-Critic framework of reinforcement learning and the Q-value network ensemble mechanism, reducing the reliance on manually labeled data and reward functions.
It improves the robot's learning stability, strategy adaptability, and autonomous improvement capabilities in complex and dynamic environments, while reducing training costs and the frequency of human intervention.
Smart Images
Figure CN120911541A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of artificial intelligence, and can be applied to the fields of medical treatment and finance, and particularly relates to a machine learning method, device, equipment and storage medium. BACKGROUND
[0002] Current robot learning strategies mainly rely on two paradigms of imitation learning and reinforcement learning, but both have significant limitations in actual scenarios: imitation learning relies on a large amount of artificial demonstration data, which not only has high acquisition cost, but also has limited data coverage, making it difficult to cope with complex scenarios; traditional reinforcement learning can achieve strategy optimization through trial and error, but it strongly depends on the artificially designed reward function and frequent environment reset mechanism, and in non-periodic tasks, the continuous evolution of the environment state will greatly reduce the stability of the learning process.
[0003] This limitation is particularly prominent in fields such as medicine and finance, which have strict requirements for safety and precision:
[0004] In a medical scenario, imitation learning requires a large amount of expert surgical operation data, but the cost of artificial labeling of high-precision actions is extremely high, and the individual differences in cases result in insufficient data generalization, making it difficult to adapt to complex conditions; traditional reinforcement learning can optimize surgical strategies through trial and error, but it is limited by artificially defined reward functions, and the irreversibility of the surgical scene (tissue damage cannot be reset) severely restricts learning efficiency.
[0005] In the field of finance, imitation learning relies on historical operation data of high-frequency trading experts when used for quantitative trading strategies, but market dynamics make such data short-lived and poorly generalized; traditional reinforcement learning faces similar difficulties - artificially defined reward functions (such as "return volatility" and "risk control indicators") are difficult to adapt to rapidly changing markets, and the irreversibility of trading behavior (the state cannot be reset after a transaction) makes trial-and-error learning carry high risks.
[0006] In addition, most reinforcement learning systems based on visual input require pre-training of visual features or additional deployment of state estimation modules, which not only increases system complexity, but also reduces the feasibility of deployment in scenarios with strict time efficiency requirements such as medical real-time decision-making and high-frequency trading.
[0007] Even existing self-improving reinforcement learning methods (such as R3L) attempt to eliminate the need for artificial reward functions and resets, but their reliance on backward strategies based on state novelty measures and frozen visual encoders makes it difficult to effectively expand as the complexity of the environment increases, resulting in decreased learning stability and training efficiency; combined with the problems of action delay and incomplete observability of the state in real systems, imitation learning cannot accurately reproduce expert behavior, further limiting the upper limit of strategy performance.
[0008] Therefore, the existing robot learning strategy has problems of high data dependence, strong manual intervention and poor environmental adaptability. SUMMARY
[0009] Embodiments of the present application provide a machine learning method, device, equipment and storage medium, aiming at solving the problems of high data dependence, strong manual intervention and poor environmental adaptability of the existing robot learning strategy.
[0010] In a first aspect, embodiments of the present application provide a machine learning method, comprising:
[0011] Expert demonstration data is collected, wherein the expert demonstration data includes forward target state data and backward target state data;
[0012] Interaction data is obtained through alternating control of the forward strategy and the backward strategy, wherein the interaction data includes forward interaction data and backward interaction data;
[0013] A forward discriminator is trained using the forward target state data and the forward interaction data to obtain a forward reward signal;
[0014] A backward discriminator is trained using the backward target state data and the backward interaction data to obtain a backward reward signal;
[0015] Based on an Actor-Critic framework of reinforcement learning, a Q value network integration mechanism is introduced, and the forward strategy and the backward strategy are jointly optimized in combination with the forward reward signal and the backward reward signal.
[0016] In a second aspect, embodiments of the present application further provide a machine learning device, comprising:
[0017] A collection unit is configured to collect expert demonstration data, wherein the expert demonstration data includes forward target state data and backward target state data;
[0018] An obtaining unit is configured to obtain interaction data through alternating control of a forward strategy and a backward strategy, wherein the interaction data includes forward interaction data and backward interaction data;
[0019] A forward training unit is configured to train a forward discriminator using the forward target state data and the forward interaction data to obtain a forward reward signal;
[0020] A backward training unit is configured to train a backward discriminator using the backward target state data and the backward interaction data to obtain a backward reward signal;
[0021] The joint optimization unit is used for introducing a Q value network integration mechanism based on an Actor-Critic framework of reinforcement learning, and jointly optimizing the forward policy and the backward policy in combination with the forward reward signal and the backward reward signal.
[0022] In a third aspect, an electronic device is provided, which includes a memory and a processor, the memory storing a computer program, and the processor implementing the method of the first aspect when executing the computer program.
[0023] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program includes program instructions executable by a processor to implement the method of the first aspect.
[0024] The present application provides a machine learning method, device, equipment and storage medium, the method comprising: collecting expert demonstration data; wherein the expert demonstration data comprises forward target state data and backward target state data; obtaining interaction data through alternating control of forward policy and backward policy; wherein the interaction data comprises forward interaction data and backward interaction data; training a forward discriminator using the forward target state data and the forward interaction data to obtain a forward reward signal; training a backward discriminator using the backward target state data and the backward interaction data to obtain a backward reward signal; introducing a Q value network integration mechanism based on an Actor-Critic framework of reinforcement learning, and jointly optimizing the forward policy and the backward policy in combination with the forward reward signal and the backward reward signal. The present application reduces the dependence on artificial labeled data and reward functions through alternating control of forward policy and backward policy, double discriminator reward generation and Q value network integration optimization, and improves the learning stability, policy adaptability and autonomous improvement ability of robots in complex dynamic environments. BRIEF DESCRIPTION OF DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0026] Figure 1 The flowchart of the machine learning method provided by the embodiments of the present application;
[0027] Figure 2 The schematic block diagram of the machine learning device provided by the embodiments of the present application;
[0028] Figure 3 A schematic block diagram of an electronic device provided for an embodiment of the present application;
[0029] Figure 4 A schematic diagram of an application environment of a machine learning method provided for an embodiment of the present application. DETAILED DESCRIPTION
[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.
[0031] It should be understood that, when used in the specification and the appended claims, the terms “comprise” and “include” indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0032] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms “a”, “an” and “the” are intended to include the plural forms, unless the context clearly indicates otherwise.
[0033] It should be further understood that the term “and / or” used in the specification and the appended claims of the present application refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. The embodiments of the present application provide a machine learning method, device, equipment and storage medium. Please refer to Figure 4 , Figure 4An application environment schematic diagram of the machine learning method provided by the embodiment of the present application. The application environment of the machine learning method is composed of a robot system and a computing processing server, forming an autonomous learning closed loop: the robot system is equipped with a visual acquisition module (such as a camera), a motion execution mechanism (such as a mechanical arm, an execution terminal) and an environment interaction component, and can operate in actual environments such as medical operating tables and financial transaction scenes, and is responsible for collecting forward target state data (such as key state images of the expert completing the task) and backward target state data (such as state images of the expert correcting errors) in the expert operation process, and simultaneously interacting with the environment to generate forward interaction data (state-action sequence of forward operation) and backward interaction data (state-action sequence of backward correction) through the alternation control of forward strategy (forward operation of task execution) and backward strategy (reverse operation of deviation correction); the computing processing server (which can be an independent server or a server cluster) receives various data transmitted by the robot system, and uses the forward target state data and the forward interaction data to train the forward discriminator to generate a forward reward signal, and uses the backward target state data and the backward interaction data to train the backward discriminator to generate a backward reward signal; then based on the Actor-Critic framework, a Q value network integration mechanism is introduced to jointly optimize the forward strategy and the backward strategy, and the optimized robot strategy is issued to the robot system to drive it to realize autonomous iteration and performance improvement of the strategy in the actual environment. Through the cooperation of the robot system and the computing processing server, the whole environment realizes the autonomous operation of the whole process of data collection, reward generation and strategy optimization, and adapts to the demand of medical, financial and other fields for the autonomous learning ability of robots. The present application will be described in detail below through specific embodiments.
[0034] Figure 1 A flowchart of the machine learning method provided by the embodiment of the present application. As shown in the figure, the method comprises the following steps S110-S150. Figure 1
[0035] S110, collecting expert demonstration data; wherein the expert demonstration data comprises forward target state data and backward target state data.
[0036] In this embodiment, before the training starts, two types of trajectory data are collected through teleoperation, namely first target trajectory data of forward demonstration and second target trajectory data of backward demonstration. The forward demonstration shows the whole process of the robot completing the task (such as grabbing, inserting, etc.), and the backward demonstration is the reverse operation of the task (such as putting down, pulling out, etc.). The last several frames of the first target trajectory data are used as the forward target state data, and the remaining frames and the terminal frame of the second target trajectory data are used as the backward target state data.
[0037] For example, in a medical scenario, the forward demonstration is the process of a doctor remotely operating a forceps to move from an initial position, accurately clamp the edge of gauze and lift it up (taking the surgical gauze with the forceps), the last few frames of the first target trajectory data (e.g., the forceps stably clamping the gauze) are the forward target state data (the successful end point that the robot needs to achieve); the remaining frames of the forward demonstration and the end frames of the backward demonstration (e.g., the gauze returning to the original position) are the backward target state data.
[0038] For example, in a financial scenario, the forward demonstration is the process of an expert completing "input amount → confirm deposit → display deposit success" on the system (bank system deposit operation), the last few frames of the first target trajectory data (the interface displaying "deposit success") are the forward target state data; other screens in the forward demonstration (such as the interface after the amount is input but before confirmation), and the end frames of the backward demonstration (the expert's operation of returning from "having input the deposit amount" to "the non-deposit state", such as canceling the deposit) (returning to the initial non-deposit interface) are the backward target state data.
[0039] In an embodiment, step S110 includes: collecting first target trajectory data of the forward demonstration; collecting second target trajectory data of the backward demonstration; taking the last few frames of the first target trajectory data as the forward target state data, and taking the remaining frames and the end frames of the second target trajectory data as the backward target state data.
[0040] In this embodiment, the expert demonstration data is trajectory data of the forward / backward demonstration, the first target trajectory data of the forward demonstration is collected; the second target trajectory data of the backward demonstration is collected; the last few frames of the first target trajectory data are taken as the forward target state data, and the remaining frames and the end frames of the second target trajectory data are taken as the backward target state data.
[0041] S120, by alternating control of the forward strategy and the backward strategy, interaction data is obtained; wherein the interaction data includes forward interaction data and backward interaction data.
[0042] In this embodiment, before training, the robot strategy is initialized, the robot strategy includes a forward strategy and a backward strategy, the forward strategy operates according to the forward execution logic of the task, and the state and action generated by the interaction with the environment are the forward interaction data; the backward strategy operates according to the logic opposite to the forward strategy, and the state and action sequence generated by the interaction with the environment are the backward interaction data. The two are alternately run, so that interaction data containing forward execution and reverse operation is collected, and the collected interaction data is stored in an experience replay pool; wherein the forward / backward strategy is independently cached, and supports bidirectional data updating.
[0043] S130, training a forward discriminator by using the forward target state data and the forward interaction data to obtain a forward reward signal.
[0044] In this embodiment, when training the forward discriminator by using the forward target state data and the forward interaction data, the core function of the forward discriminator is to distinguish the forward interaction data generated by the forward strategy from the forward target state data. Specifically, the forward interaction data contains the state and action sequence generated by the forward strategy during execution, and the forward discriminator learns the difference between these data and the forward target state data (i.e., the key state of the task successfully achieved in the expert demonstration) to output the corresponding discrimination result. The forward reward signal is derived from the log-likelihood value output by the forward discriminator: the closer the forward interaction data of the forward strategy is to the forward target state data, the higher the log-likelihood value output by the discriminator, and the stronger the corresponding forward reward signal; otherwise, the weaker the forward reward signal. This way of generating the forward reward signal through the forward discriminator does not require manual design of an explicit reward function, which not only reduces the engineering cost of manually setting the reward, but also makes the forward reward signal more suitable for the actual needs of the task, improving the adaptability of the strategy training.
[0045] S140, training a backward discriminator by using the backward target state data and the backward interaction data to obtain a backward reward signal.
[0046] In this embodiment, when training the backward discriminator by using the backward target state data and the backward interaction data, the core function of the backward discriminator is to distinguish the backward interaction data generated by the backward strategy from the backward target state data. The backward interaction data contains the state and action sequence generated by the backward strategy during execution, and the backward discriminator learns the difference between these data and the backward target state data (i.e., the key state of the reverse operation in the expert demonstration) to output the corresponding discrimination result. The backward reward signal is determined by the log-likelihood value output by the backward discriminator: the closer the backward interaction data of the backward strategy is to the backward target state data, the higher the log-likelihood value output by the discriminator, and the stronger the corresponding backward reward signal; otherwise, the weaker the reward signal. This way does not require manual design of an explicit reward function, and generates the backward reward signal through the backward discriminator, which reduces the engineering cost of reward design and makes the backward reward signal more suitable for the actual needs of the backward operation, improving the adaptability to the reverse operation scenario.
[0047] In an embodiment, after step S140, it further includes: storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, and the backward interaction data and the corresponding backward reward signal in different areas of an experience replay pool.
[0048] In the embodiment, the expert demonstration data, the forward interaction data and the corresponding forward reward signal, and the backward interaction data and the corresponding backward reward signal are stored in different areas of the experience replay pool; wherein, the expert demonstration data (including the trajectory data of the forward / backward demonstration) is stored separately as a benchmark reference for policy learning; the forward interaction data and the corresponding forward reward signal are stored in one area for iterative optimization of the forward policy; and the backward interaction data and the corresponding backward reward signal are stored in another independent area for use in training of the backward policy. Through this way of partitioned storage, the different types of data can be prevented from interfering with each other, and the corresponding data can be quickly called during training to improve the efficiency and pertinence of policy learning.
[0049] In an embodiment, after the expert demonstration data, the forward interaction data and the corresponding forward reward signal, and the backward interaction data and the corresponding backward reward signal are stored in different areas of the experience replay pool, the method further includes: during the forward training process, acquiring a forward RBG image collected by a camera; extracting forward state features from the forward RBG image; performing oversampling on the experience replay pool based on the forward state features at a fixed ratio to obtain forward mixed training data; performing value evaluation on the forward mixed training data in combination with a Q-value network integration mechanism to generate a forward target Q-value; and introducing a behavior cloning regularization term to optimize the forward policy in combination with the forward target Q-value.
[0050] In the embodiment, the forward training process takes end-to-end learning from image to action as the core, and the specific process is as follows: first, an RGB image (i.e., a forward RBG image) in the forward operation process is collected by a camera, and forward state features containing scene details, object state changes and other information are extracted from the forward RBG image as input; based on the forward state features, relevant data in the experience replay pool is oversampled at a fixed ratio (such as a preset ratio of expert demonstration data to forward interaction data) to form forward mixed training data; then, value evaluation is performed on the forward mixed training data in combination with a Q-value network integration mechanism to generate a forward target Q-value by comprehensively outputting multiple Q-value networks, thereby improving the reliability of value judgment; wherein, the forward target Q-value is the minimum value among multiple forward Q-values to alleviate the problem of overestimation of Q-value; subsequently, a behavior cloning regularization term (constraining the deviation of the output of the forward policy from the expert demonstration action) is introduced, and the forward policy is iteratively optimized in combination with the forward target Q-value.
[0051] In an embodiment, the extracting forward state features from the forward RBG image includes: extracting forward state features from the forward RBG image by a visual encoder; wherein, the visual encoder is obtained through data enhancement training.
[0052] In the embodiment, forward state features are extracted from the forward RBG image by a visual encoder; wherein the visual encoder is trained by data augmentation (such as cropping, translation).
[0053] In an embodiment, after storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, and the backward interaction data and the corresponding backward reward signal in different areas of the experience replay pool, the method further comprises: in the backward training process, acquiring a backward RBG image collected by a camera; extracting backward state features from the backward RBG image; performing oversampling on the experience replay pool based on the backward state features at a fixed ratio to obtain backward mixed training data; performing value evaluation on the backward mixed training data in combination with a Q-value network integration mechanism to generate a backward target Q-value; and introducing a behavior cloning regularization term and optimizing the backward strategy in combination with the backward target Q-value.
[0054] In the embodiment, the backward training process is consistent with the forward training logic, and is also centered on realizing end-to-end learning from image to action: first, an RGB image (i.e., a backward RGB image) in a backward operation process is collected by a camera, and a visual encoder (which is the same encoder as that used in the forward training and trained by data augmentation to ensure consistency of feature extraction) extracts backward state features containing information such as details of the reverse operation scene and backtracking changes of object states from the backward RGB image; based on the backward state features, relevant data in the experience replay pool is oversampled at a fixed ratio (such as a preset ratio of expert backward data to backward interaction data) to form backward mixed training data; then, the backward mixed training data is evaluated in value in combination with a Q-value network integration mechanism; specifically, multiple parallel backward Q-value networks respectively output value estimates, and the minimum value among the estimates is taken as a backward target Q-value to alleviate the problem of overestimation of Q-value and improve the reliability of reverse operation value judgment; subsequently, a behavior cloning regularization term (which constrains the deviation of the backward strategy output from the expert backward demonstration action) is introduced, and the backward strategy is iteratively optimized in combination with the backward target Q-value.
[0055] S150, based on an Actor-Critic framework of reinforcement learning, introducing a Q-value network integration mechanism, and jointly optimizing the forward strategy and the backward strategy in combination with the forward reward signal and the backward reward signal.
[0056] In this embodiment, when the joint optimization of the forward strategy and the backward strategy is performed based on the Actor-Critic framework of reinforcement learning, the following implementation is specifically performed: the “Actor” corresponds to the forward strategy and the backward strategy, and is responsible for outputting specific action instructions; the “Critic” relies on the Q-value network integration mechanism, combines the forward reward signal (from the forward discriminator) and the backward reward signal (from the backward discriminator), and evaluates the action value of the forward / backward strategy; specifically, the Q-value network integration mechanism outputs value estimates through multiple parallel Q-value networks, takes the minimum value as the target Q-value (to alleviate the overestimation problem), and provides more reliable value feedback for the forward strategy and the backward strategy; in the joint optimization process, the update of the forward strategy depends on the evaluation results of the forward reward signal and the corresponding Q-value network, and the update of the backward strategy relies on the feedback of the backward reward signal and the related Q-value network.
[0057] By constructing the closed-loop training mechanism of the forward strategy and the backward strategy alternately iterating, the robot can realize long-term stable strategy optimization under the condition of high-dimensional visual input and without artificial reward: the system only needs to collect a small amount of forward and backward expert demonstration data (as an initial learning benchmark) in the early stage, and can rely on the forward / backward strategy to complete task practice and environment reset in subsequent training, significantly reducing the frequency of human participation, saving training manpower and time cost.
[0058] In an embodiment, step S150 comprises: based on the Actor-Critic framework of reinforcement learning, introducing a Q-value network integration mechanism, combining the forward reward signal and the backward reward signal, and introducing a behavior cloning regularization term, to jointly optimize the forward strategy and the backward strategy; wherein the behavior cloning regularization term is constructed based on expert demonstration data, and the weight of the behavior cloning regularization term linearly decays with the training progress.
[0059] In this embodiment, in the joint optimization process, a behavior cloning regularization term constructed based on expert demonstration data is introduced: the behavior cloning regularization term guides the forward / backward strategy to quickly learn the basic operation logic in the early stage of training by constraining the deviation between the policy output and the expert demonstration action; at the same time, the weight of the behavior cloning regularization term linearly decays with the training progress, ensuring that the policy can gradually get rid of the dependence on the expert demonstration data in the later stage, and explore a better operation mode through autonomous interaction with the environment.
[0060] In summary, the embodiment of the present application reduces the dependence on artificial labeled data and reward functions by alternating control of the forward strategy and the backward strategy, double discriminators generating rewards, and Q-value network integration optimization, and improves the learning stability, strategy adaptability, and autonomous improvement ability of the robot in a complex dynamic environment.
[0061] Figure 2This is a schematic block diagram of a machine learning apparatus provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above machine learning methods, the present invention also provides a machine learning device, the device being configured in, as shown in... Figure 4 In this application environment, the system consists of a robot system and a computing server, forming a self-learning closed loop. For details, please refer to [link to relevant documentation]. Figure 2 The machine learning device 700 includes:
[0062] The acquisition unit 701 is used to acquire expert demonstration data; wherein, the expert demonstration data includes forward target state data and backward target state data;
[0063] The acquisition unit 702 is used to acquire interactive data by alternating control of forward and backward strategies; wherein the interactive data includes forward interactive data and backward interactive data;
[0064] The forward training unit 703 is used to train the forward discriminator using the forward target state data and the forward interaction data, and to obtain the forward reward signal.
[0065] The backward training unit 704 is used to train the backward discriminator using the backward target state data and the backward interaction data, and to obtain the backward reward signal;
[0066] The joint optimization unit 705 is used in the Actor-Critic framework based on reinforcement learning to introduce a Q-value network ensemble mechanism and perform joint optimization of the forward policy and the backward policy by combining the forward reward signal and the backward reward signal.
[0067] In some embodiments, when the joint optimization unit 705 executes the reinforcement learning-based Actor-Critic framework, introduces a Q-value network ensemble mechanism, and performs a joint optimization step on the forward policy and the backward policy by combining the forward reward signal and the backward reward signal, it is specifically used for:
[0068] Based on the reinforcement learning-based Actor-Critic framework, a Q-value network ensemble mechanism is introduced, which combines the forward reward signal and the backward reward signal, and introduces a behavior cloning regularization term to jointly optimize the forward policy and the backward policy; wherein, the behavior cloning regularization term is constructed based on expert demonstration data, and the weight of the behavior cloning regularization term decays linearly with the training progress.
[0069] In some embodiments, after performing the steps of training a backward discriminator using the backward target state data and the backward interaction data, and obtaining a backward reward signal, the backward training unit 704 is further configured to:
[0070] The expert demonstration data, the forward interaction data and the corresponding forward reward signal, the backward interaction data and the corresponding backward reward signal are stored in different areas of an experience replay pool.
[0071] In some embodiments, the backward training unit 704 is further configured to, after performing the step of storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, the backward interaction data and the corresponding backward reward signal in different areas of an experience replay pool:
[0072] In the forward training process, a forward RBG image captured by a camera is obtained; forward state features are extracted from the forward RBG image; forward mixed training data are obtained by oversampling from the experience replay pool based on the forward state features at a fixed ratio; a forward target Q value is generated by performing value evaluation on the forward mixed training data in combination with a Q value network integration mechanism; and the forward policy is optimized in combination with the forward target Q value by introducing a behavior cloning regularization term.
[0073] In some embodiments, the backward training unit 704 is specifically configured to, when performing the step of extracting forward state features from the forward RBG image:
[0074] Forward state features are extracted from the forward RBG image by a visual encoder; wherein the visual encoder is obtained by data enhancement training.
[0075] In some embodiments, the backward training unit 704 is further configured to, after performing the step of storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, the backward interaction data and the corresponding backward reward signal in different areas of an experience replay pool:
[0076] In the backward training process, a backward RBG image captured by a camera is obtained; backward state features are extracted from the backward RBG image; backward mixed training data are obtained by oversampling from the experience replay pool based on the backward state features at a fixed ratio; a backward target Q value is generated by performing value evaluation on the backward mixed training data in combination with a Q value network integration mechanism; and the backward policy is optimized in combination with the backward target Q value by introducing a behavior cloning regularization term.
[0077] In some embodiments, the collection unit 701 is specifically configured to, when performing the step of collecting expert demonstration data:
[0078] First target trajectory data of forward demonstration are collected; second target trajectory data of backward demonstration are collected; the last several frames of the first target trajectory data are taken as the forward target state data, and the remaining frames and the end frame of the second target trajectory data are taken as the backward target state data.
[0079] It should be noted that the specific implementation process of the machine learning device and each unit can be clearly understood by those skilled in the art, and can refer to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.
[0080] The machine learning device described above can be implemented in the form of a computer program, which can run on an electronic device as shown in the specification. Figure 3 The electronic device 800 can be a terminal or a server. The terminal can be an electronic device with communication function. The server can be a stand-alone server or a server cluster composed of multiple servers.
[0081] Please refer to Figure 3 , Figure 3 is a schematic block diagram of an electronic device provided by an embodiment of the present application. The electronic device 800 can be a terminal or a server. The terminal can be an electronic device with communication function. The server can be a stand-alone server or a server cluster composed of multiple servers.
[0082] Please refer to Figure 3 , the electronic device 800 includes a processor 802, a memory and a network interface 805 connected through a system bus 801, wherein the memory can include a non-volatile storage medium 803 and an internal memory 804.
[0083] The non-volatile storage medium 803 can store an operating system 8031 and a computer program 8032. The computer program 8032 includes program instructions which, when executed, can cause the processor 802 to perform a machine learning method.
[0084] The processor 802 is configured to provide computing and control capabilities to support the operation of the entire electronic device 800.
[0085] The internal memory 804 provides an environment for the running of the computer program 8032 in the non-volatile storage medium 803, which, when executed by the processor 802, can cause the processor 802 to perform a machine learning method.
[0086] The network interface 805 is configured to perform network communication with other devices. Those skilled in the art can understand that Figure 3 The structure shown in the specification is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device 800 to which the scheme of the present application is applied. The specific electronic device 800 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0087] The processor 802 is configured to run the computer program 8032 stored in the memory to implement the following steps:
[0088] The expert demonstration data includes forward target state data and backward target state data; interaction data is obtained through alternating control of the forward strategy and the backward strategy; the interaction data includes forward interaction data and backward interaction data; a forward discriminator is trained using the forward target state data and the forward interaction data to obtain a forward reward signal; a backward discriminator is trained using the backward target state data and the backward interaction data to obtain a backward reward signal; and the forward strategy and the backward strategy are jointly optimized based on an Actor-Critic framework of reinforcement learning, by introducing a Q-value network integration mechanism and combining the forward reward signal and the backward reward signal.
[0089] In some embodiments, when implementing the step of implementing an Actor-Critic framework of reinforcement learning, introducing a Q-value network integration mechanism, and jointly optimizing the forward strategy and the backward strategy by combining the forward reward signal and the backward reward signal, the processor 802 specifically implements the following steps:
[0090] The forward strategy and the backward strategy are jointly optimized based on an Actor-Critic framework of reinforcement learning, by introducing a Q-value network integration mechanism, combining the forward reward signal and the backward reward signal, and introducing a behavior cloning regularization term; the behavior cloning regularization term is constructed based on expert demonstration data, and the weight of the behavior cloning regularization term is linearly decayed with training progress.
[0091] In some embodiments, after implementing the step of training a backward discriminator using the backward target state data and the backward interaction data to obtain a backward reward signal, the processor 802 further implements the following steps:
[0092] The expert demonstration data, the forward interaction data and corresponding forward reward signal, the backward interaction data and corresponding backward reward signal are stored in different areas of an experience replay pool.
[0093] In some embodiments, after implementing the step of storing the expert demonstration data, the forward interaction data and corresponding forward reward signal, the backward interaction data and corresponding backward reward signal in different areas of an experience replay pool, the processor 802 further implements the following steps:
[0094] In the forward training process, a forward RBG image captured by a camera is obtained; forward state features are extracted from the forward RBG image; forward mixed training data is obtained by oversampling from the experience replay pool based on the forward state features at a fixed ratio; a forward target Q-value is generated by performing value evaluation on the forward mixed training data in combination with the Q-value network integration mechanism; and the forward strategy is optimized in combination with the forward target Q-value by introducing a behavior cloning regularization term.
[0095] In some embodiments, the processor 802, in implementing the step of extracting forward state features from the forward RBG image, implements the following step:
[0096] extracting forward state features from the forward RBG image by a visual encoder; wherein the visual encoder is trained by data augmentation.
[0097] In some embodiments, the processor 802, after implementing the step of storing the expert demonstration data, the forward interaction data and corresponding forward reward signal, the backward interaction data and corresponding backward reward signal into different areas of the experience replay pool, implements the following step:
[0098] In the backward training process, a backward RBG image captured by a camera is obtained; backward state features are extracted from the backward RBG image; the backward mixed training data is obtained by oversampling from the experience replay pool based on the backward state features at a fixed ratio; the backward target Q value is generated by combining the Q value network integration mechanism to evaluate the value of the backward mixed training data; and the backward strategy is optimized by introducing a behavior cloning regularization term and combining the backward target Q value.
[0099] In some embodiments, the processor 802, in implementing the step of collecting expert demonstration data, implements the following step:
[0100] collecting first target trajectory data of forward demonstration; collecting second target trajectory data of backward demonstration; taking the last several frames of the first target trajectory data as the forward target state data, and taking the remaining frames and the end frame of the second target trajectory data as the backward target state data.
[0101] It should be understood that, in the embodiments of the present application, the processor 802 can be a central processing unit (CPU), and the processor 802 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0102] Those skilled in the art can understand that all or part of the processes in the method of implementing the above embodiments can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.
[0103] Therefore, the present application also provides a storage medium. The storage medium can be a computer readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions are executed by the processor to make the processor perform the following steps:
[0104] collecting expert demonstration data; wherein the expert demonstration data includes forward target state data and backward target state data; obtaining interaction data through alternating control of the forward strategy and the backward strategy; wherein the interaction data includes forward interaction data and backward interaction data; training a forward discriminator using the forward target state data and the forward interaction data to obtain a forward reward signal; training a backward discriminator using the backward target state data and the backward interaction data to obtain a backward reward signal; introducing a Q-value network integration mechanism based on an Actor-Critic framework of reinforcement learning, and combining the forward reward signal and the backward reward signal to jointly optimize the forward strategy and the backward strategy.
[0105] In an embodiment, when the processor executes the program instructions to implement the step of introducing a Q-value network integration mechanism based on an Actor-Critic framework of reinforcement learning, combining the forward reward signal and the backward reward signal, and jointly optimizing the forward strategy and the backward strategy, the processor specifically implements the following steps:
[0106] introducing a Q-value network integration mechanism based on an Actor-Critic framework of reinforcement learning, combining the forward reward signal and the backward reward signal, and introducing a behavior cloning regularization term to jointly optimize the forward strategy and the backward strategy; wherein the behavior cloning regularization term is constructed based on the expert demonstration data, and the weight of the behavior cloning regularization term is linearly decayed with the training progress.
[0107] In an embodiment, after the processor executes the program instructions to implement the step of training a backward discriminator using the backward target state data and the backward interaction data to obtain a backward reward signal, the processor further implements the following steps:
[0108] storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, the backward interaction data and the corresponding backward reward signal into different areas in an experience replay pool.
[0109] In an embodiment, the processor, after implementing the different area step of storing the expert demonstration data, the forward interaction data and corresponding forward reward signal, the backward interaction data and corresponding backward reward signal into the experience replay pool while executing the program instructions, further implements the following steps:
[0110] In the forward training process, a forward RBG image collected by a camera is obtained; a forward state feature is extracted from the forward RBG image; a forward mixed training data is obtained by oversampling from the experience replay pool based on the forward state feature at a fixed ratio; a forward target Q value is generated by combining a Q value network integration mechanism to evaluate the value of the forward mixed training data; and the forward strategy is optimized in combination with the forward target Q value by introducing a behavior cloning regularization term.
[0111] In an embodiment, the processor, when implementing the step of extracting a forward state feature from the forward RBG image while executing the program instructions, specifically implements the following steps:
[0112] A forward state feature is extracted from the forward RBG image by a visual encoder; wherein the visual encoder is obtained by data enhancement training.
[0113] In an embodiment, the processor, after implementing the different area step of storing the expert demonstration data, the forward interaction data and corresponding forward reward signal, the backward interaction data and corresponding backward reward signal into the experience replay pool while executing the program instructions, further implements the following steps:
[0114] In the backward training process, a backward RBG image collected by a camera is obtained; a backward state feature is extracted from the backward RBG image; a backward mixed training data is obtained by oversampling from the experience replay pool based on the backward state feature at a fixed ratio; a backward target Q value is generated by combining a Q value network integration mechanism to evaluate the value of the backward mixed training data; and the backward strategy is optimized in combination with the backward target Q value by introducing a behavior cloning regularization term.
[0115] In an embodiment, the processor, when implementing the step of collecting expert demonstration data while executing the program instructions, specifically implements the following steps:
[0116] First target trajectory data of a forward demonstration is collected; second target trajectory data of a backward demonstration is collected; the last several frames of the first target trajectory data are taken as the forward target state data, and the remaining frames and the end frame of the second target trajectory data are taken as the backward target state data.
[0117] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.
[0118] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0119] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of each unit is only a logical functional division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0120] The steps in the method embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the device embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.
[0121] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing an electronic device (which can be a personal computer, a terminal or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.
[0122] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A machine learning method, characterized by, The method comprises: collecting expert demonstration data; wherein the expert demonstration data comprises forward target state data and backward target state data; acquiring interaction data through alternating control of the forward strategy and the backward strategy; wherein the interaction data comprises forward interaction data and backward interaction data; training a forward discriminator using the forward target state data and the forward interaction data to acquire a forward reward signal; training a backward discriminator using the backward target state data and the backward interaction data to acquire a backward reward signal; based on an Actor-Critic framework of reinforcement learning, introducing a Q-value network integration mechanism, combining the forward reward signal and the backward reward signal to jointly optimize the forward strategy and the backward strategy.
2. The machine learning method of claim 1, wherein, The Actor-Critic framework of reinforcement learning introduces a Q-value network integration mechanism, combines the forward reward signal and the backward reward signal to jointly optimize the forward strategy and the backward strategy, which comprises: based on an Actor-Critic framework of reinforcement learning, introducing a Q-value network integration mechanism, combining the forward reward signal and the backward reward signal, and introducing a behavior cloning regularization term to jointly optimize the forward strategy and the backward strategy; wherein the behavior cloning regularization term is constructed based on expert demonstration data, and the weight of the behavior cloning regularization term linearly decays with the training progress.
3. The machine learning method of claim 1, wherein, After the backward discriminator is trained using the backward target state data and the backward interaction data to acquire a backward reward signal, the method further comprises: storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, the backward interaction data and the corresponding backward reward signal in different areas of an experience replay pool.
4. The machine learning method of claim 3, wherein, After storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, the backward interaction data and the corresponding backward reward signal in different areas of an experience replay pool, the method further comprises: in the forward training process, acquiring a forward RBG image collected by a camera; extracting forward state features from the forward RBG image; based on the forward state features, oversampling from the experience replay pool at a fixed ratio to acquire forward mixed training data; combining the Q-value network integration mechanism to evaluate the value of the forward mixed training data to generate forward target Q-values; introducing a behavior cloning regularization term to optimize the forward strategy based on the forward target Q-values.
5. The machine learning method of claim 4, wherein, The method for extracting forward state features from the forward RBG image comprises: extracting forward state features from the forward RBG image through a visual encoder; wherein the visual encoder is trained through data augmentation.
6. The machine learning method of claim 3, wherein, After storing the expert demonstration data, the forward interaction data and the corresponding forward reward signal, the backward interaction data and the corresponding backward reward signal in different areas of an experience replay pool, the method further comprises: in the backward training process, acquiring a backward RBG image collected by a camera; extracting backward state features from the backward RBG image; The backward mixed training data is obtained by oversampling the experience replay pool based on the backward state features in a fixed proportion; The backward mixed training data is evaluated in value by combining a Q-value network integration mechanism to generate a backward target Q-value; A behavior cloning regularization term is introduced, and the backward strategy is optimized by combining the backward target Q-value.
7. The machine learning method of claim 1, wherein, The expert demonstration data includes: First target trajectory data of forward demonstration is collected; Second target trajectory data of backward demonstration is collected; The last several frames of the first target trajectory data are used as the forward target state data, and the remaining frames and the last frame of the second target trajectory data are used as the backward target state data.
8. A machine learning device, comprising: The device includes: An acquisition unit is configured to acquire expert demonstration data, wherein the expert demonstration data includes forward target state data and backward target state data; An acquisition unit is configured to acquire interaction data by alternating control of the forward strategy and the backward strategy, wherein the interaction data includes forward interaction data and backward interaction data; A forward training unit is configured to train a forward discriminator using the forward target state data and the forward interaction data to obtain a forward reward signal; A backward training unit is configured to train a backward discriminator using the backward target state data and the backward interaction data to obtain a backward reward signal; A joint optimization unit is configured to introduce a Q-value network integration mechanism based on an Actor-Critic framework of reinforcement learning, and to jointly optimize the forward strategy and the backward strategy by combining the forward reward signal and the backward reward signal.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the machine learning method of any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program includes program instructions which, when executed by a processor, cause the processor to execute the machine learning method of any one of claims 1-7.